-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix constraints to use .drop_all() instead of dropping database #1403
Conversation
Maintaining database compatibility causes nothing but problems and I do not see any benefit to keeping Postgres support inside of CTFd. For example, by specifying the name you likely stop the automatic foreign key name generation so there's probably a weird oddly named foreign key inside of MySQL (the actual CTFd database). You can already deploy MySQL/MariaDB on Heroku so I don't see why you need to fight to support Postgres. |
Codecov Report
@@ Coverage Diff @@
## master #1403 +/- ##
=========================================
Coverage ? 89.01%
=========================================
Files ? 105
Lines ? 5591
Branches ? 0
=========================================
Hits ? 4977
Misses ? 614
Partials ? 0
Continue to review full report at Codecov.
|
PostgreSQL on Heroku brings a 10k rows database for free. The options for MySQL/MariaDB only brings a 5Mb databases for free, this is around 1K rows, maybe less so is impossible to use in a minimal CTFd setup. Also, the resource usage of PostgreSQL in Docker is lower than MySQL/MariaDB so my students can develop in our low resource machines. PostgreSQL docker container is taking 8Mb of RAM while MySQL/MariaDB is taking more than 200Mb in idle. These are the main reasons to want to try to maintain the compatibility with PostgreSQL, I would like to collaborate with this project giving patches, but if it's a problem I can maintain my own fork. Best regards! |
Thanks for your PR, I'm appreciative that you're using CTFd for education. There isn't an issue with sending a PR but I take issue with PRs that solve one-off problems specific to one particular use case because they end up very disorganized. In this case this tramples on a general expectation that CTFd only supports Postgres unofficially and starts adding warts on MySQL. I admittedly have considered retaining Postgres support because I have read the literature that it's "superior" but then I remember the hours that I've spent investigating weird differences between how MySQL and Postgres handle data. There's a lot of added complexity for a database that's not really very widely used when the recommendation is MySQL. To retain Postgres support I would have to see most if not all of the Postgres specific code removed from the CTFd code base and replaced with db-agnostic code. |
Thanks to you for your project, I started the course doing an easy CTF about informatics, later they learn how to deploy locally, later to deploy on the cloud, and finally how to modify the code. This project fits all needs of the course. I agree about don't add complexity to support different databases, that is why I did this new PR, the constraint problem is also in MySQL, so this fixes it, you can see the raised error when calling to
This PR fixes this error and seems a cleaner approach because is the recommended usage in sqlachemy docs. More info about why using use_alter=True to avoid constraint errors: https://docs.sqlalchemy.org/en/13/core/constraints.html#creating-dropping-foreign-key-constraints-via-alter In the future, my idea is removing the I checked the code and maybe the messiest problem with different database engines is in the import/export system, that can be changed for an easier approach using the built-in About your reads: I worked as DevOps for years and yes, PostgreSQL is so much better, that's why it is the default engine in other projects like Django. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm leaving this review as a record of my comments and thoughts. As it stands, without the upgrade line the code won't work for old imports so that at the minimum must be fixed.
Overall, I'm not sold that this PR has significant benefits and I don't think supporting Postgres outweighs the benefits of standardization.
CTFd/utils/exports/__init__.py
Outdated
@@ -165,16 +159,11 @@ def import_ctf(backup, erase=True): | |||
|
|||
members = first + members | |||
|
|||
upgrade(revision=alembic_version) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By removing this line, tables are created to the latest migration and imports will fail if they are from a previous version that had different data.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Restored that line
# will fail: https://github.com/kvesteri/sqlalchemy-utils/pull/372 | ||
# in that case we asume that the database was previously created and we can't | ||
# drop or create the database | ||
exists = database_exists_util(url) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even if we ignore the Postgres support discussion, I don't know if I would accept this change. Too hacky for my tastes and probably could affect other things. Maybe better to only exception handle under Postgres.
We would need to wait for kvesteri/sqlalchemy-utils#372
@@ -123,8 +118,7 @@ def import_ctf(backup, erase=True): | |||
) | |||
|
|||
if erase: | |||
drop_database() | |||
create_database() | |||
app.db.drop_all() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While I was working on #1419, I experimented with this functionality to see if it resolved an issue I was running into but it didn't. This might be better practice but I am not convinced that it has an actual benefit in CTFd.
CTFd/models/__init__.py
Outdated
team_id = db.Column(db.Integer, db.ForeignKey("teams.id")) | ||
# use_alter=True along with name='' adds this foreign key after Teams has been created to avoid circular dependency | ||
team_id = db.Column( | ||
db.Integer, db.ForeignKey("teams.id", use_alter=True, name="users_team_id_fkey") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't like pinning the name here. If we can tap into the Sqlalchemy built-in name generator, I would be more willing to merge this in but I still am not certain I want to go down the path of supporting multiple databases overall.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I removed the name for alter=true is mandatory manually set the name of the constraint using its internal conventions: https://alembic.sqlalchemy.org/en/latest/naming.html
Closing this PR because don't seems more needed |
Fix constraints to use
use_alter=True
in the foreign key between Users and Teams, this allows to use the sqlalchemy's built-in.drop_all()
function instead of dropping database via sqlalchemy-utils'drop_database()
.Drop a running database in production is a totally discouraged practice, so a better approach is dropping all database objects. This is the main reason why sqlalchemy don't have built-in methods for dropping and creating databases.
This PR improves PostgreSQL support to try to maintain currently supported databases in the upcoming 3.x versión
This PR also fixes #1397 so I can close that PR
More info about why using
use_alter=True
to avoid constraint errors: https://docs.sqlalchemy.org/en/13/core/constraints.html#creating-dropping-foreign-key-constraints-via-alter