New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration to PostgreSQL #275

Open
docapotamus opened this Issue Jun 27, 2016 · 5 comments

Comments

Projects
None yet
2 participants
@docapotamus
Member

docapotamus commented Jun 27, 2016

Migrate database infrastructure from MongoDB to PostgreSQL.

There is nothing wrong with MongoDB. There are a few features which would be really handy that we get from PostgreSQL such as recursive CTEs and JOINs.

I don't want to start a religious arguments about database technologies. This will allow us to perform some more things with the database and that is why the move is being proposed.

I do not want to use MySQL/MariaDB as the CTEs are not available there.

We WILL NOT be using an ORM. I personally do not thing they adding anything that can't be done with SQL itself and in some ways make it more complex that straight SQL.

I have changed my mind on the above. We will more than likely use SQLAlchemy due to support and documentation. We can use raw SQL in the places where we need it. We just need to make sure using the ORM does not obscure the code or hide the details of how the underlying SQL is being used.

We will use psycopg2 as the driver.

I am hoping we can clean a lot of the code out during this migration and hopefully simplify the internal back end APIs.

SQL is a very stable technology and in particular PostgreSQL. This should make deploying Pjuu much easier for other users. Also SQL knowledge is much more wide spread than MongoDB knowledge so it should lower the barrier to contribution.

Another thing we have never had is a proper migration framework. We have used customer scripts in the scripts/ folder to handle transforming data between upgrade.

I will add more notes in the future but these will more than likely be handled manually with an actual method of keeping track on what revision we are at. Similar to Django/Alembic. I am not sure if we will use these or not.

This work will start when I have time and will be performed in a long running PR. I will reference this issue when it is created.

@docapotamus

This comment has been minimized.

Member

docapotamus commented Sep 19, 2016

As part of this work I propose to clean up the development work flow and use the Click package (now part of Flask).

This will allow us to have a manage.py file similar to Django which we can integrate everything in to. This will remove the need for the Makefile.

Flask-Migrate can plug in to it so we can handle database migrations. We can add custom commands for running the test suite and flake.

docapotamus added a commit that referenced this issue Sep 29, 2016

Initial work for migration to PostgreSQL.
- Moved away from `Makefile` to `manage.py` script which allows easier
  integration with the migration system. Powered by the Click package
  which is now part of Flask.
- Removed CherryPy for development server to reduce dependecies and make
  keeping them up to date easier.
- Removed the old scripts folder as these will no longer be needed for
  newer deployments. Replaced with one (empty) script to migrate from
  MongoDB to Postgres when the work is done.
- Removed un-needed scripts which the Makefile called.
- Updated README, Travis and Vagrant to start reflecting how we want the
  development workflow to move along with the move to SQL.
- Removed MongoDB from application initialization and added SQLAlchemy
  and Migrations (Alembic).

This by no means works and is just some of the house keeping tasks which
will give us an easier work flow once we start the actual migration
work.

This relates to issue #275.

Please feel free to review the changes and discuss on this PR or on the
issue.

docapotamus added a commit that referenced this issue Sep 29, 2016

Initial work for migration to PostgreSQL.
- Moved away from `Makefile` to `manage.py` script which allows easier
  integration with the migration system. Powered by the Click package
  which is now part of Flask.
- Removed CherryPy for development server to reduce dependecies and make
  keeping them up to date easier.
- Removed the old scripts folder as these will no longer be needed for
  newer deployments. Replaced with one (empty) script to migrate from
  MongoDB to Postgres when the work is done.
- Removed un-needed scripts which the Makefile called.
- Updated README, Travis and Vagrant to start reflecting how we want the
  development workflow to move along with the move to SQL.
- Removed MongoDB from application initialization and added SQLAlchemy
  and Migrations (Alembic).

This by no means works and is just some of the house keeping tasks which
will give us an easier work flow once we start the actual migration
work.

This relates to issue #275.

Please feel free to review the changes and discuss on this PR or on the
issue.
@docapotamus

This comment has been minimized.

Member

docapotamus commented Oct 12, 2016

@mcrmonkey Work still under way with the PR. Going to be removing the backend.py abstraction. I don't think this has really helped with fine tuning queries with MongoDB. Also when I get round to adding an API the queries will be similar but not the exact same so it isn't that useful

@mcrmonkey

This comment has been minimized.

Member

mcrmonkey commented Oct 12, 2016

👍

@docapotamus docapotamus added on hold and removed up for grabs labels Oct 13, 2016

@docapotamus

This comment has been minimized.

Member

docapotamus commented Oct 13, 2016

Just going to put this on hold for now. I want to add a couple of features before going further just so I know the whole scope of the work.

@mcrmonkey

This comment has been minimized.

Member

mcrmonkey commented May 5, 2018

#338 blocks this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment