Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrations: Implement bookkeeping #487

Conversation

eecavanna
Copy link
Collaborator

@eecavanna eecavanna commented Mar 5, 2024

Description

In this branch, I have implemented a new class. Its name is Bookkeeper. It can be used to do two things:

  • Create documents in a Mongo collection (not represented in the NMDC Schema). Each of these documents is a "log entry" about a migration. There are two types of log entries: "migration started" and "migration completed".
  • Create (and ensure the existence of) a Mongo view. This view contains a single (virtual) document. That document is effectively the latest log entry from the aforementioned collection. People will be able to use this view to quickly see which schema version, if any, the database is "at".

Also in this branch, I have implement an example migration notebook. That can be used for team member training purposes and as a template for new migration notebooks.

Fixes #432

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration, if it is not simply make up-test && make test-run.

  • Once your environment is configured as described in the "Configuration Details" section below, run python -m unittest /path/to/bookkeeper.py at the shell of the fastapi container.

Configuration Details: Spin up a temporary Mongo server (e.g. using Docker). Then, edit the Mongo connection-related constants in test_bookkeeper.py accordingly.

Checklist:

  • My code follows the style guidelines of this project (have you run black nmdc_runtime/?)

    I have run it on the pure Python modules, but—intentionally—not on the Python notebooks.

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (in docs/ and in https://github.com/microbiomedata/NMDC_documentation/?)

    The changes in this branch don't affect the documentation.

  • I have added tests that prove my fix is effective or that my feature works, incl. considering downstream usage (e.g. https://github.com/microbiomedata/notebook_hackathons) if applicable.
  • New and existing unit and functional tests pass locally with my changes (make up-test && make test-run)

    For one of the modules I introduced in this branch, I prefixed its name with manual_test_ instead of test_ so that it would not be run by the GitHub Action workflow that runs the tests. That is after having run into issues with the way the Mongo server used for testing is configured.

@eecavanna eecavanna self-assigned this Mar 5, 2024
@eecavanna eecavanna changed the title 432 migrations implement bookkeeping so database tells us which schema it conforms to Migrations: Implement bookkeeping Mar 5, 2024
@eecavanna eecavanna marked this pull request as ready for review March 6, 2024 02:04
@eecavanna
Copy link
Collaborator Author

I don't typically submit the migration-related code for review.

@eecavanna eecavanna merged commit 89fd282 into main Mar 6, 2024
2 checks passed
@eecavanna eecavanna deleted the 432-migrations-implement-bookkeeping-so-database-tells-us-which-schema-it-conforms-to branch March 6, 2024 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Migrations: Implement bookkeeping so database tells us which schema it conforms to
1 participant