DRAFT 1.1 Migration Guide

This page covers migrating from CollectOSS v1.0 to CollectOSS v1.1. Since this release contains several changes that affect downstream consumers of the CollectOSS database and may also affect the configuration of existing instances, we are providing this migration guide with suggestions for how to handle some of the changes.

If you would like maintainer assistance with any of these steps, or if you have questions that are not addressed here, you can use the CHAOSS Slack workspace in the #wg-collectoss-8knot channel to contact the maintainers.

Database Names

This is the most breaking category of changes in this release because there is no way to enable a soft migration.

Schema names

What: v1.1 has renamed the core database schemas as follows:

• augur_data -> data
• augur_operations -> operations

Why: The old project name was encoded directly into the schema names. Unfortunately, it is not technically possible to configure CollectOSS to preserve the functioning of both names in order to provide a transition period.

Impact: Any applications that are hardcoded to relying on the schema names in queries (such as when fully-qualifying column names), must be updated to use the new names. This is a breaking change.

If your deployment has downstream applications relying on hardcoded schemas, we encourage you to update your downstream applications to utilize the postgres schema search path. We recommend setting the search path so that it contains the string data,augur_data, ensuring that any table names in queries that are not fully-qualified will first be searched for in data (the new schema name) if it is present, falling back to augur_data (the old name) if it isn't. This method is recommended because it also helps makes the transition easier since it reduces the need to coordinate and deploy updates to multiple applications (such as 8Knot) at the same time.

If your application relies on fully qualified names, replacing the old name in your queries with the new one is also a valid (but not recommended) migration strategy.

Note

The data and operations schemas are considered internal to CollectOSS and may be subject to breaking changes. In a future version, we will be introducing a stable schema that will be tested and will become part of our public/stable API commitment.

Column Names

What: some column names within the core data schema were updated as part of this version:

• data.pull_requests.pr_augur_contributor_id is now known as data.pull_requests.pr_contributor_id
• data.pull_requests.pr_augur_issue_id is now known as data.pull_requests.pr_issue_id

Why: The old project name was encoded directly into the column names. Unfortunately, it is not technically possible to configure CollectOSS to preserve the functioning of both names in order to provide a transition period.

Impact: Any applications connecting to CollectOSS that make queries involving the affected columns must be updated to use the new column names in their queries. This is a breaking change and is the one that is most likely to break downstream applications that are connecting to the CollectOSS database since the pull_requests table is relatively commonly used.

Table Names

What: Some table names were also updated as part of this version:

• operations.augur_settings is now known as operations.legacy_settings
• spdx.augur_repo_map is now known as spdx.repo_map

Why: The old project name was encoded directly into the table names. Unfortunately, it is not technically possible to configure CollectOSS to preserve the functioning of both names in order to provide a transition period.

Impact: Any applications connecting to CollectOSS that make queries involving the affected tables must be updated to use the new column names in their queries. This is a breaking change but the impact may likely be relatively small since both of these tables were either barely being used anyway, or were located inside a schema that we suspect has not been in active use for some time.

Other database Names

What: Other, more internal elements of the database schema were also renamed:

• Sequence spdx.augur_repo_map_map_id_seq was renamed to spdx.repo_map_map_id_seq
• Foreign Key Constraint data.pull_requests.pull_requests_pr_augur_contributor_id_fkey was renamed to data.pull_requests.pull_requests_contributor_id_fkey

Why: The old project name was encoded directly into the names of these elements. Unfortunately, it is not technically possible to configure CollectOSS to preserve the functioning of both names in order to provide a transition period.

Impact: Because these elements are generally more internal database structures (constraints and sequences), they are very unlikely to cause breakage.

(optional) Environment Variables

What: CollectOSS v1.1 has changed the name of its environment variables to prefer environment variables prefixed with COLLECTOSS_ rather than AUGUR_ by default. Any environment variables that don't have a prefix, such as CONFIG_DATADIR can remain the same.

Why: The old project name was encoded directly into most of the environment variable names that CollectOSS used for fetching configuration values. CollectOSS v1.1 introduces a new environment variable subsystem that allows the application to read from environment variables that are present with either the new COLLECTOSS_ or the old AUGUR_ prefixes (with the newer naming preferred if variables with both names are present. This change prepares the application for a future removal of support for the old naming scheme (the date for this is unplanned and will likely be in the far future).

Impact: This is a low-impact optional migration step that can be completed at any time. Because the new subsystem allows for ether name to be used, we recommend, but do not require you migrate to the new names at your convenience. Migration can happen one variable at a time if a slower transition is desired.

Config File Naming

CollectOSS v1.1 also changed the name of the primary config file from augur.json to collectoss.json. The config manager will first look for the new file name, falling back to the old one if it cannot be found. This config file, if used, should be located in the directory specified by the CONFIG_DATADIR environment variable.

Docker Entrypoints

RabbitMQ

The custom RabbitMQ image that Augur shipped with was known to require the RabbitMQ credentials to be provided at build time (which were then baked into the image). CollectOSS v1.1 simplifies the configuration files and relies on the upstream rabbitmq:4.1-management-alpine container image to provide the RabbitMQ service for the data collection portions of CollectOSS.

To migrate, update your docker compose file to use the rabbitmq:4.1-management-alpine image for the rabbitmq service and specify the environment variables and path mounts based on the example provided in the docker-compose.yml in the root of the repo.

Optional: Database rename

This part concerns renaming your database. This is a breaking change for anything that is connected to your database and is completely optional. It is recommended, but not required to perform this step after you have already migrated to collectoss 1.1.

To rename your database:

1. Ensure you have an adequate backup. These instructions will not alter the contents of your CollectOSS database, but this process involves potentially creating and deleting a temporary database. Mistyping these commands may lead to data loss.
2. Stop CollectOSS but keep your database online (using a command like docker stop collectoss-redis-1 collectoss-core-1 collectoss-rabbitmq-1 collectoss-flower-1 collectoss-keyman-1 collectoss-core-1 if your docker compose file was started from a folder called "collectoss")
3. Ensure that all connections to the database are disconnected. Note that some may still be present from other applications
4. If you have another database on your PostgresQL instance, disconnect and reconnect using that database. You may need to create a new temporary database for this purpose. The following step will not work if you are connecting to the database that is being renamed.
5. Run ALTER DATABASE augur RENAME TO collectoss; (or choose your own new name)
6. Ensure that the new database name is updated in your docker container configuration for CollectOSS (via environment variables, specifically for the database, core, and flower services).
7. If you created a temporary database for an earlier step, you can now connect to the newly renamed database and drop the test database.
8. Start CollectOSS again (such as with docker start collectoss-redis-1 collectoss-core-1 collectoss-rabbitmq-1 collectoss-flower-1 collectoss-keyman-1 collectoss-core-1) to ensure that everything comes up the way you expect.
9. Ensure other downstream applications that connect to your database are updated with the new database name. Examples of these downstreams may include:
   • The collectoss-monitoring grafana dashboards (if in use)
   • 8Knot
   • any other downstream tools connecting to your database for analysis work (scripts, Jupyter notebooks, etc).