DRAFT 1.1 Migration Guide

This page covers migrating from CollectOSS v1.0 to CollectOSS v1.1. Since this release contains several changes that affect downstream consumers of the CollectOSS database and may also affect the configuration of existing instances, we are providing this migration guide with suggestions for how to handle some of the changes.

If you would like maintainer assistance with any of these steps, or if you have questions that are not addressed here, you can use the CHAOSS Slack workspace in the #wg-collectoss-8knot channel to contact the maintainers.

Database Names

This is the most breaking category of changes in this release because there is no way to enable a soft migration.

Schema names

What: v1.1 has renamed the core database schemas as follows:

• augur_data -> data
• augur_operations -> operations

Why: The old project name was encoded directly into the schema names. Unfortunately, it is not technically possible to configure CollectOSS to preserve the functioning of both names in order to provide a transition period.

Impact: Any applications that are hardcoded to relying on the schema names in queries (such as when fully-qualifying column names), must be updated to use the new names. This is a breaking change. If you depend on this behavior, you must make the changes in the affected systems before migrating CollectOSS to avoid breakage.

If your deployment has downstream applications relying on hardcoded schemas, we encourage you to update your downstream applications to utilize the postgres schema search path. We recommend setting the search path so that it contains the string data,augur_data, ensuring that any table names in queries that are not fully-qualified will first be searched for in data (the new schema name) if it is present, falling back to augur_data (the old name) if it isn't. This method is recommended because it also helps makes the transition easier since it reduces the need to coordinate and deploy updates to multiple applications (such as 8Knot) at the same time.

If your application relies on fully qualified names, replacing the old name in your queries with the new one is also a valid (but not recommended) migration strategy.

Note

The data and operations schemas are considered internal to CollectOSS and may be subject to breaking changes. In a future version, we will be introducing a stable schema that will be tested and will become part of our public/stable API commitment.

Column Names

What: some column names within the core data schema were updated as part of this version:

• data.pull_requests.pr_augur_contributor_id is now known as data.pull_requests.pr_contributor_id
• data.pull_requests.pr_augur_issue_id is now known as data.pull_requests.pr_issue_id

Why: The old project name was encoded directly into the column names. Unfortunately, it is not technically possible to configure CollectOSS to preserve the functioning of both names in order to provide a transition period.

Impact: Any applications connecting to CollectOSS that make queries involving the affected columns must be updated to use the new column names in their queries. This is a breaking change and is the one that is most likely to break downstream applications that are connecting to the CollectOSS database since the pull_requests table is relatively commonly used. If you depend on this behavior, you must make the changes in the affected systems before migrating CollectOSS to avoid breakage.

Table Names

What: Some table names were also updated as part of this version:

• operations.augur_settings is now known as operations.legacy_settings
• spdx.augur_repo_map is now known as spdx.repo_map

Why: The old project name was encoded directly into the table names. Unfortunately, it is not technically possible to configure CollectOSS to preserve the functioning of both names in order to provide a transition period.

Impact: Any applications connecting to CollectOSS that make queries involving the affected tables must be updated to use the new column names in their queries. This is a breaking change but the impact may likely be relatively small since both of these tables were either barely being used anyway, or were located inside a schema that we suspect has not been in active use for some time. If you depend on this behavior, you must make the changes in the affected systems before migrating CollectOSS to avoid breakage.

Other database Names

What: Other, more internal elements of the database schema were also renamed:

• Sequence spdx.augur_repo_map_map_id_seq was renamed to spdx.repo_map_map_id_seq
• Foreign Key Constraint data.pull_requests.pull_requests_pr_augur_contributor_id_fkey was renamed to data.pull_requests.pull_requests_contributor_id_fkey

Why: The old project name was encoded directly into the names of these elements. Unfortunately, it is not technically possible to configure CollectOSS to preserve the functioning of both names in order to provide a transition period.

Impact: Because these elements are generally more internal database structures (constraints and sequences), they are very unlikely to cause breakage. If you depend on this behavior, you must make the changes in the affected systems before migrating CollectOSS to avoid breakage.

(optional) Environment Variables

What: CollectOSS v1.1 has changed the name of its environment variables to prefer environment variables prefixed with COLLECTOSS_ rather than AUGUR_ by default. Any environment variables that don't have a prefix, such as CONFIG_DATADIR can remain the same.

Why: The old project name was encoded directly into most of the environment variable names that CollectOSS used for fetching configuration values. CollectOSS v1.1 introduces a new environment variable subsystem that allows the application to read from environment variables that are present with either the new COLLECTOSS_ or the old AUGUR_ prefixes (with the newer naming preferred if variables with both names are present. This change prepares the application for a future removal of support for the old naming scheme (the date for this is unplanned and will likely be in the far future).

Impact: This is a low-impact optional migration step that can be completed at any time. Because the new subsystem allows for ether name to be used, we recommend, but do not require you migrate to the new names at your convenience. Migration can happen one variable at a time independently of other migration steps if a slower transition is desired.

Config File Naming

What: CollectOSS v1.1 has changed the name of its primary configuration file from augur.json to collectoss.json. This config file, if used at all, should be located in the directory specified by the CONFIG_DATADIR environment variable.

Why: The old project name was encoded directly into the name of this configuration file. CollectOSS v1.1 introduces logic that can check for the presence of the old filename and read from it in the event that a file with the new name is not present. This change prepares the application for a future removal of support for the old naming scheme (the date for this is unplanned and will likely be in the far future).

Impact: This is a low-impact optional migration step that can be completed at any time. Because the new logic allows for a config file with either name to be used, we recommend (but do not require) that you rename your config file at your convenience.

Docker Entrypoints

RabbitMQ

What: CollectOSS v1.1 now relies on the upstream rabbitmq:4.1-management-alpine container image to provide the RabbitMQ service, rather than a custom container.

Why: The custom RabbitMQ image that was previously shipped was known to require the RabbitMQ credentials to be provided at build time (which were then baked into the image). This was incredibly insecure and required production deployments to rebuild the image with their own credentials, creating a large inconvenience.

Impact: Medium Impact. Changes are required to the rabbitmq service within the docker compose file, however these changes are not anticipated to cause breakage.

To migrate:

1. update your docker compose file to use the rabbitmq:4.1-management-alpine image for the rabbitmq service
2. ensure the environment variables are specified based on the docker-compose.yml in the repo, as well as your own production environment.
3. ensure the any configuration path mounts are specified based on the docker-compose.yml in the repo, as well as your own production environment.

Optional: Database rename

What: The default database name for new instances is now collectoss

Why: Prior to CollectOSS v1.1, augur was the conventionally chosen name for a new database. This name was also specified as the default for instances being created with Docker.

Impact: Low to Medium Impact. Making this change requires shutting down or disconnecting any applications connected to it and will therefore require some application downtime. This change is entirely optional but may be preferred by admins who would like the database name to reflect the name of the software that is primarily writing to it. It does not have to be completed at the same time as any other migration steps.

To rename your database:

1. Ensure you have an adequate backup. These instructions will not alter the contents of your CollectOSS database, but this process involves potentially creating and deleting a temporary database. Mistyping these commands may lead to data loss.
2. Stop CollectOSS but keep your database online (using a command like docker stop collectoss-redis-1 collectoss-core-1 collectoss-rabbitmq-1 collectoss-flower-1 collectoss-keyman-1 collectoss-core-1 if your docker compose file was started from a folder called "collectoss")
3. Ensure that all connections to the database are disconnected. Note that some may still be present from other applications
4. If you have another database on your PostgresQL instance, disconnect and reconnect using that database. You may need to create a new temporary database for this purpose. The following step will not work if you are connecting to the database that is being renamed.
5. Run ALTER DATABASE augur RENAME TO collectoss; (or choose your own new name)
6. Ensure that the new database name is updated in your docker container configuration for CollectOSS (via environment variables, specifically for the database, core, and flower services).
7. If you created a temporary database for an earlier step, you can now connect to the newly renamed database and drop the test database.
8. Start CollectOSS again (such as with docker start collectoss-redis-1 collectoss-core-1 collectoss-rabbitmq-1 collectoss-flower-1 collectoss-keyman-1 collectoss-core-1) to ensure that everything comes up the way you expect.
9. Ensure other downstream applications that connect to your database are updated with the new database name. Examples of these downstreams may include:
   • The collectoss-monitoring grafana dashboards (if in use)
   • 8Knot
   • any other downstream tools connecting to your database for analysis work (scripts, Jupyter notebooks, etc).