Skip to content

Commit dcdfaa2

Browse files
danieldidescaugustus-sourcegraph
danieldides
and
caugustus-sourcegraph
authored
Decoupled migration documentation (sourcegraph#29779)
* Decoupled migration documentation * doc: single migration container for all three db * doc: Add in extra information about k8s db migrate * doc: Update kubernetes db migration docs * docs: formatting compose operations Clarify process slightly * doc: DB migrations without codeinsights-db (sourcegraph#29878) * doc: docker db migration without codeinsights-db * doc: k8s db migration process without codeinsights * doc: remove depends_on from docs * doc: update manual database migration docs * doc: make the markdown processor happy * doc: more spaces for the markdown monster * Update doc/admin/install/docker-compose/operations.md Co-authored-by: Crystal Augustus <91073224+caugustus-sourcegraph@users.noreply.github.com> * Update doc/admin/install/docker-compose/operations.md Co-authored-by: Crystal Augustus <91073224+caugustus-sourcegraph@users.noreply.github.com> * doc: integrate suggestions from crystal - remove extra psql steps - add in link to dirty-db process - add in real migrator output Co-authored-by: Crystal Augustus <91073224+caugustus-sourcegraph@users.noreply.github.com>
1 parent 55c8faa commit dcdfaa2

File tree

2 files changed

+240
-0
lines changed

2 files changed

+240
-0
lines changed

‎doc/admin/install/docker-compose/operations.md

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,161 @@ docker-compose up -d
186186

187187
You can see what's changed in the [Sourcegraph changelog](../../../CHANGELOG.md).
188188

189+
### Database Migrations
190+
191+
> NOTE: This feature is only available in versions `3.36` and later
192+
193+
194+
The `frontend` container in the `docker-compose.yaml` file will automatically run on startup and migrate the databases if any changes are required, however administrators may wish to migrate their databases before upgrading the rest of the system when working with large databases. Sourcegraph guarantees database backward compatibility to the most recent minor point release so the database can safely be upgraded before the application code.
195+
196+
> NOTE: These values will work for a standard docker-compose deployment of Sourcegraph. If you've customized your deployment (e.g., using an external database service), you will have to modify the environment variables accordingly.
197+
198+
To execute the database migrations independently, run the following commands:
199+
200+
1. Check the current migration versions of all three databases:
201+
```bash
202+
# This will output the current migration version for the frontend db
203+
docker exec -it pgsql psql -U sg -c "SELECT * FROM schema_migrations;"
204+
205+
# This will output the current migration version for the codeintel db
206+
docker exec -it codeintel-db psql -U sg -c "SELECT * FROM codeintel_schema_migrations;"
207+
208+
# This will output the current migration version for the codeinsights db
209+
docker exec -it codeinsights-db psql -U postgres -c "SELECT * FROM codeinsights_schema_migrations;"
210+
```
211+
212+
Verify all databases return `f` for `dirty` and note the current version.
213+
214+
1. Start the migrations:
215+
216+
> NOTE: This script makes the assumption that the environment has all three databases enabled. If the configuration flag `DISABLE_CODE_INSIGHTS` is set and the `codeinsights-db` is unavailable, the `migrator` container will fail. Please see the [Migrating Without Code Insights](#migrating-without-code-insights) section below for more info.
217+
218+
```bash
219+
export SOURCEGRAPH_VERSION="The version you are upgrading to"
220+
docker run --rm --name migrator_$SOURCEGRAPH_VERSION \
221+
-e PGHOST='pgsql' \
222+
-e PGPORT='5432' \
223+
-e PGUSER='sg' \
224+
-e PGPASSWORD='sg' \
225+
-e PGDATABASE='sg' \
226+
-e PGSSLMODE='disable' \
227+
-e CODEINTEL_PGHOST='codeintel-db' \
228+
-e CODEINTEL_PGPORT='5432' \
229+
-e CODEINTEL_PGUSER='sg' \
230+
-e CODEINTEL_PGPASSWORD='sg' \
231+
-e CODEINTEL_PGDATABASE='sg' \
232+
-e CODEINTEL_PGSSLMODE='disable' \
233+
-e CODEINSIGHTS_PGHOST='codeinsights-db' \
234+
-e CODEINSIGHTS_PGPORT='5432' \
235+
-e CODEINSIGHTS_PGUSER='postgres' \
236+
-e CODEINSIGHTS_PGPASSWORD='password' \
237+
-e CODEINSIGHTS_PGDATABASE='postgres' \
238+
-e CODEINSIGHTS_PGSSLMODE='disable' \
239+
--network=docker-compose_sourcegraph \
240+
sourcegraph/migrator:$SOURCEGRAPH_VERSION \
241+
up
242+
```
243+
244+
1. Observe the output:
245+
246+
Run:
247+
```bash
248+
docker logs migrator_$SOURCEGRAPH_VERSION
249+
```
250+
251+
You should see output similar to:
252+
253+
```text
254+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=frontend version=1528395964 dirty=false
255+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=codeintel version=1000000030 dirty=false
256+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=codeinsights version=1000000024 dirty=false
257+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=frontend version=1528395964 dirty=false
258+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Upgrading schema" schema=frontend
259+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Running up migration" schema=frontend migrationID=1528395965
260+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Running up migration" schema=frontend migrationID=1528395966
261+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Running up migration" schema=frontend migrationID=1528395967
262+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Running up migration" schema=frontend migrationID=1528395968
263+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=codeintel version=1000000030 dirty=false
264+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Upgrading schema" schema=codeintel
265+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=codeinsights version=1000000024 dirty=false
266+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Upgrading schema" schema=codeinsights
267+
migrator | t=2022-01-26T03:14:35+0000 lvl=info msg="Running up migration" schema=codeinsights migrationID=1000000025
268+
migrator exited with code 0
269+
```
270+
271+
If you see an error message or any of the databases have been flagged as "dirty", please follow ["How to troubleshoot a dirty database"](../../../admin/how-to/dirty_database.md). A dirty database will not affect your ability to use Sourcegraph however it will need to be resolved to upgrade further. If you are unable to resolve the issues, contact support at <mailto:support@sourcegraph.com> for further assistance and provide the output of the three `psql` commands. Otherwise, you are now safe to upgrade Sourcegraph.
272+
273+
274+
#### Migrating Without Code Insights
275+
If the `DISABLE_CODE_INSIGHTS=true` feature flag is set in Sourcegraph and the `codeinsights-db` is unavailable to the `migrator` container, the standard migration process will fail. Follow these steps to execute migrations to the `frontend` and `codeintel` databases:
276+
277+
1. Run the following `psql` commands in the containers to log the current migration state of the database:
278+
279+
```bash
280+
# This will output the current migration version for the frontend db
281+
docker exec -it pgsql psql -U sg -c "SELECT * FROM schema_migrations;"
282+
283+
# This will output the current migration version for the codeintel db
284+
docker exec -it codeintel-db psql -U sg -c "SELECT * FROM codeintel_schema_migrations;"
285+
```
286+
287+
The output should look something like (version numbers likely will not match):
288+
```text
289+
290+
version | dirty
291+
------------+-------
292+
1000000024 | f
293+
(1 row)
294+
```
295+
296+
Verify all databases return `f` for `dirty` and note the current verison.
297+
298+
1. Run the `migrator` container to migrate the `frontend` database:
299+
300+
```bash
301+
export SOURCEGRAPH_VERSION="The version you are upgrading to"
302+
docker run --rm --name migrator_$SOURCEGRAPH_VERSION_frontend \
303+
-e PGHOST='pgsql' \
304+
-e PGPORT='5432' \
305+
-e PGUSER='sg' \
306+
-e PGPASSWORD='sg' \
307+
-e PGDATABASE='sg' \
308+
-e PGSSLMODE='disable' \
309+
-e CODEINTEL_PGHOST='codeintel-db' \
310+
-e CODEINTEL_PGPORT='5432' \
311+
-e CODEINTEL_PGUSER='sg' \
312+
-e CODEINTEL_PGPASSWORD='sg' \
313+
-e CODEINTEL_PGDATABASE='sg' \
314+
-e CODEINTEL_PGSSLMODE='disable' \
315+
--network=docker-compose_sourcegraph \
316+
sourcegraph/migrator:$SOURCEGRAPH_VERSION \
317+
up -db frontend
318+
```
319+
320+
Run the `migrator` container to migrate the `codeintel` database:
321+
```bash
322+
export SOURCEGRAPH_VERSION="The version you are upgrading to"
323+
docker run --rm --name migrator_$SOURCEGRAPH_VERSION_codeintel \
324+
-e PGHOST='pgsql' \
325+
-e PGPORT='5432' \
326+
-e PGUSER='sg' \
327+
-e PGPASSWORD='sg' \
328+
-e PGDATABASE='sg' \
329+
-e PGSSLMODE='disable' \
330+
-e CODEINTEL_PGHOST='codeintel-db' \
331+
-e CODEINTEL_PGPORT='5432' \
332+
-e CODEINTEL_PGUSER='sg' \
333+
-e CODEINTEL_PGPASSWORD='sg' \
334+
-e CODEINTEL_PGDATABASE='sg' \
335+
-e CODEINTEL_PGSSLMODE='disable' \
336+
--network=docker-compose_sourcegraph \
337+
sourcegraph/migrator:$SOURCEGRAPH_VERSION \
338+
up -db codeintel
339+
```
340+
341+
If you see an error message or any of the databases have been flagged as "dirty", please follow ["How to troubleshoot a dirty database"](../../../admin/how-to/dirty_database.md). A dirty database will not affect your ability to use Sourcegraph however it will need to be resolved to upgrade further. If you are unable to resolve the issues, contact support at <mailto:support@sourcegraph.com> for further assistance and provide the output of the three `psql` commands. Otherwise, you are now safe to upgrade Sourcegraph.
342+
343+
189344
## Monitoring
190345

191346
You can monitor the health of a deployment in several ways:

‎doc/admin/install/kubernetes/update.md

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,91 @@ the following:
7979
determine if an instance goes down.
8080
- Database migrations are handled automatically on update when they are necessary.
8181

82+
## Database Migrations
83+
84+
> NOTE: This feature is only available in versions `3.36` and later
85+
86+
By default, database migrations will be performed during application startup by the `frontend` application. These migrations _must_ succeed before Sourcegraph will become available. If the databases are large, these migrations may take a long time.
87+
88+
In some situations, administrators may wish to migrate their databases before upgrading the rest of the system to reduce downtime. Sourcegraph guarantees database backward compatibility to the most recent minor point release so the database can safely be upgraded before the application code.
89+
90+
To execute the database migrations independently, run the following commands in your fork of `deploy-sourcegraph`.
91+
92+
> NOTE: These values will work for a standard deployment of Sourcegraph with all three databases running in-cluster. If you've customized your deployment (e.g., using an external database service), you will have to modify the environment variables in `configure/migrator/migrator.Job.yaml` accordingly.
93+
94+
95+
1. Check the current migration versions of all three databases:
96+
```
97+
# This will output the current migration version for the frontend db
98+
kubectl exec $(kubectl get pod -l app=pgsql -o jsonpath="{.items[0].metadata.name}") -c pgsql -- psql -U sg -c "SELECT * FROM schema_migrations;"
99+
100+
101+
# This will output the current migration version for the codeintel db
102+
kubectl exec $(kubectl get pod -l app=codeintel-db -o jsonpath="{.items[0].metadata.name}") -c pgsql -- psql -U sg -c "SELECT * FROM codeintel_schema_migrations;"
103+
104+
105+
# This will output the current migration version for the codeinsights db
106+
kubectl exec $(kubectl get pod -l app=codeinsights-db -o jsonpath="{.items[0].metadata.name}") -- psql -U postgres -c "SELECT * FROM codeinsights_schema_migrations;"
107+
```
108+
109+
1. Start the migrations (run these commands from the root of your `deploy-sourcegraph` fork):
110+
111+
> NOTE: This script makes the assumption that the environment has all three databases enabled. If the configuration flag `DISABLE_CODE_INSIGHTS` is set and the `codeinsights-db` is unavailable, the `migrator` container will fail. Please see the [Migrating Without Code Insights](#migrating-without-code-insights) section below for more info.
112+
113+
```bash
114+
# Update the "image" value of the migrator container in the manifest
115+
export SOURCEGRAPH_VERSION="the version you're upgrading to"
116+
yq eval -i \
117+
'.spec.template.spec.containers[0].image = "index.docker.io/sourcegraph/migrator:" + strenv(SOURCEGRAPH_VERSION)' \
118+
configure/migrator/migrator.Job.yaml
119+
120+
# If you do not have yq, you can update the image tag manually to:
121+
# "index.docker.io/sourcegraph/migrator:$SOURCEGRAPH_VERSION"
122+
123+
# Apply and wait for migrations to complete before continuing
124+
kubectl delete -f configure/migrator/migrator.Job.yaml --ignore-not-found=true
125+
kubectl apply -f configure/migrator/migrator.Job.yaml
126+
# -1s timeout will wait "forever"
127+
kubectl wait -f configure/migrator/migrator.job.yaml --for=condition=complete --timeout=-1s
128+
```
129+
130+
You should see something like the following printed to the terminal:
131+
132+
```text
133+
job.batch "migrator" deleted
134+
job.batch/migrator created
135+
job.batch/migrator condition met
136+
```
137+
138+
The log output of the `migrator` container should look similar to:
139+
```
140+
t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=frontend version=1528395964 dirty=false
141+
t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=codeintel version=1000000030 dirty=false
142+
t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=codeinsights version=1000000024 dirty=false
143+
t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=frontend version=1528395964 dirty=false
144+
t=2022-01-26T03:14:35+0000 lvl=info msg="Upgrading schema" schema=frontend
145+
t=2022-01-26T03:14:35+0000 lvl=info msg="Running up migration" schema=frontend migrationID=1528395965
146+
t=2022-01-26T03:14:35+0000 lvl=info msg="Running up migration" schema=frontend migrationID=1528395966
147+
t=2022-01-26T03:14:35+0000 lvl=info msg="Running up migration" schema=frontend migrationID=1528395967
148+
t=2022-01-26T03:14:35+0000 lvl=info msg="Running up migration" schema=frontend migrationID=1528395968
149+
t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=codeintel version=1000000030 dirty=false
150+
t=2022-01-26T03:14:35+0000 lvl=info msg="Upgrading schema" schema=codeintel
151+
t=2022-01-26T03:14:35+0000 lvl=info msg="Checked current version" schema=codeinsights version=1000000024 dirty=false
152+
t=2022-01-26T03:14:35+0000 lvl=info msg="Upgrading schema" schema=codeinsights
153+
t=2022-01-26T03:14:35+0000 lvl=info msg="Running up migration" schema=codeinsights migrationID=1000000025
154+
```
155+
156+
If you see an error message or any of the databases have been flagged as "dirty", please follow ["How to troubleshoot a dirty database"](../../../admin/how-to/dirty_database.md). A dirty database will not affect your ability to use Sourcegraph however it will need to be resolved to upgrade further. If you are unable to resolve the issues, contact support at <mailto:support@sourcegraph.com> for further assistance and provide the output of the three `psql` commands. Otherwise, you are now safe to upgrade Sourcegraph.
157+
158+
### Migrating Without Code Insights
159+
If the `DISABLE_CODE_INSIGHTS=true` feature flag is set in Sourcegraph and the `codeinsights-db` is unavailable to the `migrator` container, the migration process will fail. To work around this, the `configure/migrator/migrator.Job.yaml` file will need to be updated. Please make the following changes to your fork of `deploy-sourcegraph`'s `migrator.Job.yaml` file.
160+
161+
1. Duplicate the `migrator` job manifest and rename one to `migrator-codeintel` and one to `migrator-frontend`.
162+
1. Modify the `migrator-codeintel` manifest to update the `spec.template.spec.containers[0].args` field to `["up", "-db", "codeintel"]`
163+
1. Modify the `migrator-frontend` manifest to update the `spec.template.spec.containers[0].args` field to `["up", "-db", "frontend"]`
164+
165+
You should now be able to apply both jobs and continue the migration and upgrade process as normal.
166+
82167
### Troubleshooting
83168
84169
See the [troubleshooting page](troubleshoot.md).

0 commit comments

Comments
 (0)