From b1d91da555b5d81fe0796ff97b08bd1e409db502 Mon Sep 17 00:00:00 2001 From: Eric Fritz Date: Tue, 20 Jun 2023 12:11:07 -0500 Subject: [PATCH 1/7] WIP. --- doc/admin/updates/index.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/admin/updates/index.md b/doc/admin/updates/index.md index eada60ac1b09..61f3af013ac5 100644 --- a/doc/admin/updates/index.md +++ b/doc/admin/updates/index.md @@ -1,5 +1,9 @@ # Updating sourcegraph +![queued](https://storage.googleapis.com/sourcegraph-assets/docs/images/upgrades/5.1/queued.png) +![failed](https://storage.googleapis.com/sourcegraph-assets/docs/images/upgrades/5.1/failed.png) +![oobmigrations](https://storage.googleapis.com/sourcegraph-assets/docs/images/upgrades/5.1/oobmigrations.png) + This page is intended as an entry point into Sourcegraph versioning, upgrades, and the `migrator` service which manages our database schemas. Here we'll cover general concepts and direct you toward relevant operations pages. **If you are already familiar with Sourcegraph upgrades [skip to instance specific procedures](#instance-specific-procedures).** From 64e420ed4cc0df8602bd899de415c36e4a6584b4 Mon Sep 17 00:00:00 2001 From: Eric Fritz Date: Tue, 20 Jun 2023 14:35:00 -0500 Subject: [PATCH 2/7] WIP. --- doc/admin/deploy/docker-compose/upgrade.md | 35 +++--- .../deploy/docker-single-container/index.md | 34 ++--- doc/admin/deploy/kubernetes/helm.md | 119 +++++++++--------- doc/admin/deploy/kubernetes/upgrade.md | 40 +++--- doc/admin/updates/automatic.md | 26 ++++ doc/admin/updates/index.md | 9 +- .../updates/migrator/migrator-operations.md | 103 +++++++-------- 7 files changed, 201 insertions(+), 165 deletions(-) create mode 100644 doc/admin/updates/automatic.md diff --git a/doc/admin/deploy/docker-compose/upgrade.md b/doc/admin/deploy/docker-compose/upgrade.md index 2d7dcc7bf827..76db2f82db41 100644 --- a/doc/admin/deploy/docker-compose/upgrade.md +++ b/doc/admin/deploy/docker-compose/upgrade.md @@ -10,7 +10,7 @@ A [standard upgrade](../../updates/index.md#upgrade-types) occurs between a Sour If you've [configured Docker Compose with a release branch](index.md#step-1-prepare-the-deployment-repository), please merge the upstream release tag for the next minor version into your `release` branch. In the following example, the release branch is being upgraded to v3.43.2. -```bash +```sh # first, checkout the release branch git checkout release # fetch updates @@ -34,7 +34,7 @@ For each conflict, you need to reconcile any customizations you made with the up SSH into your instance and navigate to the appropriate folder: -```bash +```sh # AWS cd /home/ec2-user/deploy-sourcegraph-docker/docker-compose # Azure, Digital Ocean, Google Cloud @@ -43,21 +43,25 @@ cd /root/deploy-sourcegraph-docker/docker-compose Download all the latest docker images to your local docker daemon: -```bash -docker-compose pull --include-deps +```sh +$ docker-compose pull --include-deps ``` Restart Docker Compose using the new minor version along with your customizations: -```bash -docker-compose up -d --remove-orphans +```sh +$ docker-compose up -d --remove-orphans ``` ### Multi-version upgrades +If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](/admin/updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). + +--- + > **⚠️ Attention:** please see our [cautionary note](../../updates/index.md#best-practices) on upgrades, if you have any concerns about running a multiversion upgrade, please reach out to us at [support@sourcegraph.com](emailto:support@sourcegraph.com) for advisement. -To perform a multi-version upgrade on a Sourcegraph instance running on Docker compose follow the procedure below: +To perform a **manual** multi-version upgrade on a Sourcegraph instance running on Docker compose follow the procedure below: 1. **Check Upgrade Readiness**: - Check the [upgrade notes](../../updates/docker_compose.md#docker-compose-upgrade-notes) for the version range you're passing through. @@ -65,16 +69,16 @@ To perform a multi-version upgrade on a Sourcegraph instance running on Docker c 2. **Disable Connections to the Database**: - Run the following command in the directory containing your `docker-compose.yaml` file. - ``` - docker-compose stop && docker-compose up -d pgsql codeintel-db codeinsights-db + ```sh + $ docker-compose stop && docker-compose up -d pgsql codeintel-db codeinsights-db ``` 3. **Run Migrator with the `upgrade` command**: - The following procedure describes running migrator in brief, for more detailed instructions and available command flags see our [migrator docs](../../updates/migrator/migrator-operations.md#docker-compose). 1. Set the migrator `image:` in your `docker-compose.yaml` to the **latest** release of `migrator`. **Example:** ```yaml - migrator: - container_name: migrator - image: 'index.docker.io/sourcegraph/migrator:5.0.4' + migrator: + container_name: migrator + image: 'index.docker.io/sourcegraph/migrator:5.0.4' ``` > *Note: Always use the latest image version of migrator for migrator commands, except the startup command `up`* 2. Set the migrator `command:` to `upgrade` you'll need to supply a `--to=` argument. **Example:** @@ -83,7 +87,7 @@ To perform a multi-version upgrade on a Sourcegraph instance running on Docker c ``` > *Note: you may add the `--dry-run` flag to the `command:` to test things out before altering the dbs* 3. Run migrator with `docker-compose up migrator` **Example:** - ```bash + ```sh $ ~/deploy-sourcegraph-docker/docker-compose/ docker-compose up migrator codeintel-db is up-to-date codeinsights-db is up-to-date @@ -116,7 +120,6 @@ To perform a multi-version upgrade on a Sourcegraph instance running on Docker c 5. **Start your containers again**: - run `docker-compose up -d` in the folder containing your `docker-compose.yaml` file. + ```sh + $ docker-compose up -d ``` - docker-compose up -d - ``` - diff --git a/doc/admin/deploy/docker-single-container/index.md b/doc/admin/deploy/docker-single-container/index.md index 1bfca9f01d42..017411df02bc 100644 --- a/doc/admin/deploy/docker-single-container/index.md +++ b/doc/admin/deploy/docker-single-container/index.md @@ -86,24 +86,24 @@ This is required to [collect debug data](../../pprof.md). The docker run command for single-container Sourcegraph needs an additional publish flag to expose the debug port: -```bash script -docker run --publish 7080:7080 --publish 127.0.0.1:3370:3370 --publish 127.0.0.1:6060:6060 --rm --volume ~/.sourcegraph/config:/etc/sourcegraph --volume ~/.sourcegraph/data:/var/opt/sourcegraph sourcegraph/server:5.0.6 +```sh +$ docker run --publish 7080:7080 --publish 127.0.0.1:3370:3370 --publish 127.0.0.1:6060:6060 --rm --volume ~/.sourcegraph/config:/etc/sourcegraph --volume ~/.sourcegraph/data:/var/opt/sourcegraph sourcegraph/server:5.0.6 ``` If Sourcegraph is deployed to a remote server, then access via an SSH tunnel using a tool such as [sshuttle](https://github.com/sshuttle/sshuttle) is required to establish a secure connection. To access the remote server using `sshuttle` from your local machine: -```bash script -sshuttle -r user@host 0/0 +```sh +$ sshuttle -r user@host 0/0 ``` ### Environment variables Add the following to your docker run command: -``` -docker run [...] +```sh +$ docker run [...] -e (YOUR CODE) sourcegraph/server:5.0.6 ``` @@ -116,16 +116,16 @@ sourcegraph/server:5.0.6 Get the Docker container ID for Sourcegraph: -```bash -docker ps +```sh +$ docker ps CONTAINER ID IMAGE d039ec989761 sourcegraph/server:VERSION ``` Open a PostgreSQL interactive terminal: -```bash -docker container exec -it d039ec989761 psql -U postgres sourcegraph +```sh +$ docker container exec -it d039ec989761 psql -U postgres sourcegraph ``` Run your SQL query: @@ -170,8 +170,8 @@ To perform a multi-version upgrade on a Sourcegraph instance running on Docker S

-```bash -docker run \ +```sh +$ docker run \ --rm \ --name migrator_${SG_VERSION} \ -e PGHOST='pgsql' \ @@ -209,8 +209,8 @@ Let `${PATH}` be the directory mounted into `/var/opt/sourcegraph` of your insta For example, `${PATH}` is `~/.sourcegraph/data` in `-v ~/.sourcegraph/data:/var/opt/sourcegraph`. -```bash -docker run --rm -it \ +```sh +$ docker run --rm -it \ -v ${PATH}/postgresql:/data/pgdata-${PG_VERSION} \ -u 70 \ -p 5432:5432 \ @@ -243,7 +243,7 @@ To achieve better performance, you can do any of the following: - For better clone performance, clone the repository on your host machine and then [add it to Sourcegraph Server](../../repo/add.md#add-repositories-already-cloned-to-disk). - Try adding the `:delegated` suffix the data volume mount. [Learn more](https://github.com/docker/for-mac/issues/1592). - ``` + ```sh --volume ~/.sourcegraph/data:/var/opt/sourcegraph:delegated ``` @@ -297,8 +297,8 @@ To test new development builds of Sourcegraph (triggered by commits to `main`), > WARNING: `insiders` builds may be unstable, so back up Sourcegraph's data and config (usually `~/.sourcegraph`) beforehand. -```bash -docker run --publish 7080:7080 --rm --volume ~/.sourcegraph/config:/etc/sourcegraph --volume ~/.sourcegraph/data:/var/opt/sourcegraph sourcegraph/server:insiders +```sh +$ docker run --publish 7080:7080 --rm --volume ~/.sourcegraph/config:/etc/sourcegraph --volume ~/.sourcegraph/data:/var/opt/sourcegraph sourcegraph/server:insiders ``` To keep this up to date, run `docker pull sourcegraph/server:insiders` to pull in the latest image, and restart the container to access new changes. diff --git a/doc/admin/deploy/kubernetes/helm.md b/doc/admin/deploy/kubernetes/helm.md index 7a96927f0ef2..3db66a39562b 100644 --- a/doc/admin/deploy/kubernetes/helm.md +++ b/doc/admin/deploy/kubernetes/helm.md @@ -48,13 +48,13 @@ Our Helm chart has a lot of sensible defaults baked into the values.yaml so that To use the Helm chart, add the Sourcegraph helm repository (on the machine used to interact with your cluster): ```sh -helm repo add sourcegraph https://helm.sourcegraph.com/release +$ helm repo add sourcegraph https://helm.sourcegraph.com/release ``` Install the Sourcegraph chart using default values: ```sh -helm install --version 5.0.6 sourcegraph sourcegraph/sourcegraph +$ helm install --version 5.0.6 sourcegraph sourcegraph/sourcegraph ``` Sourcegraph should now be available via the address set. Browsing to the url should now provide access to the Sourcegraph UI to create the initial administrator account. @@ -81,7 +81,7 @@ Example overrides can be found in the [examples](https://github.com/sourcegraph/ Providing the override file to Helm is done with the inclusion of the values flag and the name of the file: ```sh -helm upgrade --install --values ./override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph +$ helm upgrade --install --values ./override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph ``` When making configuration changes, it's recommended to review the changes that will be applied—see [Reviewing Changes](#reviewing-changes). @@ -246,7 +246,6 @@ data: In your [override.yaml](https://github.com/sourcegraph/deploy-sourcegraph-helm/tree/main/charts/sourcegraph/examples/external-object-storage/override.yaml), reference this Secret and the necessary environment variables: ```yaml - blobstore: enabled: false # Disable deployment of the built-in object storage @@ -343,9 +342,9 @@ Create a [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) tha If you have access to the ssh keys locally, you can run the command below to create the secret: ```sh -kubectl create secret generic gitserver-ssh \ - --from-file id_rsa=${HOME}/.ssh/id_rsa \ - --from-file known_hosts=${HOME}/.ssh/known_hosts +$ kubectl create secret generic gitserver-ssh \ + --from-file id_rsa=${HOME}/.ssh/id_rsa \ + --from-file known_hosts=${HOME}/.ssh/known_hosts ``` Alternatively, you may manually create the secret from a manifest file. @@ -353,7 +352,7 @@ Alternatively, you may manually create the secret from a manifest file. > WARNING: Do NOT commit the secret manifest into your Git repository unless you are okay with storing sensitive information in plaintext and your repository is private. Create a file with the following and save it as `gitserver-ssh.Secret.yaml` -```sh +```yaml apiVersion: v1 kind: Secret metadata: @@ -367,7 +366,7 @@ data: Apply the created [Secret] with the command below: ```sh -kubectl apply -f gitserver-ssh.Secret.yaml +$ kubectl apply -f gitserver-ssh.Secret.yaml ``` You should add the following values to your override file to reference the [Secret] you created earlier. @@ -522,7 +521,7 @@ This section is aimed at providing high-level guidance on deploying Sourcegraph 4. Have the [Helm CLI](https://helm.sh/docs/intro/install/) installed and run the following command to link to the Sourcegraph helm repository (on the machine used to interact with your cluster): ```sh -helm repo add sourcegraph https://helm.sourcegraph.com/release +$ helm repo add sourcegraph https://helm.sourcegraph.com/release ``` #### Steps {#gke-steps} @@ -576,13 +575,13 @@ The override file includes a [BackendConfig] CRD. This is necessary to instruct **2** – Install the chart ```sh -helm upgrade --install --values ./override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph +$ helm upgrade --install --values ./override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph ``` It will take around 10 minutes for the load balancer to be fully ready, you may check on the status and obtain the load balancer IP using the following command: ```sh -kubectl describe ingress sourcegraph-frontend +$ kubectl describe ingress sourcegraph-frontend ``` **3** – Upon obtaining the allocated IP address of the load balancer, you should create a DNS A record for the `sourcegraph.company.com` domain. Finally, it is recommended to enable TLS and you may consider using [Google-managed certificate](https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs) in GKE or your own certificate. @@ -618,7 +617,7 @@ data: ``` ```sh -kubectl apply -f ./sourcegraph-frontend-tls.Secret.yaml +$ kubectl apply -f ./sourcegraph-frontend-tls.Secret.yaml ``` Add the following values to your override file. @@ -656,7 +655,7 @@ Now the deployment is complete, more information on configuring the Sourcegraph 4. Have the [Helm CLI](https://helm.sh/docs/intro/install/) installed and run the following command to link to the Sourcegraph helm repository (on the machine used to interact with your cluster): ```sh -helm repo add sourcegraph https://helm.sourcegraph.com/release +$ helm repo add sourcegraph https://helm.sourcegraph.com/release ``` #### Steps {#eks-steps} @@ -695,13 +694,13 @@ storageClass: **2** – Install the chart ```sh -helm upgrade --install --values ./override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph +$ helm upgrade --install --values ./override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph ``` It will take some time for the load balancer to be fully ready, use the following to check on the status and obtain the load balancer address (once available): ```sh -kubectl describe ingress sourcegraph-frontend +$ kubectl describe ingress sourcegraph-frontend ``` **3** – Upon obtaining the allocated address of the load balancer, you should create a DNS record for the `sourcegraph.company.com` domain that resolves to the load balancer address. @@ -744,7 +743,7 @@ Now the deployment is complete, more information on configuring the Sourcegraph 4. Have the [Helm CLI](https://helm.sh/docs/intro/install/) installed and run the following command to link to the Sourcegraph helm repository (on the machine used to interact with your cluster): ```sh -helm repo add sourcegraph https://helm.sourcegraph.com/release +$ helm repo add sourcegraph https://helm.sourcegraph.com/release ``` #### Steps {#aks-steps} @@ -780,13 +779,13 @@ storageClass: **2** – Install the chart ```sh -helm upgrade --install --values ./override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph +$ helm upgrade --install --values ./override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph ``` It will take some time for the load balancer to be fully ready, you can check on the status and obtain the load balancer address (when ready) using: ```sh -kubectl describe ingress sourcegraph-frontend +$ kubectl describe ingress sourcegraph-frontend ``` **3** – Upon obtaining the allocated address of the load balancer, you should create a DNS record for the `sourcegraph.company.com` domain that resolves to the load balancer address. @@ -830,7 +829,7 @@ Now the deployment is complete, more information on configuring the Sourcegraph 4. Have the [Helm CLI](https://helm.sh/docs/intro/install/) installed and run the following command to link to the Sourcegraph helm repository (on the machine used to interact with your cluster): ```sh -helm repo add sourcegraph https://helm.sourcegraph.com/release +$ helm repo add sourcegraph https://helm.sourcegraph.com/release ``` #### Steps {#others-steps} @@ -866,13 +865,13 @@ storageClass: **2** – Install the chart ```sh -helm upgrade --install --values ./override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph +$ helm upgrade --install --values ./override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph ``` It may take some time before your ingress is up and ready to proceed. Depending on how your Ingress Controller works, you may be able to check on its status and obtain the public address of your Ingress using: ```sh -kubectl describe ingress sourcegraph-frontend +$ kubectl describe ingress sourcegraph-frontend ``` **3** – You should create a DNS record for the `sourcegraph.company.com` domain that resolves to the Ingress public address. @@ -914,7 +913,7 @@ data: ``` ```sh -kubectl apply -f ./sourcegraph-frontend-tls.Secret.yaml +$ kubectl apply -f ./sourcegraph-frontend-tls.Secret.yaml ``` Add the following values to your override file. @@ -957,28 +956,32 @@ A [standard upgrade](../../updates/index.md#upgrade-types) occurs between a Sour 2. Update your copy of the Sourcegraph Helm repo to ensure you have all the latest versions: -```bash -helm repo update sourcegraph +```sh +$ helm repo update sourcegraph ``` 3. (Optional) Review the changes that will be applied—see [Reviewing Changes](#reviewing-changes) for options. 4. Install the new version: -```bash -helm upgrade --install -f override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph +```sh +$ helm upgrade --install -f override.yaml --version 5.0.6 sourcegraph sourcegraph/sourcegraph ``` 5. Verify the installation has started: -```bash -kubectl get pods --watch +```sh +$ kubectl get pods --watch ``` When all pods have restarted and show as Running, you can browse to your Sourcegraph deployment and login to verify the instance is working as expected. For troubleshooting, refer to the [Operations guide](https://docs.sourcegraph.com/admin/install/kubernetes/operations) for common commands to gather more information about failures. ### Multi-version upgrades +If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](/admin/updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). + +--- + > **⚠️ Attention:** please see our [cautionary note](../../updates/index.md#best-practices) on upgrades, if you have any concerns about running a multiversion upgrade, please reach out to us at [support@sourcegraph.com](emailto:support@sourcegraph.com) for advisement. ### Multi-version upgrade procedure @@ -1004,25 +1007,25 @@ When all pods have restarted and show as Running, you can browse to your Sourceg The following convenience commands provide an example of scaling down the necessary services in a single command: Deployments: - ``` - kubectl get -n sourcegraph deploy --no-headers | awk '{print $1}' | xargs -n 1 -P 8 -I % kubectl -n sourcegraph scale deployment % --replicas=0 + ```sh + $ kubectl get -n sourcegraph deploy --no-headers | awk '{print $1}' | xargs -n 1 -P 8 -I % kubectl -n sourcegraph scale deployment % --replicas=0 ``` StatefulSets: - ``` - kubectl -n sourcegraph get sts --selector 'app.kubernetes.io/component!=codeinsights-db,app.kubernetes.io/component!=codeintel-db,app.kubernetes.io/component!=pgsql' --no-headers | awk '{print $1}' | xargs -n 1 -P 8 -I % kubectl -n sourcegraph scale sts % --replicas=0 + ```sh + $ kubectl -n sourcegraph get sts --selector 'app.kubernetes.io/component!=codeinsights-db,app.kubernetes.io/component!=codeintel-db,app.kubernetes.io/component!=pgsql' --no-headers | awk '{print $1}' | xargs -n 1 -P 8 -I % kubectl -n sourcegraph scale sts % --replicas=0 ``` > NOTE: The commands above use the `sourcegraph` namespace and are specific to the kubernetes-helm deployment. 3. **Run the migrator `upgrade` command** - The following command is the general template for running an upgrade - ``` - helm upgrade --install -n --set "migrator.args={upgrade,--from=,--to=}" sourcegraph-migrator sourcegraph/sourcegraph-migrator --version + ```sh + $ helm upgrade --install -n --set "migrator.args={upgrade,--from=,--to=}" sourcegraph-migrator sourcegraph/sourcegraph-migrator --version ``` > NOTE: The command above is general and you'll need to substitute in your own namespace, target sourcegraph version, and desired migrator image version. In general run the most recent version of migrator. **Example:** - ```bash - λ helm upgrade --install -n sourcegraph --set "migrator.args={upgrade,--from=3.41.0,--to=4.5.1}" sourcegraph-migrator sourcegraph/sourcegraph-migrator --version 5.0.6 + ```sh + $ helm upgrade --install -n sourcegraph --set "migrator.args={upgrade,--from=3.41.0,--to=4.5.1}" sourcegraph-migrator sourcegraph/sourcegraph-migrator --version 5.0.6 Release "sourcegraph-migrator" has been upgraded. Happy Helming! NAME: sourcegraph-migrator LAST DEPLOYED: Tue Mar 7 18:23:56 2023 @@ -1039,16 +1042,16 @@ When all pods have restarted and show as Running, you can browse to your Sourceg You can learn more about running migrator operations in helm in the [migrator operations doc](../../updates/migrator/migrator-operations.md#kubernetes-helm). 4. **Upgrade your instance via `helm upgrade`** - Now that the databases have been migrated to the latest versions, services can be scaled up and upgrade via the [standard procedure](#standard-upgrades). For example: - ``` - helm upgrade -n --install -f override.yaml --version sourcegraph sourcegraph/sourcegraph + ```sh + $ helm upgrade -n --install -f override.yaml --version sourcegraph sourcegraph/sourcegraph ``` ### Rollback You can revert to a previous version with the following command: -```bash -helm rollback sourcegraph +```sh +$ helm rollback sourcegraph ``` If you are rolling back more than a single version, then you must also [rollback your database](../../how-to/rollback_database.md), as database migrations (which may have run at some point during the upgrade) are guaranteed to be compatible with one previous minor version. @@ -1073,22 +1076,22 @@ For example: 1. Render the initial manifests from your existing deployment setup to an output file: -```bash -CHART_VERSION=0.7.0 # Currently deployed version -helm template sourcegraph -f override.yaml --version $CHART_VERSION sourcegraph sourcegraph/sourcegraph > original_manifests +```sh +$ CHART_VERSION=0.7.0 # Currently deployed version +$ helm template sourcegraph -f override.yaml --version $CHART_VERSION sourcegraph sourcegraph/sourcegraph > original_manifests ``` 2. Make changes to your override file, and/or update the chart version, then render that output: -```bash -CHART_VERSION=3.39.0 # Not yet deployed version -helm template sourcegraph -f override.yaml --version $CHART_VERSION sourcegraph sourcegraph/sourcegraph > new_manifests +```sh +$ CHART_VERSION=3.39.0 # Not yet deployed version +$ helm template sourcegraph -f override.yaml --version $CHART_VERSION sourcegraph sourcegraph/sourcegraph > new_manifests ``` 3. Compare the two outputs: -```bash -diff original_manifests new_manifests +```sh +$ diff original_manifests new_manifests ``` ### Using helm upgrade --dry-run @@ -1097,16 +1100,16 @@ Similar to `helm template`, the `helm upgrade --dry-run` command can be used to The following command will render and validate the manifests: -```bash - helm upgrade --install --dry-run -f override.yaml sourcegraph sourcegraph/sourcegraph +```sh +$ helm upgrade --install --dry-run -f override.yaml sourcegraph sourcegraph/sourcegraph ``` Any validation errors will be displayed instead of the rendered manifests. If you are having difficulty tracking down the cause of an issue, add the `--debug` flag to enable verbose logging: -```bash -helm upgrade --install --dry-run --debug -f override.yaml sourcegraph sourcegraph/sourcegraph +```sh +$ helm upgrade --install --dry-run --debug -f override.yaml sourcegraph sourcegraph/sourcegraph ``` The `--debug` flag will enable verbose logging and additional context, including the computed values used by the chart. This is useful when confirming your overrides have been interpreted correctly. @@ -1117,14 +1120,14 @@ The [Helm Diff] plugin can provide a diff against a deployed chart. It is simila To install the plugin, run: -```bash -helm plugin install https://github.com/databus23/helm-diff +```sh +$ helm plugin install https://github.com/databus23/helm-diff ``` Then, display a diff between a live deployment and an upgrade, with 5 lines of context: -```bash -helm diff upgrade -f override.yaml sourcegraph sourcegraph/sourcegraph -C 5 +```sh +$ helm diff upgrade -f override.yaml sourcegraph sourcegraph/sourcegraph -C 5 ``` For more examples and configuration options, reference the [Helm Diff] plugin documentation. @@ -1134,7 +1137,7 @@ For more examples and configuration options, reference the [Helm Diff] plugin do Sourcegraph can be uninstalled by running the following command: ```sh -helm uninstall sourcegraph +$ helm uninstall sourcegraph ``` Some Persistent Volumes may be retained after the uninstall is complete. In your cloud provider, check for unattached disks and delete them as necessary. diff --git a/doc/admin/deploy/kubernetes/upgrade.md b/doc/admin/deploy/kubernetes/upgrade.md index ddd7889075ed..1e826028d5d0 100644 --- a/doc/admin/deploy/kubernetes/upgrade.md +++ b/doc/admin/deploy/kubernetes/upgrade.md @@ -30,7 +30,7 @@ If the Sourcegraph upgrade fails, you can redeploy using the current `cluster.ya **Step 2**: Merge the new version of Sourcegraph into your release branch. - ```bash + ```sh cd $DEPLOY_SOURCEGRAPH_FORK # get updates git fetch upstream @@ -46,13 +46,13 @@ If the Sourcegraph upgrade fails, you can redeploy using the current `cluster.ya Generate a new set of manifests locally using your current overlay `instances/$INSTANCE_NAME` (e.g. INSTANCE_NAME=my-sourcegraph) without applying to the cluster. - ```bash + ```sh $ kubectl kustomize instances/my-sourcegraph -o cluster.yaml ``` Review the generated manifests to ensure they match your intended configuration and have the images for the `$NEW_VERSION` version. - ```bash + ```sh $ less cluster.yaml ``` @@ -62,7 +62,7 @@ Review the generated manifests to ensure they match your intended configuration Apply the new manifests from the ouput file `cluster.yaml` to your cluster: - ```bash + ```sh $ kubectl apply --prune -l deploy=sourcegraph -f cluster.yaml ``` @@ -70,7 +70,7 @@ Apply the new manifests from the ouput file `cluster.yaml` to your cluster: **Step 5**: Monitor the status of the deployment to determine its success. - ```bash + ```sh $ kubectl get pods -o wide --watch ``` @@ -82,7 +82,7 @@ The following procedure is for performing a **standard upgrade** with Sourcegrap **Step 1**: Merge the new version of Sourcegraph into your release branch. - ```bash + ```sh cd $DEPLOY_SOURCEGRAPH_FORK # get updates git fetch upstream @@ -106,13 +106,13 @@ For example, if you use [overlays to make changes to the manifests](https://gith **Step 3**: Apply the updates to your cluster. - ```bash + ```sh $ ./kubectl-apply-all.sh ``` **Step 4**: Monitor the status of the deployment to determine its success. - ```bash + ```sh $ kubectl get pods -o wide --watch ``` @@ -120,9 +120,13 @@ For example, if you use [overlays to make changes to the manifests](https://gith ## Multi-version upgrades +If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](/admin/updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). + +--- + > **⚠️ Attention:** please see our [cautionary note](../../updates/index.md#best-practices) on upgrades, if you have any concerns about running a multiversion upgrade, please reach out to us at [support@sourcegraph.com](emailto:support@sourcegraph.com) for advisement. -To perform a multi-version upgrade on a Sourcegraph instance running on our **kubernetes** repo follow the procedure below: +To perform a **manual** multi-version upgrade on a Sourcegraph instance running on our **kubernetes** repo follow the procedure below: 1. **Check Upgrade Readiness**: - Check the [upgrade notes](../../updates/kubernetes.md#kubernetes-upgrade-notes) for the version range you're passing through. @@ -143,8 +147,8 @@ To perform a multi-version upgrade on a Sourcegraph instance running on our **ku - indexed-search *convenience command:* - ``` - kubectl scale deployment precise-code-intel-worker repo-updater searcher sourcegraph-frontend sourcegraph-frontend-internal symbols worker --replicas=0 && kubectl scale sts gitserver indexed-search --replicas=0 + ```sh + $ kubectl scale deployment precise-code-intel-worker repo-updater searcher sourcegraph-frontend sourcegraph-frontend-internal symbols worker --replicas=0 && kubectl scale sts gitserver indexed-search --replicas=0 ``` 3. **Run Migrator with the `upgrade` command**: @@ -162,7 +166,7 @@ To perform a multi-version upgrade on a Sourcegraph instance running on our **ku > - *Always use the latest image version of migrator for migrator commands, except the startup command `up`* > - *You may add the `--dry-run` flag to the `command:` to test things out before altering the dbs* 2. Run the following commands to schedule the migrator job with the upgrade command and monitor its progress: - ```bash + ```sh # To ensure no previous job invocations will conflict with our current invocation kubectl delete -f configure/migrator/migrator.Job.yaml # Start the migrator job @@ -171,7 +175,7 @@ To perform a multi-version upgrade on a Sourcegraph instance running on our **ku kubectl logs job.batch/migrator -f ``` **Example:** - ```bash + ```sh $ kubectl -n sourcegraph apply -f ./migrator.Job.yaml job.batch/migrator created $ kubectl -n sourcegraph get jobs @@ -236,13 +240,13 @@ If you are rolling back more than a single version, then you must also [rollback For instances deployed using the [deploy-sourcegraph-k8s](https://github.com/sourcegraph/deploy-sourcegraph-k8s) repository: - ```bash + ```sh # Re-generate manifests - $ kubectl kustomize instances/$YOUR_INSTANCE -o cluster-rollback.yaml + kubectl kustomize instances/$YOUR_INSTANCE -o cluster-rollback.yaml # Review manifests - $ less cluster-rollback.yaml + less cluster-rollback.yaml # Re-deploy - $ kubectl apply --prune -l deploy=sourcegraph -f cluster-rollback.yaml + kubectl apply --prune -l deploy=sourcegraph -f cluster-rollback.yaml ``` ### Rollback without Kustomize @@ -251,7 +255,7 @@ For instances deployed using the [deploy-sourcegraph-k8s](https://github.com/sou For instances deployed using the old [deploy-sourcegraph](https://github.com/sourcegraph/deploy-sourcegraph) repository: - ```bash + ```sh $ ./kubectl-apply-all.sh ``` diff --git a/doc/admin/updates/automatic.md b/doc/admin/updates/automatic.md new file mode 100644 index 000000000000..7ebe0448028b --- /dev/null +++ b/doc/admin/updates/automatic.md @@ -0,0 +1,26 @@ +# Automatic multi-version upgrades + +From **Sourcegraph 5.1 and later**, multi-version upgrades can be performed **automatically** as if they were a standard upgrade for the same deployment type. Automatic multi-version upgrades take the following general form: + +1. Determine if your instance is ready to Upgrade (check upgrade notes) +2. Merge the latest Sourcegraph release into your deployment manifests +3. With upstream changes to your manifests merged, start the new instance + +The upgrade magic now happens when the new version is booted. In more detail, starting a new `frontend` container will: + +1. Detect that a previous version of Sourcegraph was/is currently running +2. Plan the and persist the upgrade (which migrations to apply) based on the previously running version, the new target version, and the database state +3. Start a new internal server that sends poison pills to disconnect old services from the databases and prevents new services from connecting before the upgrade completes +4. Start a status server in place of the primary exposed port +5. Runs the migration plan (performing the same steps as `migration upgrade ...` ) +6. Shuts down the internal and status servers and continues to boot normally + +## Viewing progress + +During an automatic multi-version upgrade, we'll attempt to boot a status serve in the frontend container that is running (or blocking) on an active upgrade attempt. If there is an upgrade failure that affects the frontend, this status page will not be available and the `frontend` container logs should be viewed. Optimistically, the status server will also be unreachable in the case that an upgrade performs quickly enough that there's no time for the status server to start. + +In the case that there's a migration failure, or an unfinished out-of-band migration that needs to be complete, the status server will be served instead of the normal Sourcegraph React app. The following screenshots show an upgrade from Sourcegraph v3.37.1 to Sourcegraph 5.0, in which the `frontend` schema is applying (or waiting to apply) a set of schema migrations, the `codeintel` schema has a pair of schema migration failures, and a single unfinished out-of-band migration is still actively being performed to completion. + +![An example in-progress upgrade with schema migrations queued for application](https://storage.googleapis.com/sourcegraph-assets/docs/images/upgrades/5.1/queued.png) +![An example in-progress upgrade with a few schema migration failures](https://storage.googleapis.com/sourcegraph-assets/docs/images/upgrades/5.1/failed.png) +![An example in-progress upgrade with unfinished out-of-band migrations](https://storage.googleapis.com/sourcegraph-assets/docs/images/upgrades/5.1/oobmigrations.png) diff --git a/doc/admin/updates/index.md b/doc/admin/updates/index.md index 61f3af013ac5..32f07c9d6572 100644 --- a/doc/admin/updates/index.md +++ b/doc/admin/updates/index.md @@ -30,7 +30,7 @@ To learn more about our release schedule see our [handbook](https://handbook.sou ### Upgrade types -Sourcegraph has two upgrade types. **Standard** upgrades and **Multiversion** upgrades. We generally recommend standard upgrades. +Sourcegraph has two upgrade types. **Standard** upgrades and **Multiversion** upgrades. **Standard** - Moves Sourcegraph one version forward (`v5.0.0` to `v5.1.0`), *this is usually one minor version unless the next version released is a major version*. @@ -38,8 +38,7 @@ Sourcegraph has two upgrade types. **Standard** upgrades and **Multiversion** up **Multiversion** - Moves Sourcegraph multiple versions forward (`v5.0.0` to `v5.2.0`). -- Requires manual `migrator` operations -- Requires downtime +- Requires downtime while the database schemas and rewritten and unfinished out-of-band migrations are applied - We currently support jumping from version `v3.20` or later to any future version. - **Sourcegraph App and AMIs do not yet support multiversion upgrades. We hope to improve this soon.** @@ -78,8 +77,8 @@ It is highly recommended to: Sourcegraph upgrades take the following general form: 1. Determine if your instance is ready to Upgrade (check upgrade notes) 2. Merge the latest Sourcegraph release into your deployment manifests -3. Run migrator by reapplying your manifests in a **standard upgrade** (migrator by default uses the `up` command), or by disabling services connected to your databases and running `migrator` with the `upgrade` argument in a **multiversion upgrade**. -4. With upstream changes to your manifests merged, and schema migrations conducted by migrator your instance is ready to start. +3. If updating more than a single minor version, perform an [**automatic multi-version upgrade**](./automatic.md) if targeting **Sourcegraph 5.1 or later**; [manual multi-verison upgrades](./migrator/migrator-operations.md) are required if upgrading to an earlier version, which requires shutting off the instance and invoking the `migrator` container or job to perform the database rewrite and application of unfinished out-of-band migrations +4. With upstream changes to your manifests merged, start the new instance > Note: For more explicit steps, specific to your deployment see the operations guides linked below. diff --git a/doc/admin/updates/migrator/migrator-operations.md b/doc/admin/updates/migrator/migrator-operations.md index 887c83564d0f..ea6b5ede2c86 100644 --- a/doc/admin/updates/migrator/migrator-operations.md +++ b/doc/admin/updates/migrator/migrator-operations.md @@ -42,25 +42,25 @@ Manifest loactions: Example default environment variables: ```yaml - PGHOST: "pgsql" - PGPORT: "5432" - PGUSER: "sg" - PGPASSWORD: "sg" - PGDATABASE: "sg" - PGSSLMODE: "disable" - CODEINTEL_PGHOST: "codeintel-db" - CODEINTEL_PGPORT: "5432" - CODEINTEL_PGUSER: "sg" - CODEINTEL_PGPASSWORD: "sg" - CODEINTEL_PGDATABASE: "sg" - CODEINTEL_PGSSLMODE: "disable" - CODEINSIGHTS_PGHOST: "codeinsights-db" - CODEINSIGHTS_PGPORT: "5432" - CODEINSIGHTS_PGUSER: "postgres" - CODEINSIGHTS_PGPASSWORD: "password" - CODEINSIGHTS_PGDATABASE: "postgres" - CODEINSIGHTS_PGSSLMODE: "disable" - ``` +PGHOST: "pgsql" +PGPORT: "5432" +PGUSER: "sg" +PGPASSWORD: "sg" +PGDATABASE: "sg" +PGSSLMODE: "disable" +CODEINTEL_PGHOST: "codeintel-db" +CODEINTEL_PGPORT: "5432" +CODEINTEL_PGUSER: "sg" +CODEINTEL_PGPASSWORD: "sg" +CODEINTEL_PGDATABASE: "sg" +CODEINTEL_PGSSLMODE: "disable" +CODEINSIGHTS_PGHOST: "codeinsights-db" +CODEINSIGHTS_PGPORT: "5432" +CODEINSIGHTS_PGUSER: "postgres" +CODEINSIGHTS_PGPASSWORD: "password" +CODEINSIGHTS_PGDATABASE: "postgres" +CODEINSIGHTS_PGSSLMODE: "disable" +``` ## Commands @@ -70,7 +70,7 @@ The `migrator` service exposes the following commands: The `upgrade` command performs database schema migrations and out-of-band migrations to rewrite existing instance data in-place into the shaped expected by a given target Sourcegraph version. This command is used by site-administrators to perform [multi-version upgrades](../index.md#upgrade-types). -``` +```sh upgrade \ --from= --to= \ [--dry-run=false] \ @@ -104,7 +104,7 @@ upgrade \ The `drift` command describes the current (live) database schema and compares it against the expected schema at the given version. The output of this command will include all relevant schema differences that could affect application correctness and performance. When schema drift is detected, a diff of the expected and actual Postgres object definitions will be shown, along with instructions on how to manually resolve the disparity. [Learn more here.](./schema-drift.md) -``` +```sh drift \ --db= \ [--version=] \ @@ -128,7 +128,7 @@ Exactly one of `--version` and `--file` must be supplied. The `downgrade` command performs database schema migrations and (reverse-applied) out-of-band migrations to rewrite existing instance data in-place into the shaped expected by a given target Sourcegraph version. -``` +```sh downgrade \ --from= --to= \ [--dry-run=false] \ @@ -161,7 +161,7 @@ downgrade \ The `add-log` command adds an entry to the `migration_logs` table after a site administrator has explicitly applied the contents of a migration definition. This command may be performed by a site-administrator as part of [repairing a dirty database](../../how-to/dirty_database.md#3-add-a-migration-log-entry). -``` +```sh add-log \ --db= \ --version= \ @@ -181,7 +181,7 @@ add-log \ The `validate` command validates the current state of the database (both schema and data migration progress). This command is used on Sourcegraph instance startup of database-dependent services to ensure that the migrator has been run to the expected version. -``` +```sh validate \ [--db=all] \ [--skip-out-of-band-migrations=false] @@ -204,7 +204,7 @@ The `up` command (the default behavior of the `migrator` service) applies all mi Users should generally prefer the command [`upto`](#upto), which accepts more explicit bounds and does not depend on the migrator compilation version. -``` +```sh up \ [--db=all] \ [--skip-upgrade-validation=false] \ @@ -229,7 +229,7 @@ up \ The `run-out-of-band-migrations` command runs out-of-band migrations within the `migrator`. This command may be performed by a site-administrator as part of [repairing an unfinished migration](../../how-to/unfinished_migration.md). -``` +```sh run-out-of-band-migrations \ [--id ]+ \ [--apply-reverse=false] \ @@ -249,7 +249,7 @@ run-out-of-band-migrations \ The `describe` command outputs a dump of your database schema. -``` +```sh describe \ --db= \ --format= \ @@ -293,16 +293,17 @@ To run `migrator` with a specific *command*: For example here the `upgrade` command: ```yaml - containers: - - name: migrator - image: "index.docker.io/sourcegraph/migrator:5.0.3" - args: ["upgrade", "--from=v3.41.0", "--to=v4.5.1"] - envFrom: +containers: + - name: migrator + image: "index.docker.io/sourcegraph/migrator:5.0.3" + args: ["upgrade", "--from=v3.41.0", "--to=v4.5.1"] + envFrom: ``` 2. Apply the job and wait for it to complete. -```bash +```sh +# Delete previous job (if one exists) kubectl delete -f configure/migrator/migrator.Job.yaml --ignore-not-found=true # Apply the manifest and wait for the operation to complete before continuing @@ -311,8 +312,8 @@ kubectl apply -f configure/migrator/migrator.Job.yaml 3. Check the logs from execution -```bash -kubectl logs job.batch/migrator -f +```sh +$ kubectl logs job.batch/migrator -f ``` ### Kubernetes Helm @@ -322,7 +323,7 @@ Running migrator operations in helm takes advantage of the [sourcegraph-migrator Generally the migrator is run via the `helm upgrade`, for example: ``` -helm upgrade --install -n sourcegraph --set "migrator.args={drift,--db=frontend,--version=v3.39.1}" sourcegraph-migrator sourcegraph/sourcegraph-migrator --version 4.4.2 +$ helm upgrade --install -n sourcegraph --set "migrator.args={drift,--db=frontend,--version=v3.39.1}" sourcegraph-migrator sourcegraph/sourcegraph-migrator --version 4.4.2 ``` In the example above the `drift` operation is run with flags `--db` and `--version`. The migrator is run using image version `v4.4.2`. @@ -332,7 +333,7 @@ Arguments are set with the `--set "migrator.args={operation-arg,flag-arg-1,flag- In the most general form running operations follows this template: ``` -helm upgrade --install -n --set "migrator.args={,,}" sourcegraph-migrator sourcegraph/sourcegraph-migrator --version +$ helm upgrade --install -n --set "migrator.args={,,}" sourcegraph-migrator sourcegraph/sourcegraph-migrator --version ``` > NOTE: You can troubleshoot a migrators operations with the command `kubectl -n sourcegraph logs -l job=migrator -f`. This will show you logs from the migrator jobs operation steps. @@ -347,24 +348,24 @@ To run `migrator` with a specific *command*: For example here the `upgrade` command: ```yaml - migrator: - container_name: migrator - image: 'index.docker.io/sourcegraph/migrator:5.0.3' - cpus: 0.5 - mem_limit: '500m' - command: ['upgrade', '--from=v3.41.0', '--to=v4.5.1'] - environment: +migrator: + container_name: migrator + image: 'index.docker.io/sourcegraph/migrator:5.0.3' + cpus: 0.5 + mem_limit: '500m' + command: ['upgrade', '--from=v3.41.0', '--to=v4.5.1'] + environment: ``` 2. Apply the job and wait for it to complete. -```bash -docker-compose up -d migrator +```sh +$ docker-compose up -d migrator ``` 3. The log output of `migrator` can be checked with the command -```bash -docker logs migrator +```sh +$ docker logs migrator ``` > Note: Remember to set the `command:` back to `up` and `image:` back to your deployment version if you are starting Souregraph again. @@ -375,7 +376,7 @@ Run the following commands on your Docker host. > NOTE: These values will work for a standard docker-compose deployment of Sourcegraph. If you've customized your deployment (e.g., using an external database service), you will have to modify the environment variables accordingly. -```bash +```sh export MIGRATOR_SOURCEGRAPH_VERSION="..." docker run \ @@ -407,8 +408,8 @@ docker run \ Observe the output of the `migrator` container via: -```bash -docker logs migrator_$SOURCEGRAPH_VERSION +```sh +$ docker logs migrator_$SOURCEGRAPH_VERSION ``` The log output of the `migrator` should include `INFO`-level logs and successfully terminate with `migrator exited with code 0`. If you see an error message or any of the databases have been flagged as "dirty", please follow ["How to troubleshoot a dirty database"](../../../admin/how-to/dirty_database.md). A dirty database will not affect your ability to use Sourcegraph however it will need to be resolved to upgrade further. If you are unable to resolve the issues, contact support at for further assistance. Otherwise, you are now safe to upgrade Sourcegraph. From f12b7b3db12b18a2f6a108e11c407fb9f1b0e134 Mon Sep 17 00:00:00 2001 From: Eric Fritz Date: Tue, 20 Jun 2023 14:35:43 -0500 Subject: [PATCH 3/7] WIP. --- doc/admin/updates/index.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/doc/admin/updates/index.md b/doc/admin/updates/index.md index 32f07c9d6572..cb35ba908dd9 100644 --- a/doc/admin/updates/index.md +++ b/doc/admin/updates/index.md @@ -1,9 +1,5 @@ # Updating sourcegraph -![queued](https://storage.googleapis.com/sourcegraph-assets/docs/images/upgrades/5.1/queued.png) -![failed](https://storage.googleapis.com/sourcegraph-assets/docs/images/upgrades/5.1/failed.png) -![oobmigrations](https://storage.googleapis.com/sourcegraph-assets/docs/images/upgrades/5.1/oobmigrations.png) - This page is intended as an entry point into Sourcegraph versioning, upgrades, and the `migrator` service which manages our database schemas. Here we'll cover general concepts and direct you toward relevant operations pages. **If you are already familiar with Sourcegraph upgrades [skip to instance specific procedures](#instance-specific-procedures).** From 1a1c36387cd27275c72c79da0accdeaf6e129e72 Mon Sep 17 00:00:00 2001 From: Eric Fritz Date: Mon, 26 Jun 2023 08:30:26 -0500 Subject: [PATCH 4/7] Apply suggestions from code review Co-authored-by: Noah S-C --- doc/admin/updates/automatic.md | 7 +++++-- doc/admin/updates/index.md | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/admin/updates/automatic.md b/doc/admin/updates/automatic.md index 7ebe0448028b..686f42bd1141 100644 --- a/doc/admin/updates/automatic.md +++ b/doc/admin/updates/automatic.md @@ -4,7 +4,10 @@ From **Sourcegraph 5.1 and later**, multi-version upgrades can be performed **au 1. Determine if your instance is ready to Upgrade (check upgrade notes) 2. Merge the latest Sourcegraph release into your deployment manifests -3. With upstream changes to your manifests merged, start the new instance +3. Enable automatic upgrade for this upgrade by either: + a. In the Updates section in site-admin (available on instances of Sourcegraph 5.1 and later), or + b. set the `SRC_AUTOUPGRADE` environment variable to `true` on the migrator and frontend 5.1 or later deployment manifests (if the instance is of a version earlier than Sourcegraph 5.1) +4. With upstream changes to your manifests merged, start the new instance The upgrade magic now happens when the new version is booted. In more detail, starting a new `frontend` container will: @@ -17,7 +20,7 @@ The upgrade magic now happens when the new version is booted. In more detail, st ## Viewing progress -During an automatic multi-version upgrade, we'll attempt to boot a status serve in the frontend container that is running (or blocking) on an active upgrade attempt. If there is an upgrade failure that affects the frontend, this status page will not be available and the `frontend` container logs should be viewed. Optimistically, the status server will also be unreachable in the case that an upgrade performs quickly enough that there's no time for the status server to start. +During an automatic multi-version upgrade, we'll attempt to boot a status server in the frontend container that is running (or blocking on) an active upgrade attempt. If there is an upgrade failure that affects the frontend, this status page will not be available and the `frontend` container logs should be viewed. Optimistically, the status server will also be unreachable in the case that an upgrade performs quickly enough that there's no time for the status server to start. In the case that there's a migration failure, or an unfinished out-of-band migration that needs to be complete, the status server will be served instead of the normal Sourcegraph React app. The following screenshots show an upgrade from Sourcegraph v3.37.1 to Sourcegraph 5.0, in which the `frontend` schema is applying (or waiting to apply) a set of schema migrations, the `codeintel` schema has a pair of schema migration failures, and a single unfinished out-of-band migration is still actively being performed to completion. diff --git a/doc/admin/updates/index.md b/doc/admin/updates/index.md index cb35ba908dd9..41db64f92397 100644 --- a/doc/admin/updates/index.md +++ b/doc/admin/updates/index.md @@ -34,7 +34,7 @@ Sourcegraph has two upgrade types. **Standard** upgrades and **Multiversion** up **Multiversion** - Moves Sourcegraph multiple versions forward (`v5.0.0` to `v5.2.0`). -- Requires downtime while the database schemas and rewritten and unfinished out-of-band migrations are applied +- Requires downtime while the database schemas are rewritten and unfinished out-of-band migrations are applied - We currently support jumping from version `v3.20` or later to any future version. - **Sourcegraph App and AMIs do not yet support multiversion upgrades. We hope to improve this soon.** From 9adae24fd90513281ff7e3cb782702701512aa01 Mon Sep 17 00:00:00 2001 From: Eric Fritz Date: Mon, 26 Jun 2023 08:32:25 -0500 Subject: [PATCH 5/7] Add note about drift --- doc/admin/updates/automatic.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/admin/updates/automatic.md b/doc/admin/updates/automatic.md index 686f42bd1141..01e90e1965b6 100644 --- a/doc/admin/updates/automatic.md +++ b/doc/admin/updates/automatic.md @@ -18,6 +18,8 @@ The upgrade magic now happens when the new version is booted. In more detail, st 5. Runs the migration plan (performing the same steps as `migration upgrade ...` ) 6. Shuts down the internal and status servers and continues to boot normally +Note that if you have unresolved schema drift, the upgrade will refuse to continue to avoid future migration failures which are more difficult to resolve. Drift should be resolved prior to the beginning of an upgrade. Drift can also be explicitly ignored (which might be required when an upgrade is *resumed* after a failure) by setting the `SRC_AUTOUPGRADE_IGNORE_DRIFT` envvar to true on the migrator and frontend containers. + ## Viewing progress During an automatic multi-version upgrade, we'll attempt to boot a status server in the frontend container that is running (or blocking on) an active upgrade attempt. If there is an upgrade failure that affects the frontend, this status page will not be available and the `frontend` container logs should be viewed. Optimistically, the status server will also be unreachable in the case that an upgrade performs quickly enough that there's no time for the status server to start. From eb7f9caf18c8dc71938ba6d3848288a05fb644e8 Mon Sep 17 00:00:00 2001 From: Eric Fritz Date: Mon, 26 Jun 2023 08:42:41 -0500 Subject: [PATCH 6/7] Fix links. --- doc/admin/deploy/docker-compose/upgrade.md | 2 +- doc/admin/deploy/kubernetes/helm.md | 2 +- doc/admin/deploy/kubernetes/upgrade.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/admin/deploy/docker-compose/upgrade.md b/doc/admin/deploy/docker-compose/upgrade.md index 76db2f82db41..33efa6440897 100644 --- a/doc/admin/deploy/docker-compose/upgrade.md +++ b/doc/admin/deploy/docker-compose/upgrade.md @@ -55,7 +55,7 @@ $ docker-compose up -d --remove-orphans ### Multi-version upgrades -If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](/admin/updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). +If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](/../../updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). --- diff --git a/doc/admin/deploy/kubernetes/helm.md b/doc/admin/deploy/kubernetes/helm.md index 3db66a39562b..3baa0174e53d 100644 --- a/doc/admin/deploy/kubernetes/helm.md +++ b/doc/admin/deploy/kubernetes/helm.md @@ -978,7 +978,7 @@ When all pods have restarted and show as Running, you can browse to your Sourceg ### Multi-version upgrades -If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](/admin/updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). +If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](/../../updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). --- diff --git a/doc/admin/deploy/kubernetes/upgrade.md b/doc/admin/deploy/kubernetes/upgrade.md index 1e826028d5d0..48390959291f 100644 --- a/doc/admin/deploy/kubernetes/upgrade.md +++ b/doc/admin/deploy/kubernetes/upgrade.md @@ -120,7 +120,7 @@ For example, if you use [overlays to make changes to the manifests](https://gith ## Multi-version upgrades -If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](/admin/updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). +If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](../../updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). --- From f0f5abbe082796703294de6644227c661567dcd5 Mon Sep 17 00:00:00 2001 From: Eric Fritz Date: Mon, 26 Jun 2023 08:55:06 -0500 Subject: [PATCH 7/7] Fix links (take 2). --- doc/admin/deploy/docker-compose/upgrade.md | 2 +- doc/admin/deploy/kubernetes/helm.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/admin/deploy/docker-compose/upgrade.md b/doc/admin/deploy/docker-compose/upgrade.md index 33efa6440897..8974716a841d 100644 --- a/doc/admin/deploy/docker-compose/upgrade.md +++ b/doc/admin/deploy/docker-compose/upgrade.md @@ -55,7 +55,7 @@ $ docker-compose up -d --remove-orphans ### Multi-version upgrades -If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](/../../updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). +If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](../../updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). --- diff --git a/doc/admin/deploy/kubernetes/helm.md b/doc/admin/deploy/kubernetes/helm.md index 3baa0174e53d..886141b4aa06 100644 --- a/doc/admin/deploy/kubernetes/helm.md +++ b/doc/admin/deploy/kubernetes/helm.md @@ -978,7 +978,7 @@ When all pods have restarted and show as Running, you can browse to your Sourceg ### Multi-version upgrades -If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](/../../updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). +If you are upgrading to **Sourcegraph 5.1 or later**, we encourage you to perform an [**automatic multi-version upgrade**](../../updates/automatic.md). The following instructions are still applicable, just the manual version of what automatic multi-version upgrade do for you now (and therefore is valuable information during a bumpy upgrade). ---