diff --git a/_data/3.10-manual.yml b/_data/3.10-manual.yml index fc48482855..207018f519 100644 --- a/_data/3.10-manual.yml +++ b/_data/3.10-manual.yml @@ -467,7 +467,7 @@ - text: Monitoring href: monitoring-dc2dc.html - text: Troubleshooting - href: troubleshooting-dc2dc.html + href: troubleshooting-dc2dc.html - text: Foxx Microservices href: foxx.html children: diff --git a/_data/3.11-manual.yml b/_data/3.11-manual.yml index 3c8ce9932e..a71466b438 100644 --- a/_data/3.11-manual.yml +++ b/_data/3.11-manual.yml @@ -467,7 +467,7 @@ - text: Monitoring href: monitoring-dc2dc.html - text: Troubleshooting - href: troubleshooting-dc2dc.html + href: troubleshooting-dc2dc.html - text: Foxx Microservices href: foxx.html children: diff --git a/_data/oasis.yml b/_data/oasis.yml index ca2c3a3fd0..fc9205eaff 100644 --- a/_data/oasis.yml +++ b/_data/oasis.yml @@ -13,6 +13,8 @@ href: projects.html - text: Deployments href: deployments.html +- text: Migrate to the cloud + href: cloud-migration-tool.html - text: My Account href: my-account.html - text: Monitoring & Metrics diff --git a/_plugins/HintBlock.rb b/_plugins/HintBlock.rb index db4203dc0b..16531b5790 100644 --- a/_plugins/HintBlock.rb +++ b/_plugins/HintBlock.rb @@ -9,12 +9,15 @@ def render(context) converter = site.find_converter_instance(::Jekyll::Converters::Markdown) # Remove indention based on first actual line (but only spaces) - content = super.sub(/^[\r\n]+/, '') + content = super.sub(/^\R+/, '') + #content.rstrip! indent = content.index(/[^ ]/) || 0 content = content.lines.map{ |line| line.slice([indent, line.index(/[^ ]/) || 0].min, line.length) }.join '' - # Parse Markdown and strip trailing whitespace (especially line breaks). - # Otherwise below
s will be wrapped in

for some reason! - content = converter.convert(content).sub(/\s+$/, '') + # Parse Markdown and collapse line breaks to spaces. + # Line breaks can cause pre-mature closing of elements it seems, + # observed when placing a hint block in a Markdown list, + # leading to unwanted literal "

" (HTML-encoded) in the output. + content = converter.convert(content).gsub(/\R+/, ' ') case @type when "tip" @@ -35,9 +38,9 @@ def render(context) end return "
" + - "" + - "
#{content}
" + - "
" + "" + + "
#{content}
" + + "" end end Liquid::Template.register_tag('hint', HintTag) diff --git a/oasis/cloud-migration-tool.md b/oasis/cloud-migration-tool.md new file mode 100644 index 0000000000..cfa9b0c880 --- /dev/null +++ b/oasis/cloud-migration-tool.md @@ -0,0 +1,266 @@ +--- +layout: default +description: >- + Migrating data from bare metal servers to the cloud with minimal downtime +--- +# Cloud Migration Tool + +{{ page.description }} +{:class="lead"} + +{% hint 'info' %} +This documentation describes the preview version of the Cloud Migration tool +and is subject to change. +{% endhint %} + +The `arangosync-migration` tool allows you to easily move from on-premises to +the cloud while ensuring a smooth transition with minimal downtime. +Start the cloud migration, let the tool do the job and, at the same time, +keep your local cluster up and running. + +Some of the key benefits of the cloud migration tool include: +- Safety comes first - pre-checks and potential failures are carefully handled. +- Your data is secure and fully encrypted. +- Ease-of-use with a live migration while your local cluster is still in use. +- Get access to what a cloud-based fully managed service has to offer: +high availability and reliability, elastic scalability, and much more. + +## Downloading the tool + +The `arangosync-migration` tool is available to download for the following +operating systems: + +**Linux** +- [AMD64 (x86_64) architecture](https://download.arangodb.com/arangosync-migration/linux/amd64/arangosync-migration){:target="_blank"} +- [ARM64 (AArch64) architecture](https://download.arangodb.com/arangosync-migration/linux/arm64/arangosync-migration){:target="_blank"} + +**macOS / Darwin** +- [AMD64 (x86_64) architecture](https://download.arangodb.com/arangosync-migration/darwin/amd64/arangosync-migration){:target="_blank"} +- [ARM64 (AArch64) architecture](https://download.arangodb.com/arangosync-migration/darwin/arm64/arangosync-migration){:target="_blank"} + +**Windows** +- [AMD64 (x86_64) architecture](https://download.arangodb.com/arangosync-migration/windows/amd64/arangosync-migration.exe){:target="_blank"} +- [ARM64 (AArch64) architecture](https://download.arangodb.com/arangosync-migration/windows/arm64/arangosync-migration.exe){:target="_blank"} + +For macOS as well as other Unix-based operating systems, run the following +command to make sure you can execute the binary: + +```bash +chmod 755 ./arangosync-migration +``` + +## Prerequisites + +Before getting started, make sure the following prerequisites are in place: + +- Go to [Oasis Cloud](https://cloud.arangodb.com/home){:target="_blank"} +and sign in. If you don’t have an account yet, sign-up to create one. + +- Generate an Oasis API key and API secret. See a detailed guide on +[how to create an API key](api-getting-started.html#creating-an-api-key). + +- Your on-premises deployment should be able to communicate with the Oasis +platform. Check if the host is available and your firewall is not blocking +port `8629`. + +{% hint 'info' %} +The cloud migration tool is only available for clusters. +{% endhint %} + +### Setting up the target deployment in Oasis + +Continue by [creating a new Oasis deployment](deployments.html#how-to-create-a-new-deployment) +or choose an existing one. + +The target deployment in Oasis requires specific configuration rules to be +set up before the migration can start: + +- **Configuration settings**: The target deployment must be compatible with the +source data cluster. This includes the ArangoDB version that is being used, +the DB-Servers count, and disk space. +- **Deployment region and cloud provider**: Choose the closest region to your +data cluster. This factor can speed up your migration to the cloud. + +After setting up your Oasis deployment, wait for a few minutes for it to become +fully operational. + +{% hint 'info' %} +Note that Developer mode deployments are not supported. +{% endhint %} + +## Running the migration tool + +The `arangosync-migration` tool provides a set of commands that allow you to: +- start the migration process +- check whether your source and target clusters are fully compatible +- get the current status of the migration process +- stop or abort the migration process +- switch the local cluster to read-only mode + +### Starting the migration process + +To start the migration process, run the following command: + +```bash +arangosync-migration start +``` +The `start` command runs some pre-checks. Among other things, it measures +the disk space which is occupied by your ArangoDB cluster. If you are using the +same data volume for ArangoDB servers and other data as well, the measurements +can be incorrect. Provide the `--source.ignore-metrics` option to overcome this. + +You also have the option of doing a `--check-only` without starting the actual +migration. If specified, this checks if your local cluster and target deployment +are compatible without sending any data to Oasis. + +Once the migration starts, the local cluster enters into monitoring mode and the +synchronization status is displayed in real-time. If you don't want to see the +status you can also terminate this process, as the underlying agent process +continues to work. If something goes wrong, restarting the same command restores +the replication state. + +To restart the migration, first `stop` or `stop --abort` the migration. Then, +start it again using the `start` command. +Note that restarting works only if you are not using +auto-generated certificates. + +{% hint 'warning' %} +Starting the migration creates a full copy of all data from the source cluster +to the target deployment in Oasis. All data that has previously existed in the +target deployment will be lost. +{% endhint %} + +### During the migration + +The following takes place during an active migration: +- The source data cluster remains usable. +- The target deployment in Oasis is switched to read-only mode. +- Your root user password is not copied to the target deployment in Oasis. +To get your root password, select the target deployment from the Oasis +Dashboard and go to the **Overview** tab. All other users are fully synchronized. + +{% hint 'warning' %} +The migration tool increases the CPU and memory usage of the server you are +running it on. Depending on your ArangoDB usage pattern, it may take a lot of CPU +to handle the replication. You can stop the migration process anytime +if you see any problems. +{% endhint %} + +During active migration, the agent HTTPS server is executed on the `$MG_HOST:8629` +endpoint. Make sure the host and port are available for the Oasis platform. +To change the default port, use the `--agent.master-port` option. + +```bash +export MG_HOST= +./arangosync-migration start \ + --source.cacert=tls-ca.crt \ + --source.keyfile=client-auth.keyfile \ + --source.endpoint=$COORDINATOR_ENDPOINT \ + --source.jwtSecret=clusterSecret \ + --oasis.api-key=$OASIS_API_KEY \ + --oasis.api-secret=$OASIS_API_SECRET \ + --oasis.deployment-id=$OASIS_DEPLOYMENT_ID \ + --agent.listen-host=$MG_HOST +``` + +### TLS server certificates + +The migration agent HTTPS server uses TLS certificate pairs to ensure a secure +connection between your local cluster and the Oasis platform. +If you do not provide them, the migration tool creates self-signed certificates. +If you wish to provide TLS certificates, use the `arangodb` tool to convert them in +a suitable format for the migration tool. +See a detailed guide on how to [create a new certificate/keyfile pair](../programs-starter-security.html). +Make sure to specify your publicly available host name, `$MG_HOST`, when creating +the keyfile. + +When starting the migration, specify the generated files in the command line: + +```bash +--agent.cacert=tls.crt --agent.keyfile=tls.keyfile --agent.client-auth-cacert=client-auth-ca.crt --agent.client-auth-keyfile=client-auth.keyfile +``` + +### How long does it take? + +The total time required to complete the migration depends on how much data you +have and how often write operations are executed during the process. + +You can also track the progress by checking the **Migration status** section of +your target deployment in Oasis dashboard. + +![Oasis Cloud Migration Progress](images/oasis-migration-agent.png) + +### Getting the current status + +To print the current status of the migration, run the following command: + +```bash +./arangosync-migration status --watch \ + --oasis.api-key=$OASIS_API_KEY \ + --oasis.api-secret=$OASIS_API_SECRET \ + --oasis.deployment-id=$OASIS_DEPLOYMENT_ID +``` + +You can also add the `--watch` option to start monitoring the status in real-time. + +### Stopping the migration process + +The `arangosync-migration stop` command stops the migration and terminates +the migration agent process. + +If replication is running normally, the command waits until all shards are +in sync. The local cluster is then switched into read-only mode. +After all shards are in-sync and the migration stopped, the target deployment +is switched into normal mode (read/write). + +```bash +./arangosync-migration stop \ + --oasis.api-key=$OASIS_API_KEY \ + --oasis.api-secret=$OASIS_API_SECRET \ + --oasis.deployment-id=$OASIS_DEPLOYMENT_ID +``` + +The additional `--abort` option is supported. If specified, the `stop` command +will not check anymore if both deployments are in-sync and stops all +migration-related processes as soon as possible. + +### Switching the local cluster to read-only mode + +The `arangosync-migration set-server-mode` command allows switching [read-only mode](../http/administration-and-monitoring.html#update-whether-or-not-a-server-is-in-read-only-mode) +for your local cluster on and off. + +In a read-only mode, all write operations are going to fail with an error code +of `1004` (ERROR_READ_ONLY). +Creating or dropping databases and collections are also going to fail with +error code `11` (ERROR_FORBIDDEN). + +```bash +./arangosync-migration set-server-mode \ + --source.cacert=tls-ca.crt \ + --source.keyfile=client-auth.keyfile \ + --source.endpoint=$COORDINATOR_ENDPOINT \ + --source.jwtSecret=clusterSecret \ + --source.server-mode=readonly +``` +The `--source.server-mode` option allows you to specify the desired server mode. +Allowed values are `readonly` or `default`. + +## Cloud migration workflow for minimal downtime + +1. Download and start the `arangosync-migration` tool. The target deployment + is switched into read-only mode automatically. +2. Wait until all shards are in sync. You can use the `status` or the `start` + command with the same parameters to track that. +3. Optional: when all shards are in-sync, you can switch your applications + to use the Oasis deployment, but note that it stays in read-only mode + until the migration process is fully completed. +4. Stop the migration using the `stop` subcommand. The following steps are executed: + - The source data cluster is switched into read-only mode. + - It waits until all shards are synchronized. + - The target deployment is switched into default read/write mode. + + {% hint 'info' %} + If you switched the source data cluster into read-only mode, + you can switch it back to default (read/write) mode using the + `set-server-mode` subcommand. + {% endhint %} diff --git a/oasis/deployments.md b/oasis/deployments.md index 88256985e9..514f28c29f 100644 --- a/oasis/deployments.md +++ b/oasis/deployments.md @@ -37,7 +37,7 @@ Also see the video The configuration options depend on the tier your organization belongs to. For more details about available resources and usage limits, refer to the [Oasis tiers](organizations.html#oasis-tiers) section. - {% endhint %} + {% endhint %} ![Oasis New Deployment](images/oasis-new-deployment1.png) @@ -228,22 +228,22 @@ For more details about available resources and usage limits, refer to the 5. In the **Configuration** section, you can do the following: - Upgrade the memory size per node. - Modify the CPU per node from General to Low or vice-versa, if made available - by the cloud provider. + by the cloud provider. - Select a different disk size per node. The available ranges for the disk size - depend on the selected memory size. To enable automatic disk size scaling, move - the slider to a value higher than the current disk size. + depend on the selected memory size. To enable automatic disk size scaling, move + the slider to a value higher than the current disk size. - Change **OneShard** deployments into **Sharded** deployments. To do so, - click **Sharded**. In addition to the other configuration options, you can - select the number of nodes for your deployment. This can also be modified later on. + click **Sharded**. In addition to the other configuration options, you can + select the number of nodes for your deployment. This can also be modified later on. {% hint 'warning' %} Notice that you cannot switch from **Sharded** back to **OneShard**. {% endhint %} - AWS deployments have an additional option that allows you to select the - **Disk Performance** either with general settings, or optimised for large - and very large data sets. This option is dependent on the selected memory - size. For example, larger deployments have optimised settings by default. + **Disk Performance** either with general settings, or optimised for large + and very large data sets. This option is dependent on the selected memory + size. For example, larger deployments have optimised settings by default. {% hint 'warning' %} When upgrading the memory size, disk size, and/or disk performance in AWS deployments, diff --git a/oasis/images/oasis-migration-agent.png b/oasis/images/oasis-migration-agent.png new file mode 100644 index 0000000000..8eac6ebbf3 Binary files /dev/null and b/oasis/images/oasis-migration-agent.png differ