From 2c7ea394941d2ab0f43faa695057c937d236f73d Mon Sep 17 00:00:00 2001 From: Ethan Mosbaugh Date: Fri, 4 Oct 2024 14:29:52 -0700 Subject: [PATCH 1/5] feat(ec): new data directory location and configuration --- docs/partials/embedded-cluster/_requirements.mdx | 5 ++--- docs/partials/embedded-cluster/_storage-reqs.mdx | 16 +++++++++++++--- .../support-bundles/_generate-bundle-ec.mdx | 2 +- docs/release-notes/rn-embedded-cluster.md | 2 +- docs/vendor/embedded-overview.mdx | 2 +- 5 files changed, 18 insertions(+), 9 deletions(-) diff --git a/docs/partials/embedded-cluster/_requirements.mdx b/docs/partials/embedded-cluster/_requirements.mdx index ea96596d7b..e5c8b58235 100644 --- a/docs/partials/embedded-cluster/_requirements.mdx +++ b/docs/partials/embedded-cluster/_requirements.mdx @@ -4,8 +4,7 @@ * At least 2GB of memory and 2 CPU cores * Disk space requirements: * The filesystem at `/var/lib/embedded-cluster` has 40Gi or more of total space - * The filesystem at `/var/lib/k0s` has 40Gi or more of total space and must be no more than 80% full - * The filesystem at `/var/openebs` has 5Gi or more of total space - * The filesystem at `/tmp` has 5Gi or more of total space + * The filesystem at `/var/lib/embedded-cluster/k0s` has 40Gi or more of total space and must be no more than 80% full + * The filesystem at `/var/lib/embedded-cluster/openebs-local` has 5Gi or more of total space * (Online installations only) Access to replicated.app and proxy.replicated.com or your custom domain for each * Embedded Cluster is based on k0s, so all k0s system requirements and external runtime dependencies apply. See [System requirements](https://docs.k0sproject.io/stable/system-requirements/) and [External runtime dependencies](https://docs.k0sproject.io/stable/external-runtime-deps/) in the k0s documentation. diff --git a/docs/partials/embedded-cluster/_storage-reqs.mdx b/docs/partials/embedded-cluster/_storage-reqs.mdx index f90be74f64..ab097f348c 100644 --- a/docs/partials/embedded-cluster/_storage-reqs.mdx +++ b/docs/partials/embedded-cluster/_storage-reqs.mdx @@ -1,5 +1,15 @@ Embedded Cluster stores data in the following primary locations, all of which must be writable: +The default location of the data directory can be modified by the user at install time by passing the `--data-dir` flag to the `install` command. +This directory cannot be changed after the cluster is installed. + +For example: + +```bash +sudo ./APP_SLUG install --data-dir /data/embedded-cluster --license license.yaml --airgap-bundle APP_SLUG.airgap +``` +Where `APP_SLUG` is the unique application slug. + :::note Replicated does not support using symlinked directories for these locations. Instead, Replicated recommends bind mounts, which provide greater consistency across different applications, preserve the original directory permissions and ownership, and have less overhead compared to symlinks. For more information, see [Relocating data directories with bind mounts](https://community.replicated.com/t/relocating-data-directories-with-bind-mounts/1435) in Replicated Community. ::: @@ -8,12 +18,11 @@ Replicated does not support using symlinked directories for these locations. Ins - Used to store your installer - Includes: - * Logs * Temporary files * Supporting binaries (for example, kubectl and Replicated’s preflight and support-bundle plugins) - Should be large enough to fit at least 2 installer binaries -#### `/var/lib/k0s` +#### `/var/lib/embedded-cluster/k0s` - Where the Kubernetes cluster runs - Includes: @@ -29,7 +38,7 @@ Replicated does not support using symlinked directories for these locations. Ins - Calculation example: If the sum of required storage for logs, images, and components is X GB, the total allocated space should be at least X / 0.85 GB to ensure it remains below 85% utilization. -#### `/var/openebs` +#### `/var/lib/embedded-cluster/openebs-local` - Where PVC data is stored - Needs enough storage to support all volumes used in your application @@ -65,5 +74,6 @@ In addition to the primary directories above, Embedded Cluster creates directori - /var/lib/kubelet - /var/log/calico - /var/log/containers +- /var/log/embedded-cluster - /var/log/pods - /usr/local/bin/k0s diff --git a/docs/partials/support-bundles/_generate-bundle-ec.mdx b/docs/partials/support-bundles/_generate-bundle-ec.mdx index 5befc5ebec..455e1b19aa 100644 --- a/docs/partials/support-bundles/_generate-bundle-ec.mdx +++ b/docs/partials/support-bundles/_generate-bundle-ec.mdx @@ -18,7 +18,7 @@ To generate a support bundle: ((____!___/) Type 'exit' (or CTRL+d) to exit. \0\0\0\0\/ Happy hacking. ~~~~~~~~~~~ - root@alex-ec-2:/home/alex# export KUBECONFIG="/var/lib/k0s/pki/admin.conf" + root@alex-ec-2:/home/alex# export KUBECONFIG="/var/lib/embedded-cluster/k0s/pki/admin.conf" root@alex-ec-2:/home/alex# export PATH="$PATH:/var/lib/embedded-cluster/bin" root@alex-ec-2:/home/alex# source <(kubectl completion bash) root@alex-ec-2:/home/alex# source /etc/bash_completion diff --git a/docs/release-notes/rn-embedded-cluster.md b/docs/release-notes/rn-embedded-cluster.md index 8bc41f15a0..01cf642f0b 100644 --- a/docs/release-notes/rn-embedded-cluster.md +++ b/docs/release-notes/rn-embedded-cluster.md @@ -277,7 +277,7 @@ Released on August 23, 2024 ### Bug Fixes {#bug-fixes-1-11-0} -* Fixes several issues that caused node resets to fail. Single-node clusters are no longer drained before being reset. Resets will no longer fail with the error `unable to get installation` if the installation failed early on. And node resets will now work if bind mounts are used for `/var/lib/embedded-cluster`, `/var/lib/k0s`, and `/var/openebs`. +* Fixes several issues that caused node resets to fail. Single-node clusters are no longer drained before being reset. Resets will no longer fail with the error `unable to get installation` if the installation failed early on. And node resets will now work if bind mounts are used for `/var/lib/embedded-cluster`, `/var/lib/embedded-cluster/k0s`, and `/var/lib/embedded-cluster/openebs-local`. * Fixes an issue where preflight checks for `modprobe`, `mount`, and `unmount` in `PATH` did not use absolute paths. * Fixes an issue where restoring did not work with S3-compatible object stores other than AWS S3. diff --git a/docs/vendor/embedded-overview.mdx b/docs/vendor/embedded-overview.mdx index 1c381a6370..3a0bca8e45 100644 --- a/docs/vendor/embedded-overview.mdx +++ b/docs/vendor/embedded-overview.mdx @@ -215,7 +215,7 @@ To access the cluster and use other included binaries: ((____!___/) Type 'exit' (or CTRL+d) to exit. \0\0\0\0\/ Happy hacking. ~~~~~~~~~~~ - root@alex-ec-2:/home/alex# export KUBECONFIG="/var/lib/k0s/pki/admin.conf" + root@alex-ec-2:/home/alex# export KUBECONFIG="/var/lib/embedded-cluster/k0s/pki/admin.conf" root@alex-ec-2:/home/alex# export PATH="$PATH:/var/lib/embedded-cluster/bin" root@alex-ec-2:/home/alex# source <(kubectl completion bash) root@alex-ec-2:/home/alex# source /etc/bash_completion From e64ec2c51832441f5a32e3461768361bb7118010 Mon Sep 17 00:00:00 2001 From: Paige Calvert Date: Tue, 8 Oct 2024 09:47:22 -0600 Subject: [PATCH 2/5] docs edits --- docs/enterprise/installing-embedded.mdx | 18 ++++++- .../embedded-cluster/_storage-reqs.mdx | 48 +------------------ 2 files changed, 19 insertions(+), 47 deletions(-) diff --git a/docs/enterprise/installing-embedded.mdx b/docs/enterprise/installing-embedded.mdx index c50d7dde69..1c60dbf09e 100644 --- a/docs/enterprise/installing-embedded.mdx +++ b/docs/enterprise/installing-embedded.mdx @@ -208,4 +208,20 @@ The Embedded Cluster host preflight checks verify the default ports or the user- ### Limitation -It is not possible to change the port for the Admin Console during a restore with Embedded Cluster. For more information, see [Disaster Recovery for Embedded Cluster (Alpha)](/vendor/embedded-disaster-recovery). \ No newline at end of file +It is not possible to change the port for the Admin Console during a restore with Embedded Cluster. For more information, see [Disaster Recovery for Embedded Cluster (Alpha)](/vendor/embedded-disaster-recovery). + +## Change the Default Data Directory + +The default location of the data directory can be modified by the user at install time by passing the `--data-dir` flag to the `install` command. +This directory cannot be changed after the cluster is installed. + +For example: + +```bash +sudo ./APP_SLUG install --data-dir /data/embedded-cluster --license license.yaml --airgap-bundle APP_SLUG.airgap +``` +Where `APP_SLUG` is the unique application slug. + +:::note +Replicated does not support using symlinked directories for these locations. Instead, Replicated recommends bind mounts, which provide greater consistency across different applications, preserve the original directory permissions and ownership, and have less overhead compared to symlinks. For more information, see [Relocating data directories with bind mounts](https://community.replicated.com/t/relocating-data-directories-with-bind-mounts/1435) in Replicated Community. +::: \ No newline at end of file diff --git a/docs/partials/embedded-cluster/_storage-reqs.mdx b/docs/partials/embedded-cluster/_storage-reqs.mdx index ab097f348c..5c001c5414 100644 --- a/docs/partials/embedded-cluster/_storage-reqs.mdx +++ b/docs/partials/embedded-cluster/_storage-reqs.mdx @@ -1,55 +1,11 @@ -Embedded Cluster stores data in the following primary locations, all of which must be writable: - -The default location of the data directory can be modified by the user at install time by passing the `--data-dir` flag to the `install` command. -This directory cannot be changed after the cluster is installed. - -For example: - -```bash -sudo ./APP_SLUG install --data-dir /data/embedded-cluster --license license.yaml --airgap-bundle APP_SLUG.airgap -``` -Where `APP_SLUG` is the unique application slug. - -:::note -Replicated does not support using symlinked directories for these locations. Instead, Replicated recommends bind mounts, which provide greater consistency across different applications, preserve the original directory permissions and ownership, and have less overhead compared to symlinks. For more information, see [Relocating data directories with bind mounts](https://community.replicated.com/t/relocating-data-directories-with-bind-mounts/1435) in Replicated Community. -::: - -#### `/var/lib/embedded-cluster` +Embedded Cluster stores data in `/var/lib/embedded-cluster`, which must be writable. - Used to store your installer - Includes: * Temporary files * Supporting binaries (for example, kubectl and Replicated’s preflight and support-bundle plugins) - Should be large enough to fit at least 2 installer binaries - -#### `/var/lib/embedded-cluster/k0s` - -- Where the Kubernetes cluster runs -- Includes: - * All Kubernetes components and binaries - * Pod log files - * Images currently in use by containerd -- Storage requirements: - * The total space should be sufficient to hold the following while remaining below 85% utilization: - * All Pod logs - * Two copies of infrastructure images included in k0s - * All Pod images for your application - * Kubernetes will initiate garbage collection of images when utilization reaches 85% -- Calculation example: - If the sum of required storage for logs, images, and components is X GB, the total allocated space should be at least X / 0.85 GB to ensure it remains below 85% utilization. - -#### `/var/lib/embedded-cluster/openebs-local` - -- Where PVC data is stored -- Needs enough storage to support all volumes used in your application -- For airgap installs: - * Registry uses a volume - * Requires space for two copies of your images (for registry) - * Additional space needed for other application volume requirements - * If high availability is enabled, all controller nodes will have the same requirements because data will be replicated to all controller nodes. - -#### `/tmp` -- Temporary space during installation +at least 40 Gi and less than 80% full #### Additional Directories and Files From ff80bf439eac98d9e9e393753e28c4b69b8442be Mon Sep 17 00:00:00 2001 From: Paige Calvert Date: Tue, 8 Oct 2024 10:36:10 -0600 Subject: [PATCH 3/5] docs edits --- docs/enterprise/installing-embedded.mdx | 17 +++++---- .../installing-general-requirements.mdx | 5 --- .../embedded-cluster/_requirements.mdx | 15 +++++--- .../embedded-cluster/_storage-reqs.mdx | 35 ------------------- docs/vendor/embedded-overview.mdx | 5 --- 5 files changed, 22 insertions(+), 55 deletions(-) delete mode 100644 docs/partials/embedded-cluster/_storage-reqs.mdx diff --git a/docs/enterprise/installing-embedded.mdx b/docs/enterprise/installing-embedded.mdx index 1c60dbf09e..cbb0236568 100644 --- a/docs/enterprise/installing-embedded.mdx +++ b/docs/enterprise/installing-embedded.mdx @@ -210,10 +210,11 @@ The Embedded Cluster host preflight checks verify the default ports or the user- It is not possible to change the port for the Admin Console during a restore with Embedded Cluster. For more information, see [Disaster Recovery for Embedded Cluster (Alpha)](/vendor/embedded-disaster-recovery). -## Change the Default Data Directory +## Change the Default Data Directory {#data-dir} -The default location of the data directory can be modified by the user at install time by passing the `--data-dir` flag to the `install` command. -This directory cannot be changed after the cluster is installed. +The default location of the data directory for Embedded Cluster is `/var/lib/embedded-cluster`. + +This directory can be modified by the user by passing the `--data-dir` flag to the `install` command. For example: @@ -222,6 +223,10 @@ sudo ./APP_SLUG install --data-dir /data/embedded-cluster --license license.yaml ``` Where `APP_SLUG` is the unique application slug. -:::note -Replicated does not support using symlinked directories for these locations. Instead, Replicated recommends bind mounts, which provide greater consistency across different applications, preserve the original directory permissions and ownership, and have less overhead compared to symlinks. For more information, see [Relocating data directories with bind mounts](https://community.replicated.com/t/relocating-data-directories-with-bind-mounts/1435) in Replicated Community. -::: \ No newline at end of file +### Limtiations + +* The data directory for Embedded Cluster cannot be changed after the cluster is installed. + +* If you use the `--data-dir` flag to change the data directory during installation, then you must use the same location when restoring in a disaster recovery scenario. For more information about disaster recovery with Embedded Cluster, see [Disaster Recovery for Embedded Cluster](/vendor/embedded-disaster-recovery). + +* Replicated does not support using symlinked directories for the Embedded Cluster data diretory. Instead, Replicated recommends bind mounts, which provide greater consistency across different applications, preserve the original directory permissions and ownership, and have less overhead compared to symlinks. For more information, see [Relocating data directories with bind mounts](https://community.replicated.com/t/relocating-data-directories-with-bind-mounts/1435) in Replicated Community. \ No newline at end of file diff --git a/docs/enterprise/installing-general-requirements.mdx b/docs/enterprise/installing-general-requirements.mdx index a66d22b02d..c462dd7f2d 100644 --- a/docs/enterprise/installing-general-requirements.mdx +++ b/docs/enterprise/installing-general-requirements.mdx @@ -1,7 +1,6 @@ import DockerCompatibility from "../partials/image-registry/_docker-compatibility.mdx" import KubernetesCompatibility from "../partials/install/_kubernetes-compatibility.mdx" import EmbeddedClusterRequirements from "../partials/embedded-cluster/_requirements.mdx" -import EmbeddedClusterStorageReqs from "../partials/embedded-cluster/_storage-reqs.mdx" import EmbeddedClusterPortRequirements from "../partials/embedded-cluster/_port-reqs.mdx" # Installation Requirements @@ -259,10 +258,6 @@ To install with the Embedded Cluster installer, your environment must meet the f -### Host Storage Requirements - - - ### Port Requirements diff --git a/docs/partials/embedded-cluster/_requirements.mdx b/docs/partials/embedded-cluster/_requirements.mdx index e5c8b58235..d4038bf13a 100644 --- a/docs/partials/embedded-cluster/_requirements.mdx +++ b/docs/partials/embedded-cluster/_requirements.mdx @@ -1,10 +1,17 @@ * Linux operating system + * x86-64 architecture + * systemd + * At least 2GB of memory and 2 CPU cores -* Disk space requirements: - * The filesystem at `/var/lib/embedded-cluster` has 40Gi or more of total space - * The filesystem at `/var/lib/embedded-cluster/k0s` has 40Gi or more of total space and must be no more than 80% full - * The filesystem at `/var/lib/embedded-cluster/openebs-local` has 5Gi or more of total space + +* The filesystem at `/var/lib/embedded-cluster` has 40Gi or more of total space and must be less than 80% full + + :::note + The directory used for data storage can be changed by passing the `--data-dir` flag with the Embedded Cluster install command. For more information, see [Change the Default Data Directory](/enterprise/installing-embedded#data-dir) in _Installing with Embedded Cluster_. + ::: + * (Online installations only) Access to replicated.app and proxy.replicated.com or your custom domain for each + * Embedded Cluster is based on k0s, so all k0s system requirements and external runtime dependencies apply. See [System requirements](https://docs.k0sproject.io/stable/system-requirements/) and [External runtime dependencies](https://docs.k0sproject.io/stable/external-runtime-deps/) in the k0s documentation. diff --git a/docs/partials/embedded-cluster/_storage-reqs.mdx b/docs/partials/embedded-cluster/_storage-reqs.mdx deleted file mode 100644 index 5c001c5414..0000000000 --- a/docs/partials/embedded-cluster/_storage-reqs.mdx +++ /dev/null @@ -1,35 +0,0 @@ -Embedded Cluster stores data in `/var/lib/embedded-cluster`, which must be writable. - -- Used to store your installer -- Includes: - * Temporary files - * Supporting binaries (for example, kubectl and Replicated’s preflight and support-bundle plugins) -- Should be large enough to fit at least 2 installer binaries -at least 40 Gi and less than 80% full - -#### Additional Directories and Files - -In addition to the primary directories above, Embedded Cluster creates directories and files in the following locations: - -- /etc/cni -- /etc/k0s -- /opt/cni -- /opt/containerd -- /run/calico -- /run/containerd -- /run/k0s -- /sys/fs/cgroup/kubepods -- /sys/fs/cgroup/system.slice/containerd.service -- /sys/fs/cgroup/system.slice/containerd.service -- /sys/fs/cgroup/system.slice/k0scontroller.service -- /sys/fs/cgroup/system.slice/k0scontroller.service -- /usr/libexec/k0s -- /var/lib/calico -- /var/lib/cni -- /var/lib/containers -- /var/lib/kubelet -- /var/log/calico -- /var/log/containers -- /var/log/embedded-cluster -- /var/log/pods -- /usr/local/bin/k0s diff --git a/docs/vendor/embedded-overview.mdx b/docs/vendor/embedded-overview.mdx index 3a0bca8e45..bb532e86f7 100644 --- a/docs/vendor/embedded-overview.mdx +++ b/docs/vendor/embedded-overview.mdx @@ -2,7 +2,6 @@ import EmbeddedCluster from "../partials/embedded-cluster/_definition.mdx" import KurlComparison from "../partials/embedded-cluster/_kurl-comparison.mdx" import Requirements from "../partials/embedded-cluster/_requirements.mdx" import UpdateOverview from "../partials/embedded-cluster/_update-overview.mdx" -import EmbeddedClusterStorageReqs from "../partials/embedded-cluster/_storage-reqs.mdx" import EmbeddedClusterSupportBundle from "../partials/support-bundles/_generate-bundle-ec.mdx" import EcConfig from "../partials/embedded-cluster/_ec-config.mdx" import EmbeddedClusterPortRequirements from "../partials/embedded-cluster/_port-reqs.mdx" @@ -37,10 +36,6 @@ As shown in the diagram above, the Embedded Cluster Config is included in the ap -#### Host Storage Requirements - - - #### Port Requirements From 212a30404f68274efee42211166dae2e7f5f2351 Mon Sep 17 00:00:00 2001 From: Paige Calvert Date: Tue, 8 Oct 2024 11:28:32 -0600 Subject: [PATCH 4/5] Update docs/enterprise/installing-embedded.mdx Co-authored-by: Alex Parker <7272359+ajp-io@users.noreply.github.com> --- docs/enterprise/installing-embedded.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/enterprise/installing-embedded.mdx b/docs/enterprise/installing-embedded.mdx index cbb0236568..4328c57903 100644 --- a/docs/enterprise/installing-embedded.mdx +++ b/docs/enterprise/installing-embedded.mdx @@ -229,4 +229,4 @@ Where `APP_SLUG` is the unique application slug. * If you use the `--data-dir` flag to change the data directory during installation, then you must use the same location when restoring in a disaster recovery scenario. For more information about disaster recovery with Embedded Cluster, see [Disaster Recovery for Embedded Cluster](/vendor/embedded-disaster-recovery). -* Replicated does not support using symlinked directories for the Embedded Cluster data diretory. Instead, Replicated recommends bind mounts, which provide greater consistency across different applications, preserve the original directory permissions and ownership, and have less overhead compared to symlinks. For more information, see [Relocating data directories with bind mounts](https://community.replicated.com/t/relocating-data-directories-with-bind-mounts/1435) in Replicated Community. \ No newline at end of file +* Replicated does not support using symlinked directories for the Embedded Cluster data directory. Use the `--data-dir` flag instead of symlinking `/var/lib/embedded-cluster`. \ No newline at end of file From ba3820f301a836d8e0577e1364c8de894d6fcab0 Mon Sep 17 00:00:00 2001 From: Paige Calvert Date: Tue, 8 Oct 2024 15:02:37 -0600 Subject: [PATCH 5/5] undo rn change --- docs/release-notes/rn-embedded-cluster.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/rn-embedded-cluster.md b/docs/release-notes/rn-embedded-cluster.md index 01cf642f0b..8bc41f15a0 100644 --- a/docs/release-notes/rn-embedded-cluster.md +++ b/docs/release-notes/rn-embedded-cluster.md @@ -277,7 +277,7 @@ Released on August 23, 2024 ### Bug Fixes {#bug-fixes-1-11-0} -* Fixes several issues that caused node resets to fail. Single-node clusters are no longer drained before being reset. Resets will no longer fail with the error `unable to get installation` if the installation failed early on. And node resets will now work if bind mounts are used for `/var/lib/embedded-cluster`, `/var/lib/embedded-cluster/k0s`, and `/var/lib/embedded-cluster/openebs-local`. +* Fixes several issues that caused node resets to fail. Single-node clusters are no longer drained before being reset. Resets will no longer fail with the error `unable to get installation` if the installation failed early on. And node resets will now work if bind mounts are used for `/var/lib/embedded-cluster`, `/var/lib/k0s`, and `/var/openebs`. * Fixes an issue where preflight checks for `modprobe`, `mount`, and `unmount` in `PATH` did not use absolute paths. * Fixes an issue where restoring did not work with S3-compatible object stores other than AWS S3.