feat: add migrate harbor registry pvc storage to s3.#49
Conversation
WalkthroughAdds an English documentation page that describes an end-to-end procedure to migrate Harbor registry data from PVC storage to S3-compatible storage using rclone via a Kubernetes Job, including prerequisites, ConfigMap/Job YAML, execution and verification steps, Helm values to enable S3, diagnostics, and troubleshooting. (36 words) Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor User
participant K8s as Kubernetes API
participant Init as InitContainer (rclone sync)
participant PVC as Harbor PVC
participant S3 as S3 Bucket
participant Main as Main Container (rclone check)
User->>K8s: kubectl apply ConfigMap + Job (sync-and-check-s3)
activate K8s
K8s->>Init: Launch initContainer with rclone config
activate Init
Init->>PVC: Read registry data
Init->>S3: rclone sync uploads objects
Init-->>K8s: Exit when sync complete
deactivate Init
K8s->>Main: Start verification container
activate Main
Main->>S3: rclone check (compare)
Main->>PVC: Compare contents via mounted PVC
Main-->>K8s: Log result ("0 differences found" on success)
deactivate Main
K8s-->>User: Job status & logs
deactivate K8s
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (8)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (8)
168-172: Clarify success criterion and multi-pod logs.When using a Job, multiple pods can spawn on retries; use
--max-log-requests=10or select the pod explicitly to avoid mixing logs. Also, the exact success string may vary by rclone version.Suggested doc tweak:
-kubectl logs -n $HARBOR_NS -l job-name=sync-and-check-s3 | grep "0 differences found" +kubectl logs -n $HARBOR_NS job/sync-and-check-s3 --all-containers | grep -E "0 differences found|Everything is in sync"
199-203: Set Harbor “secure”/TLS flags appropriately when using HTTP endpoints.If
regionendpointishttp://..., Harbor usually needssecure: false(and sometimesskipverify: true). Otherwise connections may fail.Proposed addition:
s3: accesskey: xxxxx @@ - regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included + regionendpoint: http://xxxxx # Include port if non-443 + secure: false # Required when using HTTP endpoints + # skipverify: true # Optionally disable TLS verification for self-signed certs (only if using HTTPS)Please confirm the exact field names supported by your Harbor operator version.
24-31: Tighten wording for prerequisites; link usage and architecture hints.Minor language/formatting improvements and make it explicit that either the container image or the tarballs can be used.
Apply:
-- An S3-compatible storage service (MinIO, Ceph, AWS S3, etc.) with appropriate access credentials -- A pre-created S3 bucket for storing Harbor registry data -- The rclone migration tool image synced to your internal registry +- An S3-compatible storage service (MinIO, Ceph RGW, AWS S3, etc.) with valid access credentials. +- A pre-created S3 bucket for Harbor registry data. +- The rclone tool available, either as a container image (preferred) or the tarballs below.
16-16: Minor grammar fix.Use x-series notation.
Apply:
-This solution is compatible with Alauda Build of Harbor v2.12.z. +This solution is compatible with Alauda Build of Harbor v2.12.x.
62-69: Endpoint formatting and region note may trip users.rclone accepts endpoints with or without scheme depending on version; some S3-compatible providers require DNS/path-style differences and region settings. Call this out explicitly.
Doc tweak:
-export S3_HOST=http://xxxxx:xxx # S3 storage endpoint +export S3_HOST=http://xxxxx:xxx # S3 endpoint; include scheme/port if required by your provider. # ... -export S3_REGION=us-east-1 # If S3 doesn't have regions, this is not needed. If it exists, configure it and add region = $S3_REGION in the config below +export S3_REGION=us-east-1 # Set only if your provider requires a region; otherwise leave empty.And add a note below rclone.conf:
# Note: For some providers, the endpoint must NOT include a scheme. # If you see connection/signature errors, try removing http(s):// from endpoint.
214-221: Add a validation step to confirm Harbor actually uses S3.Include a quick runtime check to ensure writes land in the bucket and that the registry config reports S3.
Add:
2. **Check Storage**: Confirm that new images are being stored in the S3 bucket 3. **Verify Existing Images**: Ensure that previously migrated images can still be pulled successfully +4. **Confirm backend**: Exec into the registry pod and verify the storage driver is S3: + kubectl exec -n $HARBOR_NS deploy/harbor-registry -- cat /etc/registry/config.yml | grep storage:
222-230: Troubleshooting: add common S3-compatibility pitfalls.Add items for path-style addressing and clock skew.
Append:
- Ensure the S3 endpoint is accessible from the Harbor cluster - Review the migration job logs for any data integrity issues +- If you see SignatureDoesNotMatch or PermanentRedirect, enable path-style requests (rclone: force_path_style=true; Harbor: set s3.storageclass/pathstyle if available). +- Check node/system clock skew; large skews can break S3 signatures.
155-161: Optional: add activeDeadline/ttlSecondsAfterFinished and cleanup guidance.Improve ops hygiene and automatic cleanup.
Doc/YAML tweaks:
spec: backoffLimit: 0 + ttlSecondsAfterFinished: 3600 template: @@ volumes: - configMap: name: s3-config name: rclone-configAnd later in the doc:
# Cleanup after successful migration kubectl delete -f sync-and-check-s3.yaml
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
[grammar] ~16-~16: There might be a mistake here.
Context: ...nvironment This solution is compatible with Alauda Build of Harbor v2.12.z. ## Res...
(QB_NEW_EN)
[grammar] ~24-~24: There might be a mistake here.
Context: ...tc.) with appropriate access credentials - A pre-created S3 bucket for storing Harb...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ... bucket for storing Harbor registry data - The rclone migration tool image synced t...
(QB_NEW_EN)
kycheng
force-pushed
the
feat/add-harbor-migrate-to-s3
branch
from
6c1e2da to
2929426
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (4)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (4)
199-209: Remove leading “+” markers from the YAML; current snippet won’t apply.These diff artifacts break copy/paste for users.
-+ enabled: true -+ imageChartStorage: -+ disableredirect: true -+ s3: -+ accesskey: xxxxx # S3 cluster Access Key -+ bucket: harbor # Storage bucket created in S3 cluster -+ region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph) -+ regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included -+ secretkey: xxxx # S3 cluster Secret Key -+ v4auth: true -+ type: s3 + enabled: true + imageChartStorage: + disableredirect: true + s3: + accesskey: xxxxx # S3 Access Key + bucket: harbor # Pre-created bucket + region: us-east-1 # Required for AWS; optional for MinIO/Ceph + regionendpoint: http://xxxxx # Include port if non-443 + secretkey: xxxx # S3 Secret Key + v4auth: true + type: s3
95-96: Insert a Secret manifest for credentials and consume it via env in the Job.Add a Secret before the ConfigMap, and reference it from both containers. This removes keys from the ConfigMap and shell env.
metadata: name: s3-config namespace: $HARBOR_NS --- +apiVersion: v1 +kind: Secret +metadata: + name: s3-credentials + namespace: $HARBOR_NS +type: Opaque +stringData: + AWS_ACCESS_KEY_ID: $S3_KEY_ID + AWS_SECRET_ACCESS_KEY: $S3_ACCESS_KEY +--- apiVersion: batch/v1 kind: Job
111-121: Make rclone config path explicit and load creds from Secret; avoid HOME coupling.Mount to a neutral path and set
RCLONE_CONFIG. AddenvFromto pick up AWS creds from the Secret. This also avoids reliance on--configwhich can vary by rclone build.- image: $SYNC_IMAGE imagePullPolicy: IfNotPresent name: sync-data args: - sync - /data - harbor-s3:$S3_BUCKET - --progress + env: + - name: RCLONE_CONFIG + value: /etc/rclone/rclone.conf + envFrom: + - secretRef: + name: s3-credentials resources: @@ volumeMounts: - - mountPath: /root/.config/rclone/ - name: rclone-config + - mountPath: /etc/rclone/ + name: rclone-config - mountPath: /data name: dataAlso applies to: 126-131
80-90: Do not keep S3 keys in a ConfigMap; switch to Secret + env_auth and add force_path_style.ConfigMap is world-readable to users with read on configmaps. Move credentials to a Secret, set
env_auth = true, and addforce_path_style = truefor most S3-compatible endpoints (MinIO/Ceph). Also avoid relying on inline region lines unless required.Apply:
rclone.conf: |- [harbor-s3] type = s3 provider = $S3_PROVIDER - env_auth = false - access_key_id = $S3_KEY_ID - secret_access_key = $S3_ACCESS_KEY + env_auth = true endpoint = $S3_HOST acl = private + force_path_style = true # Add region configuration if your S3 service requires it # region = $S3_REGION
🧹 Nitpick comments (7)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (7)
68-68: Clarify provider values and case-sensitivity.rclone’s
provideris case-sensitive. Add examples and point to mapping (e.g., Minio, Ceph, AWS). If using AWS, omitendpointor set the regional endpoint; for generic S3-compatible, setendpointand keepforce_path_style = true.-export S3_PROVIDER=Minio # Configure based on S3 type. Supported providers: Minio, Ceph, AWS, etc. Refer to: https://rclone.org/docs/#configure +export S3_PROVIDER=Minio # Case-sensitive. Examples: Minio, Ceph, AWS. See https://rclone.org/s3/
120-125: Right-size requests/limits; document guidance to avoid starvation.The 1/4 (req/limit) ratio may oversubscribe busy clusters. Provide sizing guidance (e.g., start with
requests: {cpu: "2", memory: "2Gi"}and tune--transfersbased on network/IO), or state expected footprint for typical datasets.resources: limits: - cpu: 4 - memory: 4Gi + cpu: "4" + memory: 4Gi requests: - cpu: 1 - memory: 1Gi + cpu: "2" + memory: 2GiAlso applies to: 143-148
165-167: Elevate log monitoring into its own subsection.Make it harder to miss and add a
-c check-syncexample as well.-# Monitor the migration progress +#### Monitor migration progress kubectl logs -n $HARBOR_NS -l job-name=sync-and-check-s3 -c sync-data -f +# Or verify the check container: +# kubectl logs -n $HARBOR_NS -l job-name=sync-and-check-s3 -c check-sync -f
16-16: Minor copy edits for clarity.Tighten wording and fix small grammar nits.
-This solution is compatible with Alauda Build of Harbor v2.12.z. +This solution is compatible with Alauda Build of Harbor v2.12.x. @@ -- A fully deployed Harbor instance with `read-only mode` enabled. To enable read-only mode, Navigate to `Administration → Configuration → System Settings → Repository Read Only`. -- An S3-compatible storage service (MinIO, Ceph, AWS S3, etc.) with appropriate access credentials -- A pre-created S3 bucket for storing Harbor registry data -- The rclone migration tool image synced to your internal registry -- Ensure sufficient resources are available in the cluster where Harbor is deployed. +- A fully deployed Harbor instance with read-only mode enabled. Navigate to Administration → Configuration → System Settings → Repository Read Only. +- An S3-compatible storage service (MinIO, Ceph, AWS S3, etc.) with valid access credentials. +- A pre-created S3 bucket for storing Harbor registry data. +- The rclone migration tool image synced to your internal registry. +- Sufficient cluster resources in the Harbor namespace to run the migration Job.Also applies to: 24-29
31-33: Confirm attachment IDs/architectures and provide import steps.Links look placeholder-like; document how to load these archives into the internal registry (and specify amd64/arm64).
I can add a short section with “ctr/crictl/docker load && docker tag/push” commands and per-arch notes if you confirm final attachment URLs.
174-183: Nice: clear success criteria with “0 differences found.”This is the right log line to grep. Consider also noting that
rclone check --downloadstrengthens verification at the cost of time.
220-231: Good troubleshooting on resumability; call out idempotency of sync.You already note re-running the Job is safe; optionally add “use
--dry-runfor previews” to minimize risk.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-09-02T02:07:54.086Z
Learnt from: kycheng
PR: alauda/knowledge#49
File: docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md:111-121
Timestamp: 2025-09-02T02:07:54.086Z
Learning: The rclone sync command may not support --config parameter in all versions, despite documentation indicating it should be available as a global option. Alternative approaches include using RCLONE_CONFIG environment variable or configuring rclone entirely through environment variables.
Applied to files:
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
🪛 LanguageTool
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
[grammar] ~16-~16: There might be a mistake here.
Context: ...nvironment This solution is compatible with Alauda Build of Harbor v2.12.z. ## Res...
(QB_NEW_EN)
[grammar] ~24-~24: There might be a mistake here.
Context: ...System Settings → Repository Read Only`. - An S3-compatible storage service (MinIO,...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ...tc.) with appropriate access credentials - A pre-created S3 bucket for storing Harb...
(QB_NEW_EN)
[grammar] ~26-~26: There might be a mistake here.
Context: ... bucket for storing Harbor registry data - The rclone migration tool image synced t...
(QB_NEW_EN)
[grammar] ~27-~27: There might be a mistake here.
Context: ...l image synced to your internal registry - Ensure sufficient resources are availabl...
(QB_NEW_EN)
🔇 Additional comments (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (1)
73-73: Good: rclone image is pinned.Pinned to
rclone/rclone:1.71.0improves reproducibility. Ensure this matches the binaries provided in the attachment links and the target clusters’ CPU archs.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (8)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (8)
111-132: Inject credentials via Secret and use an explicit rclone config path.Make the config path explicit and avoid relying on
/root/.config; consume creds viaenvFrom. Given prior learning about--configinconsistencies, preferRCLONE_CONFIG.
[signal: using retrieved_learnings]- image: $SYNC_IMAGE imagePullPolicy: IfNotPresent name: sync-data args: - sync - /data - harbor-s3:$S3_BUCKET - --progress + env: + - name: RCLONE_CONFIG + value: /etc/rclone/rclone.conf + envFrom: + - secretRef: + name: s3-credentials resources: @@ volumeMounts: - - mountPath: /root/.config/rclone/ - name: rclone-config + - mountPath: /etc/rclone/ + name: rclone-config - mountPath: /data name: data
133-154: Apply the same config/env to the check container.Keep parity between sync and check to avoid “works in init, fails in main.”
- image: $SYNC_IMAGE imagePullPolicy: IfNotPresent name: check-sync args: - check - /data - harbor-s3:$S3_BUCKET - --one-way - --progress + env: + - name: RCLONE_CONFIG + value: /etc/rclone/rclone.conf + envFrom: + - secretRef: + name: s3-credentials resources: @@ volumeMounts: - - mountPath: /root/.config/rclone/ - name: rclone-config + - mountPath: /etc/rclone/ + name: rclone-config - mountPath: /data name: data
62-66: Quiesce Harbor writes and add a brief final sync.Read-only helps, but background jobs can still write. Scale down write-producing components before initial sync; after switching to S3, run a short final sync/check.
-1. **Data Synchronization**: Copy all registry data from PVC to S3 -2. **Data Verification**: Verify the integrity of migrated data +0. **Quiesce Harbor writes**: Scale down harbor-core, harbor-jobservice, harbor-registry +1. **Data Synchronization**: Copy all registry data from PVC to S3 +2. **Data Verification**: Verify integrity +3. **Final delta sync**: Quick pass to catch any drift before/after cutoverkubectl apply -f sync-and-check-s3.yaml - -# Monitor the migration progress +# Optionally ensure components are scaled down: +# kubectl -n $HARBOR_NS scale deploy harbor-core harbor-jobservice harbor-registry --replicas=0 + +# Monitor the migration progress kubectl logs -n $HARBOR_NS -l job-name=sync-and-check-s3 -c sync-data -f + +# After switching Harbor to S3, run a short final check: +# kubectl logs -n $HARBOR_NS job/sync-and-check-s3 -c check-sync | grep "0 differences found"Also applies to: 164-168
143-149: Same as above for the check container.
31-34: Validate attachment URLs/IDs.Confirm these artifact IDs and arch names before publishing.
68-74: Do not export S3 credentials into shell history.Avoid
export S3_KEY_ID/S3_ACCESS_KEYin plaintext. Read securely or load from a protected file.-export S3_KEY_ID=xxxx -export S3_ACCESS_KEY=xxxxx +read -r -p "S3 Access Key ID: " S3_KEY_ID +read -rs -p "S3 Secret Access Key: " S3_ACCESS_KEY; echo +export S3_KEY_ID S3_ACCESS_KEYAlternatively:
set -o allexport; source ./s3.env; set +o allexportwithchmod 600 s3.env. Want me to add a brief “.env” snippet?
81-92: Secrets must not live in a ConfigMap; switch to Secret + env_auth. Also add force_path_style.Access keys end up stored and distributed via ConfigMap (world-readable to many). Use a Secret and let rclone read from env; add
force_path_stylefor MinIO/Ceph.Apply within rclone.conf:
- env_auth = false - access_key_id = $S3_KEY_ID - secret_access_key = $S3_ACCESS_KEY + env_auth = true + force_path_style = trueAdd this Secret (outside this block) and reference it from the Job:
apiVersion: v1 kind: Secret metadata: name: s3-credentials namespace: $HARBOR_NS type: Opaque stringData: AWS_ACCESS_KEY_ID: $S3_KEY_ID AWS_SECRET_ACCESS_KEY: $S3_ACCESS_KEY ---I can wire this into the Job spec for you (see next comments).
199-210: Remove leading “+” markers; YAML is not copy/paste-safe.These plus signs break application.
- enabled: true - imageChartStorage: - disableredirect: true - s3: - accesskey: xxxxx # S3 cluster Access Key - bucket: harbor # Storage bucket created in S3 cluster - region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph) - regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included - secretkey: xxxx # S3 cluster Secret Key - v4auth: true - type: s3 + enabled: true + imageChartStorage: + disableredirect: true + s3: + accesskey: xxxxx + bucket: harbor + region: us-east-1 + regionendpoint: http://xxxxx + secretkey: xxxx + v4auth: true + type: s3
🧹 Nitpick comments (3)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (3)
16-16: Version wording nit.-This solution is compatible with Alauda Build of Harbor v2.12.z. +This solution is compatible with Alauda Build of Harbor v2.12.x.
24-26: Tighten wording and UX.-- **Important**: A fully deployed Harbor instance with `read-only mode` enabled. To enable read-only mode, Navigate to `Administration → Configuration → System Settings → Repository Read Only`. -- **Important**: Since Harbor needs to be set to read-only mode during migration, it's recommended to simulate this process in a test environment first, evaluate the migration time, and allocate sufficient maintenance window. +- **Important**: Enable Harbor’s read-only mode: Administration → Configuration → System Settings → Repository Read-Only. +- **Important**: Rehearse in a test environment, estimate migration time, and reserve a sufficient maintenance window.
104-109: Optional: auto-clean finished Jobs.spec: backoffLimit: 0 + ttlSecondsAfterFinished: 86400 template:
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-09-02T02:07:54.086Z
Learnt from: kycheng
PR: alauda/knowledge#49
File: docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md:111-121
Timestamp: 2025-09-02T02:07:54.086Z
Learning: The rclone sync command may not support --config parameter in all versions, despite documentation indicating it should be available as a global option. Alternative approaches include using RCLONE_CONFIG environment variable or configuring rclone entirely through environment variables.
Applied to files:
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
🪛 LanguageTool
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
[grammar] ~16-~16: There might be a mistake here.
Context: ...nvironment This solution is compatible with Alauda Build of Harbor v2.12.z. ## Res...
(QB_NEW_EN)
[grammar] ~24-~24: There might be a mistake here.
Context: ...System Settings → Repository Read Only`. - Important: Since Harbor needs to be se...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ...first, evaluate the migration time, and allocate sufficient maintenance window. - An S3-...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ... allocate sufficient maintenance window. - An S3-compatible storage service (MinIO,...
(QB_NEW_EN)
🔇 Additional comments (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (1)
120-126: Right-size resources or document rationale.Limits 4 CPU/4Gi vs requests 1 CPU/1Gi may overcommit in busy clusters. Provide expected data size/throughput and recommended flags (e.g.,
--transfers,--checkers) to tune.
kycheng
force-pushed
the
feat/add-harbor-migrate-to-s3
branch
from
16a482d to
b3b4f88
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (6)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (6)
162-166: Show log tail for both containers and the final short sync.Include optional final short sync/check after cutover (or when resuming from interruption).
kubectl apply -f sync-and-check-s3.yaml -# Monitor the migration progress +# Monitor the migration progress (sync then check) kubectl logs -n $HARBOR_NS -l job-name=sync-and-check-s3 -c sync-data -f +# After cutover (if not fully quiesced), rerun a brief sync/check: +# kubectl delete job -n $HARBOR_NS sync-and-check-s3 --ignore-not-found +# kubectl apply -f sync-and-check-s3.yaml +# kubectl logs -n $HARBOR_NS job/sync-and-check-s3 -c check-sync | grep "0 differences found"
20-20: State execution environment/permissions explicitly.Add prerequisites: k8s cluster access to create ConfigMaps/Secrets/Jobs, read access to Harbor PVC, egress to S3 endpoint, and sufficient quota.
24-26: Quiesce writes beyond read‑only; add scale-down/final sync.Read‑only helps, but background writers can still race. Add an explicit “quiesce” step and a short final sync at cutover.
-1. **Data Synchronization**: Copy all registry data from PVC to S3 -2. **Data Verification**: Verify the integrity of migrated data +0. **Quiesce Harbor writes**: Temporarily stop write-producing components +1. **Data Synchronization**: Copy all registry data from PVC to S3 +2. **Data Verification**: Verify the integrity of migrated data +3. **Final delta sync**: Brief sync immediately before/after cutoverAnd show concrete commands to scale down core/jobservice/registry and run a brief final sync after switching to S3.
136-152: Apply the same env/config/mount changes to the check container.Keep parity so
checkreads the same config/creds.- image: $SYNC_IMAGE imagePullPolicy: IfNotPresent name: check-sync args: - check - /data - harbor-s3:$S3_BUCKET - --one-way - --progress + env: + - name: RCLONE_CONFIG + value: /etc/rclone/rclone.conf + envFrom: + - secretRef: + name: s3-credentials @@ volumeMounts: - - mountPath: /root/.config/rclone/ - name: rclone-config + - mountPath: /etc/rclone/ + name: rclone-config - mountPath: /data name: data
81-91: Remove plaintext creds from ConfigMap; enable env_auth and path‑style.ConfigMaps are world‑readable; don’t bake creds. Also add
force_path_stylefor S3‑compatible endpoints.rclone.conf: |- [harbor-s3] type = s3 provider = $S3_PROVIDER - env_auth = false - access_key_id = $S3_KEY_ID - secret_access_key = $S3_ACCESS_KEY + env_auth = true endpoint = $S3_HOST acl = private + force_path_style = true # Add region configuration if your S3 service requires it # region = $S3_REGIONNote: If using HTTP endpoint intentionally, document the risk and keep Harbor/rclone configs aligned (TLS settings).
186-209: Remove stray “+” diff markers; fix indentation; verify S3 TLS setting.The plus signs will break copy/paste. Also align indentation to 6 spaces under helmValues → persistence. Verify whether your operator requires
secure: falsewhen using an HTTPregionendpoint.- persistence: - enabled: true -+ imageChartStorage: -+ disableredirect: true -+ s3: -+ accesskey: xxxxx # S3 cluster Access Key -+ bucket: harbor # Storage bucket created in S3 cluster -+ region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph) -+ regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included -+ secretkey: xxxx # S3 cluster Secret Key -+ v4auth: true -+ type: s3 + persistence: + enabled: true + imageChartStorage: + disableredirect: true + s3: + accesskey: xxxxx # S3 Access Key + bucket: harbor # Pre-created bucket + region: us-east-1 # Required for AWS; optional for MinIO/Ceph + regionendpoint: http://xxxxx + secretkey: xxxx # S3 Secret Key + v4auth: true + type: s3
🧹 Nitpick comments (6)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (6)
16-16: Tighten compatibility wording.Suggest: “Compatible with Alauda Harbor 2.12.x.” (“v2.12.z” is unconventional.)
-This solution is compatible with Alauda Build of Harbor v2.12.z. +This solution is compatible with Alauda Harbor 2.12.x.
38-55: Clarify provider-specific region and path‑style guidance.Add that most MinIO/Ceph RGW require path‑style and that AWS requires a valid region.
- Incorrect region configuration may cause authentication or connection failures + - For AWS S3, region is required. For MinIO/Ceph, set a dummy region if needed and enable path‑style requests (see rclone.conf: force_path_style = true).
118-124: Right-size requests/limits; document expected usage.4 vCPU/4Gi limits vs 1/1Gi requests may be too wide for oversubscribed clusters. Provide sizing guidance by dataset size and network bandwidth, or lower defaults.
Also applies to: 142-147
219-230: Add retry/backoff flags and interruption guidance.Document rclone’s restart-safety and suggest conservative flags for flaky links.
- **Upload Failures**: If upload fails, you can delete the job and recreate it. rclone will detect already transferred content and only transfer missing parts. + - **Transient failures**: Use flags like `--retries 10 --retries-sleep 10s --low-level-retries 20 --contimeout 30s --timeout 5m`. + - **Interruption**: Re-running the Job resumes safely; rclone only transfers missing/changed objects.
56-64: Add a brief Overview before deep steps.Helps estimations and stakeholder comms (phases, downtime window, validation).
### Migration Process +#### Overview +The migration consists of: (1) quiesce Harbor writes, (2) bulk sync PVC→S3, (3) verify, (4) cutover Harbor to S3, (5) optional final delta sync, (6) resume writes.
168-172: Minor wording: init container vs main container.Nit: “init container” → “initContainer” (Kubernetes term), but keep current casing if you prefer readability.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-09-02T02:07:54.135Z
Learnt from: kycheng
PR: alauda/knowledge#49
File: docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md:111-121
Timestamp: 2025-09-02T02:07:54.135Z
Learning: The rclone sync command may not support --config parameter in all versions, despite documentation indicating it should be available as a global option. Alternative approaches include using RCLONE_CONFIG environment variable or configuring rclone entirely through environment variables.
Applied to files:
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
🪛 LanguageTool
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
[grammar] ~16-~16: There might be a mistake here.
Context: ...nvironment This solution is compatible with Alauda Build of Harbor v2.12.z. ## Res...
(QB_NEW_EN)
[grammar] ~24-~24: There might be a mistake here.
Context: ...System Settings → Repository Read Only`. - Important: Since Harbor needs to be se...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ...first, evaluate the migration time, and allocate sufficient maintenance window. - An S3-...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ... allocate sufficient maintenance window. - An S3-compatible storage service (MinIO,...
(QB_NEW_EN)
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (2)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (2)
31-39: Prefer a pinned OCI image over brittle tarball URLs.Use a versioned image with digest you control; confirm final artifact locations.
-https://cloud.alauda.cn/attachments/knowledge/337969938/rclone-amd64.tgz -https://cloud.alauda.cn/attachments/knowledge/337969938/rclone-arm64.tgz +export SYNC_IMAGE=registry.local/ops/rclone:1.71.0@sha256:<digest>
67-69: Quiesce Harbor writes and add a final delta sync step.Read-only helps but background writers can still race; scale down write-producing components before sync and do a brief final sync after cutover.
-1. **Data Synchronization**: Copy all registry data from PVC to S3 -2. **Data Verification**: Verify the integrity of migrated data +0. **Quiesce Harbor writes**: Option A: enable read-only and scale down core/jobservice/registry. Option B: stop Harbor during sync. +1. **Data Synchronization**: Copy all registry data from PVC to S3 +2. **Data Verification**: Verify the integrity of migrated data +3. **Final delta sync**: Run a short sync/check immediately before/after cutover.Example commands (before Line 167):
+# Optionally scale down writers before syncing: +# kubectl -n $HARBOR_NS scale deploy harbor-core harbor-jobservice harbor-registry --replicas=0After switching Harbor to S3, optionally:
+# Re-run the job for a quick delta check if not fully quiesced. +# kubectl delete job -n $HARBOR_NS sync-and-check-s3 --ignore-not-found +# kubectl apply -f sync-and-check-s3.yaml
🧹 Nitpick comments (7)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (7)
108-112: Add basic Job hygiene (retry/cleanup).Zero retries and no TTL makes ops harder.
spec: - backoffLimit: 0 + backoffLimit: 1 + ttlSecondsAfterFinished: 86400 template: spec: restartPolicy: Never
169-171: Target the correct container when tailing logs.Avoid mixing init/main logs.
-kubectl logs -n $HARBOR_NS -l job-name=sync-and-check-s3 -c sync-data -f +kubectl logs -n $HARBOR_NS job/sync-and-check-s3 -c sync-data -f
180-187: Narrow verification to the check container and fail fast.Make grep explicit and propagate a non-zero exit on mismatch.
-kubectl logs -n $HARBOR_NS -l job-name=sync-and-check-s3 | grep "0 differences found" -Defaulted container "check-sync" out of: check-sync, sync-data (init) -2025/09/01 07:30:12 NOTICE: S3 bucket harbor: 0 differences found +kubectl logs -n $HARBOR_NS job/sync-and-check-s3 -c check-sync | grep -q "0 differences found" && echo "OK" || (echo "Mismatch detected" >&2; exit 1)
72-82: Avoid exporting secrets into shell history; prefer Secret + stdin.Use read -s or a protected .env, then create the Kubernetes Secret (see earlier comment).
-export S3_KEY_ID=xxxx -export S3_ACCESS_KEY=xxxxx +read -r S3_KEY_ID +read -rs S3_ACCESS_KEY +export S3_KEY_ID +export S3_ACCESS_KEY
74-79: Clarify provider names and region handling.List accepted provider values and note when region is mandatory.
-export S3_PROVIDER=Minio # Configure based on S3 type. Supported providers: Minio, Ceph, AWS, etc. Refer to: https://rclone.org/docs/#configure +export S3_PROVIDER=Minio # One of: Minio | Ceph | AWS +# AWS requires a valid S3_REGION; MinIO/Ceph usually don't (leave empty).
115-123: Consider performance flags for large datasets.Optional: expose transfers/checkers and timeouts.
- harbor-s3:$S3_BUCKET - - --progress + - --progress + - --transfers=32 + - --checkers=64 + - --contimeout=30s + - --timeout=5m @@ - harbor-s3:$S3_BUCKET - --one-way - - --progress + - --progress + - --checkers=64 + - --contimeout=30s + - --timeout=5mAdjust numbers per cluster capacity.
Also applies to: 140-146
16-16: Minor wording fixes for clarity.-This solution is compatible with Alauda Build of Harbor v2.12.z. +This solution is compatible with Alauda Build of Harbor v2.12.x. @@ -... allocate sufficient maintenance window. +... allocate a sufficient maintenance window.Also applies to: 24-25
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-09-02T02:07:54.135Z
Learnt from: kycheng
PR: alauda/knowledge#49
File: docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md:111-121
Timestamp: 2025-09-02T02:07:54.135Z
Learning: The rclone sync command may not support --config parameter in all versions, despite documentation indicating it should be available as a global option. Alternative approaches include using RCLONE_CONFIG environment variable or configuring rclone entirely through environment variables.
Applied to files:
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
🪛 LanguageTool
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
[grammar] ~16-~16: There might be a mistake here.
Context: ...nvironment This solution is compatible with Alauda Build of Harbor v2.12.z. ## Res...
(QB_NEW_EN)
[grammar] ~24-~24: There might be a mistake here.
Context: ...System Settings → Repository Read Only`. - Important: Since Harbor needs to be se...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ...first, evaluate the migration time, and allocate sufficient maintenance window. - An S3-...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ... allocate sufficient maintenance window. - An S3-compatible storage service (MinIO,...
(QB_NEW_EN)
🔇 Additional comments (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (1)
124-129: Right-size resources or document rationale.Limits 4 CPU/4Gi vs requests 1 CPU/1Gi may oversubscribe nodes; please provide guidance or tune defaults.
If typical usage is lower, consider 2 CPU/2Gi limits or add a note explaining expected throughput and sizing.
Also applies to: 147-152
| rclone.conf: |- | ||
| [harbor-s3] | ||
| type = s3 | ||
| provider = $S3_PROVIDER | ||
| env_auth = false | ||
| access_key_id = $S3_KEY_ID | ||
| secret_access_key = $S3_ACCESS_KEY | ||
| endpoint = $S3_HOST | ||
| acl = private | ||
| # Add region configuration if your S3 service requires it | ||
| # region = $S3_REGION | ||
| kind: ConfigMap |
There was a problem hiding this comment.
Do not embed S3 credentials in a ConfigMap; switch to a Secret and env_auth.
Leaking access keys into a world-readable ConfigMap is a security risk. Move creds to a Secret, enable env_auth, and remove keys from rclone.conf.
Apply:
apiVersion: v1
data:
rclone.conf: |-
[harbor-s3]
type = s3
provider = $S3_PROVIDER
- env_auth = false
- access_key_id = $S3_KEY_ID
- secret_access_key = $S3_ACCESS_KEY
+ env_auth = true
endpoint = $S3_HOST
acl = private
+ # Strongly recommended for MinIO/Ceph:
+ force_path_style = true
# Add region configuration if your S3 service requires it
# region = $S3_REGION
kind: ConfigMapInsert this Secret before the ConfigMap:
apiVersion: v1
kind: Secret
metadata:
name: s3-credentials
namespace: $HARBOR_NS
type: Opaque
stringData:
AWS_ACCESS_KEY_ID: $S3_KEY_ID
AWS_SECRET_ACCESS_KEY: $S3_ACCESS_KEY
---And inject env into both containers (see line 115 and 137 blocks):
- image: $SYNC_IMAGE
name: sync-data
args:
- sync
- /data
- harbor-s3:$S3_BUCKET
- --progress
+ envFrom:
+ - secretRef:
+ name: s3-credentials
@@
- image: $SYNC_IMAGE
name: check-sync
args:
- check
- /data
- harbor-s3:$S3_BUCKET
- --one-way
- --progress
+ envFrom:
+ - secretRef:
+ name: s3-credentials🤖 Prompt for AI Agents
In docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md around
lines 86 to 97, the rclone.conf sample embeds S3 credentials in a ConfigMap;
replace that by creating a Kubernetes Secret (Opaque, stringData keys
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) inserted before the ConfigMap,
remove access_key_id and secret_access_key lines from rclone.conf, set env_auth
= true in rclone.conf, and update the pod/container specs referenced at lines
~115 and ~137 to inject the Secret as environment variables (AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY) instead of embedding credentials in the ConfigMap.
| - mountPath: /root/.config/rclone/ | ||
| name: rclone-config | ||
| - mountPath: /data | ||
| name: data | ||
| containers: |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Avoid relying on root HOME for rclone config; set an explicit config path.
Mount to a neutral path and set RCLONE_CONFIG so older rclone builds work even if --config isn’t honored.
Apply to both containers:
- volumeMounts:
- - mountPath: /root/.config/rclone/
- name: rclone-config
+ env:
+ - name: RCLONE_CONFIG
+ value: /etc/rclone/rclone.conf
+ volumeMounts:
+ - mountPath: /etc/rclone/
+ name: rclone-config
- mountPath: /data
name: dataAlso applies to: 153-157
| apiVersion: operator.alaudadevops.io/v1alpha1 | ||
| kind: Harbor | ||
| metadata: | ||
| name: test | ||
| spec: | ||
| helmValues: | ||
| persistence: | ||
| enabled: true | ||
| + imageChartStorage: | ||
| + disableredirect: true | ||
| + s3: | ||
| + accesskey: xxxxx # S3 cluster Access Key | ||
| + bucket: harbor # Storage bucket created in S3 cluster | ||
| + region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph) | ||
| + regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included | ||
| + secretkey: xxxx # S3 cluster Secret Key | ||
| + v4auth: true | ||
| + type: s3 | ||
| ``` |
There was a problem hiding this comment.
Fix YAML: remove leading “+” markers; add secure=false when endpoint is http.
The current block won’t parse; and with an http regionendpoint you should set secure: false.
spec:
helmValues:
persistence:
- enabled: true
-+ imageChartStorage:
-+ disableredirect: true
-+ s3:
-+ accesskey: xxxxx # S3 cluster Access Key
-+ bucket: harbor # Storage bucket created in S3 cluster
-+ region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph)
-+ regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included
-+ secretkey: xxxx # S3 cluster Secret Key
-+ v4auth: true
-+ type: s3
+ enabled: true
+ imageChartStorage:
+ disableredirect: true
+ s3:
+ accesskey: xxxxx
+ bucket: harbor
+ region: us-east-1
+ regionendpoint: http://xxxxx
+ secretkey: xxxx
+ v4auth: true
+ secure: false # set true if using https endpoint
+ type: s3📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| apiVersion: operator.alaudadevops.io/v1alpha1 | |
| kind: Harbor | |
| metadata: | |
| name: test | |
| spec: | |
| helmValues: | |
| persistence: | |
| enabled: true | |
| + imageChartStorage: | |
| + disableredirect: true | |
| + s3: | |
| + accesskey: xxxxx # S3 cluster Access Key | |
| + bucket: harbor # Storage bucket created in S3 cluster | |
| + region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph) | |
| + regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included | |
| + secretkey: xxxx # S3 cluster Secret Key | |
| + v4auth: true | |
| + type: s3 | |
| ``` | |
| apiVersion: operator.alaudadevops.io/v1alpha1 | |
| kind: Harbor | |
| metadata: | |
| name: test | |
| spec: | |
| helmValues: | |
| persistence: | |
| enabled: true | |
| imageChartStorage: | |
| disableredirect: true | |
| s3: | |
| accesskey: xxxxx | |
| bucket: harbor | |
| region: us-east-1 | |
| regionendpoint: http://xxxxx | |
| secretkey: xxxx | |
| v4auth: true | |
| secure: false # set true if using an HTTPS endpoint | |
| type: s3 |
🤖 Prompt for AI Agents
In docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md around
lines 196 to 214, the provided YAML contains leading "+" characters that will
break parsing and is missing the secure flag for an HTTP S3 endpoint; remove all
leading "+" markers so the YAML is valid and under
spec.helmValues.persistence.imageChartStorage.s3 add secure: false when using an
http regionendpoint (keep regionendpoint value and ensure correct indentation),
and ensure type: s3 remains at the same level as the s3 block.
kycheng
force-pushed
the
feat/add-harbor-migrate-to-s3
branch
from
215c04a to
012ceaf
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (3)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (3)
115-123: Make rclone config path explicit and inject creds from Secret in both containers.Avoid reliance on /root/.config and implicit HOME. Ensure check container mirrors the same env/mounts.
- image: $SYNC_IMAGE imagePullPolicy: IfNotPresent name: sync-data args: - - sync + - sync - /data - harbor-s3:$S3_BUCKET - --progress + env: + - name: RCLONE_CONFIG + value: /etc/rclone/rclone.conf + envFrom: + - secretRef: + name: s3-credentials @@ - volumeMounts: - - mountPath: /root/.config/rclone/ - name: rclone-config + volumeMounts: + - mountPath: /etc/rclone/ + name: rclone-config - mountPath: /data name: data @@ - image: $SYNC_IMAGE imagePullPolicy: IfNotPresent name: check-sync args: - check - /data - harbor-s3:$S3_BUCKET - --one-way - --progress + env: + - name: RCLONE_CONFIG + value: /etc/rclone/rclone.conf + envFrom: + - secretRef: + name: s3-credentials @@ - volumeMounts: - - mountPath: /root/.config/rclone/ - name: rclone-config + volumeMounts: + - mountPath: /etc/rclone/ + name: rclone-config - mountPath: /data name: data @@ volumes: - configMap: name: s3-config name: rclone-config - name: data persistentVolumeClaim: claimName: $HARBOR_REGISTRY_PVCNote: if your rclone build ignores --config, RCLONE_CONFIG still works per retrieved learnings.
Also applies to: 131-135, 137-146, 153-157, 158-165
86-97: Do not store S3 credentials in a ConfigMap; enable env_auth and add path-style.Leaking keys in a ConfigMap is a security risk. Read credentials from the Secret via environment variables and enable env_auth. Path-style is recommended for MinIO/Ceph.
rclone.conf: |- [harbor-s3] type = s3 provider = $S3_PROVIDER - env_auth = false - access_key_id = $S3_KEY_ID - secret_access_key = $S3_ACCESS_KEY + env_auth = true endpoint = $S3_HOST acl = private + # Strongly recommended for S3-compatible providers: + force_path_style = true # Add region configuration if your S3 service requires it # region = $S3_REGION
221-229: Remove leading “+” diff markers; add secure: false for http endpoint.As-is, this YAML won’t parse; also set secure: false when regionendpoint is http.
persistence: - enabled: true -+ imageChartStorage: -+ disableredirect: true -+ s3: -+ existingSecret: s3-secret # an secret for S3 accesskey and secretkey -+ bucket: harbor # Storage bucket created in S3 cluster -+ region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph) -+ regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included -+ v4auth: true -+ type: s3 + enabled: true + imageChartStorage: + disableredirect: true + s3: + existingSecret: s3-secret # Secret containing access/secret key + bucket: harbor # Pre-created bucket + region: us-east-1 # Required for AWS; optional for MinIO/Ceph + regionendpoint: http://xxxxx + v4auth: true + secure: false # Set true if using https + type: s3
🧹 Nitpick comments (4)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (4)
108-112: Consider Job hygiene: retries and cleanup.Allow a retry and auto-cleanup completed Jobs to reduce manual ops.
spec: - backoffLimit: 0 + backoffLimit: 1 + ttlSecondsAfterFinished: 86400 template: spec: restartPolicy: Never
169-171: Promote log monitoring to its own subsection; include a success grep.Improves discoverability and reduces the chance users miss verification.
-# Monitor the migration progress +### Monitor Logs and Verify Success +# Monitor the migration progress kubectl logs -n $HARBOR_NS -l job-name=sync-and-check-s3 -c sync-data -f + +# Verify "0 differences found" after completion: +kubectl logs -n $HARBOR_NS -l job-name=sync-and-check-s3 | grep "0 differences found"Also applies to: 178-187
31-39: Replace brittle tarball URLs with a pinned OCI image reference (or verify URLs).Prefer a pinned rclone container image (tag+digest) you control. If URLs are required, confirm final locations.
-# Download URL for China Region -https://cloud.alauda.cn/attachments/knowledge/337969938/rclone-amd64.tgz -https://cloud.alauda.cn/attachments/knowledge/337969938/rclone-arm64.tgz - -# Download URLs for Other Regions -https://cloud.alauda.io/attachments/knowledge/337969545/rclone-amd64.tgz -https://cloud.alauda.io/attachments/knowledge/337969545/rclone-arm64.tgz +# Prefer a pinned image you control: +# export SYNC_IMAGE=registry.local/ops/rclone:1.71.0@sha256:<digest>Also applies to: 79-79
16-16: Minor grammar/wording nits.Very small readability fixes.
-This solution is compatible with Alauda Build of Harbor v2.12.z. +This solution is compatible with Alauda Build of Harbor v2.12.x. @@ -- **Important**: Since Harbor needs to be set to read-only mode during migration, it's recommended to simulate this process in a test environment first, evaluate the migration time, and allocate sufficient maintenance window. +- **Important**: Because Harbor must be read-only during migration, first simulate the process in a test environment, estimate the migration time, and allocate a sufficient maintenance window.Also applies to: 24-25
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-09-02T02:07:54.135Z
Learnt from: kycheng
PR: alauda/knowledge#49
File: docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md:111-121
Timestamp: 2025-09-02T02:07:54.135Z
Learning: The rclone sync command may not support --config parameter in all versions, despite documentation indicating it should be available as a global option. Alternative approaches include using RCLONE_CONFIG environment variable or configuring rclone entirely through environment variables.
Applied to files:
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
🪛 LanguageTool
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
[grammar] ~16-~16: There might be a mistake here.
Context: ...nvironment This solution is compatible with Alauda Build of Harbor v2.12.z. ## Res...
(QB_NEW_EN)
[grammar] ~24-~24: There might be a mistake here.
Context: ...System Settings → Repository Read Only`. - Important: Since Harbor needs to be se...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ...first, evaluate the migration time, and allocate sufficient maintenance window. - An S3-...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ... allocate sufficient maintenance window. - An S3-compatible storage service (MinIO,...
(QB_NEW_EN)
[grammar] ~195-~195: There might be a mistake here.
Context: ...ACCESSKEY: Base64-encoded S3 access key - REGISTRY_STORAGE_S3_SECRETKEY`: Base64-encoded S3 secret key ```yaml ...
(QB_NEW_EN)
🔇 Additional comments (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (1)
124-129: Right-size resources or make them configurable.Requests/limits differ by 4x. Document typical usage or expose via variables to avoid scheduling failures.
Would you like me to parameterize cpu/memory via env-substitution in the YAML?
Also applies to: 147-152
| - sync | ||
| - /data | ||
| - harbor-s3:$S3_BUCKET | ||
| - --progress |
There was a problem hiding this comment.
Prevent accidental deletions when destination bucket is non-empty.
rclone sync will delete extraneous objects on the destination. Use a new/empty bucket, run a dry-run, or prefer copy to be safe.
- - sync
+ # Prefer 'copy' if the bucket may contain other data; or ensure it's empty.
+ - copy
- /data
- harbor-s3:$S3_BUCKET
- --progress
+ # Add a dry-run first if unsure:
+ # - --dry-runAlso add to the process description:
1. **Data Synchronization**: Copy all registry data from PVC to S3
2. **Data Verification**: Verify the integrity of migrated data
+> Important: If the target bucket is not empty, prefer rclone copy or run sync with --dry-run first to avoid unintended deletions.Also applies to: 65-69
🤖 Prompt for AI Agents
In docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md around
lines 119-122 (also applies to lines 65-69), the rclone example uses "rclone
sync" which will delete extraneous objects in the destination bucket; update the
docs to warn readers to avoid accidental deletions by recommending either using
a new/empty S3 bucket, running rclone with --dry-run first, or using "rclone
copy" instead of "sync", and add a short process step that explicitly instructs
these precautions before running the sync.
|
已验证通过 |
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (10)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (10)
31-38: Replace brittle tarball URLs or publish checksums and OCI alternative.Tarball links can rot. Prefer a pinned OCI image with digest, or at least provide SHA256 checksums and offline “docker load → push to internal registry” steps.
-https://cloud.alauda.cn/attachments/knowledge/337969938/rclone-amd64.tgz -https://cloud.alauda.cn/attachments/knowledge/337969938/rclone-arm64.tgz +export SYNC_IMAGE=registry.local/ops/rclone:1.71.0@sha256:<digest> +# For air‑gapped: provide amd64/arm64 .tgz plus SHA256 and: +# docker load -i rclone-<arch>.tgz && docker tag ... && docker push ...
93-97: Add basic Job hygiene (retry/cleanup).Allow one retry and auto-cleanup finished Jobs.
-spec: - backoffLimit: 0 +spec: + backoffLimit: 1 + ttlSecondsAfterFinished: 86400
125-131: Mirror the same config/env on the check container.Keep credentials and config handling consistent between sync and check.
args: - check - /data - harbor-s3:$S3_BUCKET - --one-way - --progress + env: + - name: RCLONE_CONFIG + value: /etc/rclone/rclone.conf + envFrom: + - secretRef: + name: s3-credentials
139-142: Same mount path fix for check container.- volumeMounts: - - mountPath: /root/.config/rclone/ - name: rclone-config + volumeMounts: + - mountPath: /etc/rclone/ + name: rclone-config - mountPath: /data name: data
152-156: Add optional final delta sync/check after cutover.Prevents drift if Harbor wasn’t fully quiesced.
kubectl apply -f sync-and-check-s3.yaml - -# Option: Monitor the migration progress +# Option: Monitor the migration progress kubectl logs -n $HARBOR_NS -l job-name=sync-and-check-s3 -c sync-data -f +# Optional: re-run a short final sync/check right before/after switching Harbor to S3. +# kubectl delete job -n $HARBOR_NS sync-and-check-s3 --ignore-not-found +# kubectl apply -f sync-and-check-s3.yaml
24-26: Quiesce Harbor writes, not only read‑only UI.Read‑only helps but background writers (jobservice/registry GC) can still race. Add an explicit “quiesce” step and post‑cutover quick delta sync.
This section describes how to migrate existing Harbor registry data from PVC to S3 storage using rclone. The migration process includes: -1. **Data Synchronization**: Copy all registry data from PVC to S3 -2. **Data Verification**: Verify the integrity of migrated data +0. **Quiesce Harbor writes**: Scale down write-producing components +1. **Data Synchronization**: Copy all registry data from PVC to S3 +2. **Data Verification**: Verify the integrity of migrated dataSuggested commands to add before the first sync and around cutover:
# Before initial sync kubectl -n $HARBOR_NS scale deploy harbor-core harbor-jobservice harbor-registry --replicas=0 # After switching Harbor to S3, optionally re-run a short final sync/check, then scale back up # kubectl apply -f sync-and-check-s3.yaml kubectl -n $HARBOR_NS scale deploy harbor-core harbor-jobservice harbor-registry --replicas=1
57-67: Do not export secrets into shell history; use a Kubernetes Secret.Avoid
export S3_KEY_ID/S3_ACCESS_KEY. Read interactively or from a secure file and create a Secret the Job consumes.-export S3_KEY_ID=xxxx -export S3_ACCESS_KEY=xxxxx +read -r S3_KEY_ID +read -rs S3_ACCESS_KEY +export S3_KEY_ID S3_ACCESS_KEYInsert (before ConfigMap/Job):
apiVersion: v1 kind: Secret metadata: name: s3-credentials namespace: $HARBOR_NS type: Opaque stringData: AWS_ACCESS_KEY_ID: $S3_KEY_ID AWS_SECRET_ACCESS_KEY: $S3_ACCESS_KEY ---
71-82: Move credentials out of ConfigMap; enable env_auth and add path‑style.Embedding access keys in a ConfigMap is a leak. Switch to env-based auth and add
force_path_stylefor MinIO/Ceph.rclone.conf: |- [harbor-s3] type = s3 provider = $S3_PROVIDER - env_auth = false - access_key_id = $S3_KEY_ID - secret_access_key = $S3_ACCESS_KEY + env_auth = true endpoint = $S3_HOST acl = private + # Strongly recommended for most S3‑compatible providers + force_path_style = true # Add region configuration if your S3 service requires it # region = $S3_REGION
100-108: Use copy or dry‑run first; sync can delete destination objects.If the bucket isn’t empty,
syncwill delete extraneous keys. Prefercopyor require--dry-runfirst.- args: - - sync + args: + # Prefer 'copy' unless destination is guaranteed empty; or run '--dry-run' first. + - copy - /data - harbor-s3:$S3_BUCKET - --progress + # - --dry-run
115-119: Avoid relying on /root/.config; set explicit config path and inject Secret.Older rclone builds may ignore
--config; setRCLONE_CONFIGand mount a neutral path. Also consume thes3-credentialsSecret.- volumeMounts: - - mountPath: /root/.config/rclone/ - name: rclone-config + env: + - name: RCLONE_CONFIG + value: /etc/rclone/rclone.conf + envFrom: + - secretRef: + name: s3-credentials + volumeMounts: + - mountPath: /etc/rclone/ + name: rclone-config - mountPath: /data name: data
🧹 Nitpick comments (3)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (3)
16-16: Be explicit about supported versions.Clarify exact Harbor versions (e.g., “Alauda Build of Harbor 2.12.x”) and note any differences for OSS vs Operator-based installs.
109-114: Right-size resources or document guidance.Requests 1 CPU/1Gi vs limits 4 CPU/4Gi may overcommit on tight clusters. Provide sizing guidance or parameterize via env.
175-188: Prefer stringData for Secret; confirm required key names.Use
stringDatato avoid manual base64. Verify that the Operator indeed expects keysREGISTRY_STORAGE_S3_ACCESSKEY/SECRETKEY.- data: - REGISTRY_STORAGE_S3_ACCESSKEY: <base64-encoded-access-key> - REGISTRY_STORAGE_S3_SECRETKEY: <base64-encoded-secret-key> + stringData: + REGISTRY_STORAGE_S3_ACCESSKEY: "<access-key>" + REGISTRY_STORAGE_S3_SECRETKEY: "<secret-key>"
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-09-02T02:07:54.135Z
Learnt from: kycheng
PR: alauda/knowledge#49
File: docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md:111-121
Timestamp: 2025-09-02T02:07:54.135Z
Learning: The rclone sync command may not support --config parameter in all versions, despite documentation indicating it should be available as a global option. Alternative approaches include using RCLONE_CONFIG environment variable or configuring rclone entirely through environment variables.
Applied to files:
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
🪛 LanguageTool
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
[grammar] ~16-~16: There might be a mistake here.
Context: ...nvironment This solution is compatible with Alauda Build of Harbor v2.12.z. ## Res...
(QB_NEW_EN)
[grammar] ~24-~24: There might be a mistake here.
Context: ...System Settings → Repository Read Only`. - Important: Since Harbor needs to be se...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ...first, evaluate the migration time, and allocate sufficient maintenance window. - An S3-...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ... allocate sufficient maintenance window. - An S3-compatible storage service (MinIO,...
(QB_NEW_EN)
[grammar] ~175-~175: There might be a mistake here.
Context: ...ACCESSKEY: Base64-encoded S3 access key - REGISTRY_STORAGE_S3_SECRETKEY`: Base64-encoded S3 secret key ```yaml ...
(QB_NEW_EN)
🔇 Additional comments (2)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (2)
1-7: Front‑matter looks good.Metadata structure and kinds are valid for docs tooling.
214-221: Verification steps LGTM.Covers push/pull, new writes landing in S3, and legacy pulls.
| persistence: | ||
| enabled: true | ||
| # Add the following content | ||
| imageChartStorage: | ||
| disableredirect: true | ||
| s3: | ||
| existingSecret: s3-secret # an secret for S3 accesskey and secretkey | ||
| bucket: harbor # Storage bucket created in S3 cluster | ||
| region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph) | ||
| regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included | ||
| v4auth: true | ||
| type: s3 | ||
| # END |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Harbor S3 settings: add secure/pathstyle; keep other storage intact.
When regionendpoint is http, set secure: false. Many S3‑compatible backends need pathstyle: true. Add notes about self‑signed TLS (skipverify: true).
s3:
existingSecret: s3-secret # an secret for S3 accesskey and secretkey
bucket: harbor # Storage bucket created in S3 cluster
region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph)
regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included
v4auth: true
+ secure: false # set true if endpoint is https
+ pathstyle: true # recommended for MinIO/Ceph RGW
+ # skipverify: true # if using self-signed certs on https endpoint
type: s3📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| persistence: | |
| enabled: true | |
| # Add the following content | |
| imageChartStorage: | |
| disableredirect: true | |
| s3: | |
| existingSecret: s3-secret # an secret for S3 accesskey and secretkey | |
| bucket: harbor # Storage bucket created in S3 cluster | |
| region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph) | |
| regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included | |
| v4auth: true | |
| type: s3 | |
| # END | |
| persistence: | |
| enabled: true | |
| imageChartStorage: | |
| disableredirect: true | |
| s3: | |
| existingSecret: s3-secret # an secret for S3 accesskey and secretkey | |
| bucket: harbor # Storage bucket created in S3 cluster | |
| region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph) | |
| regionendpoint: http://xxxxx # S3 cluster access address; include port if needed | |
| v4auth: true | |
| secure: false # set true if using an HTTPS endpoint | |
| pathstyle: true # recommended for MinIO/Ceph RGW (path-style requests) | |
| # skipverify: true # if using self-signed certs on an HTTPS endpoint | |
| type: s3 |
🤖 Prompt for AI Agents
In docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md around
lines 199 to 211, the provided Harbor S3 config is missing explicit secure,
pathstyle and skipverify settings; update the example to (1) set secure: false
when regionendpoint uses http (otherwise true for https), (2) add pathstyle:
true for S3‑compatible backends that require path style addressing, and (3)
document skipverify: true as an optional setting for self‑signed TLS; do this
while keeping the rest of the storage configuration unchanged and add a brief
comment explaining when to use each flag.
…larifying steps and improving formatting
kycheng
force-pushed
the
feat/add-harbor-migrate-to-s3
branch
from
87569f6 to
f91b91d
Compare
kycheng
changed the title
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (7)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (7)
93-96: Allow a retry and auto-cleanup for the Job.Single-shot Job with no retry/cleanup is brittle.
spec: - backoffLimit: 0 + backoffLimit: 1 + ttlSecondsAfterFinished: 86400 template:
30-38: Replace brittle tarball URLs with a pinned OCI image (tag+digest).Publish or mirror a vetted rclone image and reference by digest.
-```txt -# Download URL for China Region -https://cloud.alauda.cn/attachments/knowledge/337969938/rclone-amd64.tgz -https://cloud.alauda.cn/attachments/knowledge/337969938/rclone-arm64.tgz - -# Download URLs for Other Regions -https://cloud.alauda.io/attachments/knowledge/337969938/rclone-amd64.tgz -https://cloud.alauda.io/attachments/knowledge/337969938/rclone-arm64.tgz -``` +```bash +# Use a pinned image you control +export SYNC_IMAGE=registry.local/ops/rclone:1.71.0@sha256:<digest> +```
206-213: Harbor S3 values: set secure for http endpoints; consider pathstyle; document skipverify.Improve interop with S3‑compatible backends.
s3: existingSecret: s3-secret # an secret for S3 accesskey and secretkey bucket: harbor # Storage bucket created in S3 cluster region: us-east-1 # S3 region (required for AWS S3, optional for MinIO/Ceph) regionendpoint: http://xxxxx # S3 cluster access address, note that the access port must be included v4auth: true + secure: false # set true if using https + # pathstyle: true # enable if your backend requires path-style (verify for your Harbor/registry version) + # skipverify: true # only if using self-signed TLS on https endpoints type: s3
24-26: Prevent RWO PVC mount conflicts; quiesce Harbor (registry) before running the Job.With typical RWO PVCs, the Job cannot mount the same registry PVC while Harbor is running. Read‑only UI mode doesn’t release the mount. Scale down at least harbor-registry (and other writers) before applying the Job and optionally run a brief final sync before cutover.
@@ - - **Important**: A fully deployed Harbor instance with `read-only mode` enabled. To enable read-only mode, Navigate to Harbor web `Administration → Configuration → System Settings → Repository Read Only`. - - **Important**: Since Harbor needs to be set to read-only mode during migration, it's recommended to simulate this process in a test environment first, evaluate the migration time, and allocate sufficient maintenance window. + - **Important**: Put Harbor in read-only mode and scale down write-producing components so the registry PVC is not mounted elsewhere: + ```bash + kubectl -n $HARBOR_NS scale deploy harbor-registry --replicas=0 + # Optional (safer): also scale down core/jobservice to eliminate background writes + # kubectl -n $HARBOR_NS scale deploy harbor-core harbor-jobservice --replicas=0 + ``` + - Recommended: Dry-run on a test environment, estimate duration, and reserve a maintenance window. @@ -kubectl apply -f sync-and-check-s3.yaml +kubectl apply -f sync-and-check-s3.yaml +# Optional final delta sync/check right before/after cutover if you didn’t fully quiesce.Also applies to: 152-153
60-61: Don’t export credentials into shell history; read from stdin and use a Secret.Move creds to a Kubernetes Secret; avoid plain exports.
-export S3_KEY_ID=xxxx -export S3_ACCESS_KEY=xxxxx +read -rp "S3 key id: " S3_KEY_ID +read -rsp "S3 access key: " S3_ACCESS_KEY && echoAdd this Secret before the ConfigMap/Job:
apiVersion: v1 kind: Secret metadata: name: s3-credentials namespace: $HARBOR_NS type: Opaque stringData: AWS_ACCESS_KEY_ID: $S3_KEY_ID AWS_SECRET_ACCESS_KEY: $S3_ACCESS_KEY ---
71-82: Stop embedding keys in a ConfigMap; enable env_auth and inject Secret; make config path explicit.Use env-based auth, set force_path_style for S3‑compatible endpoints, mount config at a neutral path, and pass RCLONE_CONFIG.
apiVersion: v1 data: rclone.conf: |- [harbor-s3] type = s3 provider = $S3_PROVIDER - env_auth = false - access_key_id = $S3_KEY_ID - secret_access_key = $S3_ACCESS_KEY + env_auth = true endpoint = $S3_HOST acl = private + force_path_style = true # Add region configuration if your S3 service requires it # region = $S3_REGION kind: ConfigMap @@ - - image: $SYNC_IMAGE + - image: $SYNC_IMAGE imagePullPolicy: IfNotPresent name: sync-data args: - sync - /data - harbor-s3:$S3_BUCKET - --progress + env: + - name: RCLONE_CONFIG + value: /etc/rclone/rclone.conf + envFrom: + - secretRef: + name: s3-credentials @@ - volumeMounts: - - mountPath: /root/.config/rclone/ - name: rclone-config + volumeMounts: + - mountPath: /etc/rclone/ + name: rclone-config - mountPath: /data name: data @@ - - image: $SYNC_IMAGE + - image: $SYNC_IMAGE imagePullPolicy: IfNotPresent name: check-sync args: - check - /data - harbor-s3:$S3_BUCKET - --one-way - --progress + env: + - name: RCLONE_CONFIG + value: /etc/rclone/rclone.conf + envFrom: + - secretRef: + name: s3-credentials @@ - volumeMounts: - - mountPath: /root/.config/rclone/ - name: rclone-config + volumeMounts: + - mountPath: /etc/rclone/ + name: rclone-config - mountPath: /data name: data @@ - - configMap: + - configMap: name: s3-config name: rclone-configAlso applies to: 103-120, 126-142, 143-149
104-107: Avoid accidental deletions; prefer copy (or dry-run sync) if bucket isn’t empty.sync deletes extraneous destination objects.
- - sync + # Prefer 'copy' unless the destination bucket is guaranteed empty. + - copy - /data - harbor-s3:$S3_BUCKET - - --progress + - --progress + # If using 'sync', run a dry-run first: + # - --dry-run
🧹 Nitpick comments (3)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (3)
109-114: Right-size resources and rclone concurrency.The 4x limits vs 1x requests gap is large; align with expected throughput and set transfers/checkers to match.
args: - copy - /data - harbor-s3:$S3_BUCKET - --progress + - --transfers=8 + - --checkers=16 @@ - limits: - cpu: 4 - memory: 4Gi - requests: - cpu: 1 - memory: 1Gi + requests: + cpu: "2" + memory: 2Gi + limits: + cpu: "2" + memory: 2Gi @@ - limits: - cpu: 4 - memory: 4Gi - requests: - cpu: 1 - memory: 1Gi + requests: + cpu: "2" + memory: 2Gi + limits: + cpu: "2" + memory: 2GiPlease verify on your cluster (disk/network) and adjust.
Also applies to: 131-137
172-191: Add “Resume Harbor” steps post-cutover.Close the loop: switch off read-only and scale deployments back up.
#### Update Harbor Configuration to Use S3 Storage @@ After successfully migrating the data, update the Harbor configuration to use S3 storage instead of PVC. This step configures Harbor to read and write registry data directly from/to the S3 bucket. @@ ### Verification and Testing @@ 3. **Verify Existing Images**: Ensure that previously migrated images can still be pulled successfully + +### Resume Harbor +After Harbor is configured to use S3 and verification passes: +```bash +# Disable read-only mode in UI (or via API), then: +kubectl -n $HARBOR_NS scale deploy harbor-registry harbor-core harbor-jobservice --replicas=1 +```Also applies to: 217-224
16-16: Minor wording/grammar nits.Tighten phrasing.
-This solution is compatible with Alauda Build of Harbor v2.12.z. +This solution is compatible with Alauda Build of Harbor v2.12.z. @@ -- **Important**: A fully deployed Harbor instance with `read-only mode` enabled. To enable read-only mode, Navigate to Harbor web `Administration → Configuration → System Settings → Repository Read Only`. -- **Important**: Since Harbor needs to be set to read-only mode during migration, it's recommended to simulate this process in a test environment first, evaluate the migration time, and allocate sufficient maintenance window. +- **Important**: A fully deployed Harbor instance with `read-only mode` enabled. To enable it, navigate to `Administration → Configuration → System Settings → Repository Read Only`. +- **Important**: Because Harbor must be in read-only mode during migration, first simulate the process in a test environment, estimate the duration, and allocate a sufficient maintenance window. @@ -- `REGISTRY_STORAGE_S3_ACCESSKEY`: Base64-encoded S3 access key -- `REGISTRY_STORAGE_S3_SECRETKEY`: Base64-encoded S3 secret key +- `REGISTRY_STORAGE_S3_ACCESSKEY`: Base64‑encoded S3 access key +- `REGISTRY_STORAGE_S3_SECRETKEY`: Base64‑encoded S3 secret keyAlso applies to: 24-25, 178-180
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-09-02T02:07:54.135Z
Learnt from: kycheng
PR: alauda/knowledge#49
File: docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md:111-121
Timestamp: 2025-09-02T02:07:54.135Z
Learning: The rclone sync command may not support --config parameter in all versions, despite documentation indicating it should be available as a global option. Alternative approaches include using RCLONE_CONFIG environment variable or configuring rclone entirely through environment variables.
Applied to files:
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
🪛 LanguageTool
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md
[grammar] ~16-~16: There might be a mistake here.
Context: ...nvironment This solution is compatible with Alauda Build of Harbor v2.12.z. ## Res...
(QB_NEW_EN)
[grammar] ~24-~24: There might be a mistake here.
Context: ...System Settings → Repository Read Only`. - Important: Since Harbor needs to be se...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ...first, evaluate the migration time, and allocate sufficient maintenance window. - An S3-...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ... allocate sufficient maintenance window. - An S3-compatible storage service (MinIO,...
(QB_NEW_EN)
[grammar] ~178-~178: There might be a mistake here.
Context: ...ACCESSKEY: Base64-encoded S3 access key - REGISTRY_STORAGE_S3_SECRETKEY`: Base64-encoded S3 secret key ```yaml ...
(QB_NEW_EN)
🔇 Additional comments (1)
docs/en/solutions/How_to_migrate_harbor_registry_pvc_storage_to_s3.md (1)
64-64: Good: rclone image is pinned to a version.Avoids drift from :latest.
* feat：add migrate harbor registry pvc storage to s3. * docs: update migration guide for Harbor registry PVC storage to S3, clarifying steps and improving formatting
4 participants
Summary by CodeRabbit
issue id：DEVOPS-42001