Can we use milvus-backup with GCS storage and that too without minio scop in config.

[navneetrajput06]

Hi Team,

Can we use milvus-backup with GCS storage and that too without minio scop in config. i am using gcs storage as my milvus is running on GCP.
Could you please let me know gcp configuration for milvus-backup. as of now i am using below and getting workings like [WARN] [storage/minio_chunk_manager.go:97] ["failed to check blob bucket exist"] .

config:backup.yaml
[2023/10/09 17:30:18.218 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2023/10/09 17:30:18.219 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2023/10/09 17:31:18.123 +00:00] [INFO] [core/backup_impl_create_backup.go:25] ["receive CreateBackupRequest"] [requestId=a79758f4-66c9-11ee-b62b-a2a888313c5e] [backupName=test_backup] [collections="[test]"] [databaseCollections=] [async=true]
[2023/10/09 17:31:18.468 +00:00] [WARN] [storage/minio_chunk_manager.go:97] ["failed to check blob bucket exist"] [bucket=] [error="Get "https://support.google.com/accounts/answer/32050?hl=en": stopped after 5 redirects"]
[2023/10/09 17:31:18.886 +00:00] [WARN] [storage/minio_chunk_manager.go:97] ["failed to check blob bucket exist"] [bucket=] [error="XML syntax error on line 3: invalid character entity & (no semicolon)"]
[2023/10/09 17:31:19.489 +00:00] [WARN] [storage/minio_chunk_manager.go:97] ["failed to check blob bucket exist"] [bucket=***********] [error="XML syntax error on line 3: invalid character entity & (no semicolon)"]

Below is the configuration:

Related configuration of minio, which is responsible for data persistence for Milvus.

minio:
cloudProvider: "gcp"
address: storage.cloud.google.com
port: 443
#serviceAccountName: milvus
accessKeyID: "" # accessKeyID of MinIO/S3
secretAccessKey: "" # MinIO/S3 encryption string
useSSL: true
useIAM: true
iamEndpoint: "storage.googleapis.com:443"

bucketName: "bucket11" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
rootPath: "milvus/s3-access" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance

backupBucketName: "bucket2" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
backupRootPath: "milvus/gcs-backup-access" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath

[filip-halt]

@LoveEachDay

[yhmo]

The backup tool has a configuration file: https://github.com/zilliztech/milvus-backup/blob/0691b272439b52e6d6aba6acba2cc001b8765b02/configs/backup.yaml#L24

The error "XML syntax error on line 3: invalid character entity & (no semicolon)" indicates the bucket name or endpoint address is incorrect.

[navneetrajput06]

Thanks team i was able to run this successfully. its was endpoint issue as you said.

[shauryagoel]

@navneetrajput06 what exact change did you make to resolve your issue? I am also facing the same issue.

[navneetrajput06]

@shauryagoel it was just the syntax issue for me.

[Yashwanth1712094]

@navneetrajput06
can you please provide the config file for backup from local milvus minio to a gcp bucket.