Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vmBackup storage mount path not consistent with vmstorage #349

Closed
wu0407 opened this issue Oct 8, 2021 · 5 comments
Closed

vmBackup storage mount path not consistent with vmstorage #349

wu0407 opened this issue Oct 8, 2021 · 5 comments
Labels
bug Something isn't working

Comments

@wu0407
Copy link

wu0407 commented Oct 8, 2021

vmstorage-db is mount in /vm-data with vmstorage container, but that is mount in /victoria-metrics-data with vmbackup container.
operator version: v0.19.1
vmcluster configure:

 vmstorage:
    extraArgs:
      search.maxUniqueTimeseries: "600000"
    image:
      tag: v1.66.2-cluster
    replicaCount: 2
    resources:
      limits:
        cpu: "4"
        memory: 16Gi
      requests:
        cpu: 50m
        memory: 1Gi
    storage:
      volumeClaimTemplate:
        metadata: {}
        spec:
          resources:
            requests:
              storage: 100Gi
          storageClassName: cbs-ssd
        status: {}
    storageDataPath: /vm-data
    vmBackup:
      acceptEULA: true
      credentialsSecret:
        key: credentials-backup.yaml
        name: credentials-backup
      destination: s3://test/$(NODENAME)
      disableMonthly: true
      disableWeekly: true
      extraArgs:
        customS3Endpoint: https://cos.ap-shanghai.myqcloud.com
        keepLastDaily: "30"
        keepLastHourly: "72"
        runOnStart: "true"
      extraEnvs:
      - name: AWS_DEFAULT_REGION
        value: ap-shanghai
      - name: NODENAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name

result statefulset:

 spec:
      containers:
      - args:
        - -dedup.minScrapeInterval=1ms
        - -httpListenAddr=:8482
        - -retentionPeriod=14
        - -search.maxUniqueTimeseries=600000
        - -storageDataPath=/vm-data
        - -vminsertAddr=:8400
        - -vmselectAddr=:8401
        image: victoriametrics/vmstorage:v1.66.2-cluster
        imagePullPolicy: IfNotPresent
        name: vmstorage
         ...........
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /vm-data
          name: vmstorage-db
      - args:
        - -credsFilePath=/etc/vm/creds/credentials-backup.yaml
        - -customS3Endpoint=https://cos.ap-shanghai.myqcloud.com
        - -disableMonthly
        - -disableWeekly
        - -dst=s3://test/$(NODENAME)
        - -envflag.enable=true
        - -eula
        - -keepLastDaily=30
        - -keepLastHourly=72
        - -runOnStart=true
        - -snapshot.createURL=http://localhost:8482/snapshot/create
        - -snapshot.deleteURL=http://localhost:8482/snapshot/delete
        - -storageDataPath=/vm-data
        env:
        - name: AWS_DEFAULT_REGION
          value: ap-shanghai
        - name: NODENAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: victoriametrics/vmbackupmanager:v1.66.2-enterprise
        imagePullPolicy: IfNotPresent
        name: vmbackuper
    .  ...............
        volumeMounts:
        - mountPath: /victoria-metrics-data
          name: vmstorage-db
          readOnly: true
        - mountPath: /etc/vm/creds
          name: secret-credentials-backup
          readOnly: true

if set storageDataPath in extraArgs, then duplication options of storageDataPath

   extraArgs:
        customS3Endpoint: https://cos.ap-shanghai.myqcloud.com
        keepLastDaily: "30"
        keepLastHourly: "72"
        runOnStart: "true"
        storageDataPath: /victoria-metrics-data

result statefulset:

  - args:
        - -credsFilePath=/etc/vm/creds/credentials-backup.yaml
        - -customS3Endpoint=https://cos.ap-shanghai.myqcloud.com
        - -disableMonthly
        - -disableWeekly
        - -dst=s3://test/$(NODENAME)
        - -envflag.enable=true
        - -eula
        - -keepLastDaily=30
        - -keepLastHourly=72
        - -runOnStart=true
        - -snapshot.createURL=http://localhost:8482/snapshot/create
        - -snapshot.deleteURL=http://localhost:8482/snapshot/delete
        - -storageDataPath=/victoria-metrics-data
        - -storageDataPath=/vm-data
@f41gh7 f41gh7 added the bug Something isn't working label Oct 8, 2021
f41gh7 added a commit that referenced this issue Oct 8, 2021
#349

Fixes cluster version backups, adds suffix with POD_NAME
it must prevent race condition for cluster backups
@f41gh7
Copy link
Collaborator

f41gh7 commented Oct 8, 2021

Thanks for reporting!

Must be fixed at commit cdba28e4b9914f19c0fbb9d8794bce1692d5228d. Will be included into the next release

@wu0407
Copy link
Author

wu0407 commented Oct 9, 2021

dynamic backup path is feature, but this issue not associate with dynamic backup.
default vmbacker container volume mount path is /victoria-metrics-data and args - -storageDataPath=/vm-data :

volumeMounts:
        - mountPath: /victoria-metrics-data
          name: vmstorage-db
          readOnly: true

vmstorage container volume mount path is /vm-data

volumeMounts:
        - mountPath: /vm-data
          name: vmstorage-db

@wu0407
Copy link
Author

wu0407 commented Oct 9, 2021

This issue is cause by MountPath is fixed with vmSingleDataDir

mounts := []corev1.VolumeMount{
{
Name: dataVolumeName,
MountPath: vmSingleDataDir,
ReadOnly: true,
},

if cr.Spec.VMStorage.VMBackup != nil {
vmBackupManagerContainer, err := makeSpecForVMBackuper(cr.Spec.VMStorage.VMBackup, c, cr.Spec.VMStorage.Port, cr.Spec.VMStorage.StorageDataPath, cr.Spec.VMStorage.GetStorageVolumeName(), cr.Spec.VMStorage.ExtraArgs)
if err != nil {
return nil, err
}
if vmBackupManagerContainer != nil {
operatorContainers = append(operatorContainers, *vmBackupManagerContainer)
}
}

@f41gh7
Copy link
Collaborator

f41gh7 commented Oct 10, 2021

Yep, it was fixed at related commit:

cdba28e#diff-5fa80506fe946da51e98eb94e27e68d650cc0c5014909c9171bc7811e196393fR453

f41gh7 added a commit that referenced this issue Oct 12, 2021
* fixes issue with incosistent name for backupmanager folder
#349

Fixes cluster version backups, adds suffix with POD_NAME
it must prevent race condition for cluster backups

* updates readme with eula

* updates readme
@f41gh7
Copy link
Collaborator

f41gh7 commented Oct 28, 2021

Fix added to the release version with v0.20.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants