Using medusa with local storage throws error in medusa container

[Oro]

Bug Report

Describe the bug
Enabling medusa with "local" as storage for testing purpose leads to the following error in the medusa container:

MEDUSA_MODE = GRPC
sleeping for 0 sec
Starting Medusa gRPC service
/home/cassandra/.local/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.3) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
ERROR:root:Required configuration "bucket_name" is missing in [storage] section.

To Reproduce

1. Use a values.yaml with at least the following:

medusa:
  enabled: true
  storage: local
  bucketName: /tmp

This will result in the cassandra pod never fully starting as the medusa container inside will crash.

Expected behavior
Medusa container coming up normally

Screenshots
n/a

Environment (please complete the following information):

• Helm charts version info

Installed via ArgoCD (i.e. through helm template | kubectl apply -f -). Therefore no helm ls directly possible.
Chart version 1.0.0

• Helm charts user-supplied values

See above. Full values.yaml (which I am still trying to get up and running, so please excuse anything seemingly obviously wrong - this should(tm) however not matter for this report since Medusa will not start without bucket_name in any case.)

        cassandra:
          # -- The Cassandra version to use. The supported versions include the following:
          #    - 3.11.7
          #    - 3.11.8
          #    - 3.11.9
          #    - 3.11.10
          #    - 4.0.0
          version: "4.0.0"

          # -- Specifies the image to use for a particular Cassandra version. Exercise
          # care and caution with changing these values! cass-operator is not designed to work with
          # arbitrary Cassandra images. It expects the cassandra container to be running
          # management-api images. If you do want to change one of these mappings, the new value
          # should be a management-api image.
          versionImageMap:
            3.11.7: datastax/cassandra-mgmtapi-3_11_7:v0.1.22
            3.11.8: datastax/cassandra-mgmtapi-3_11_8:v0.1.22
            3.11.9: datastax/cassandra-mgmtapi-3_11_9:v0.1.22
            3.11.10: datastax/cassandra-mgmtapi-3_11_10:v0.1.22
            4.0.0: datastax/cassandra-mgmtapi-4_0_0:v0.1.22

          # -- Overrides the default image mappings. This is intended for advanced use cases
          # like development or testing. By default the Cassandra version has to be one that is in
          # versionImageMap. Template rendering will fail if the version is not in the map. When
          # you set the image directly, the version mapping check is skipped. Note that you are
          # still constrained to the versions supported by cass-operator.
          #
          # image:

          # -- Cluster name defaults to release name when not specified.
          clusterName: "cassandra"

          # -- Authentication and authorization related settings.
          auth:
            enabled: true

            # -- Configures the default Cassandra superuser when authentication is
            # enabled. If neither `superuser.secret` nor `superuser.username` are set,
            # then a user and a secret with the user's credentials will be created. The
            # username and secret name will be of the form {clusterName}-superuser. The
            # password will be a random 20 character password. If `superuser.secret` is
            # set, then the Cassandra user will be created from the contents of the
            # secret. If `superuser.secret` is not set and if `superuser.username` is
            # set, a secret will be generated using the specified username. The
            # password will be generated as previously described.
            #
            # JMX credentials will also be created for the superuser. The same
            # username/password that is used here will be used for JMX. If you change
            # the Cassandra superuser credentials through cqlsh for example, the JMX
            # credentials will not be updated. You need to update the credentials via
            # helm upgrade in order for the change to propagate to JMX. This will be
            # fixed in https://github.com/k8ssandra/k8ssandra/issues/323.
            superuser:
              secret: ""
              username: ""
          cassandraLibDirVolume:
            storageClass: standard
            size: 5Gi

          # -- Permits running multiple Cassandra pods per Kubernetes worker. If enabled
          # resources.limits and resources.requests **must** be defined.
          allowMultipleNodesPerWorker: true

          # -- Resource requests for each Cassandra pod.
          resources:
            limits:
              cpu: 500m
              memory: 2Gi
            requests:
              cpu: 250m
              memory: 2Gi


          datacenters:
          - name: eu-central-1

            # -- Number of nodes within the datacenter. This value should, at a minimum,
            # match the number of racks and be no less than 3 for non-development
            # environments.
            size: 1
            # -- Specifies the racks for the data center, if unset the datacenter will
            # be composed of a single rack named `default`. The number of racks should
            # equal the replication factor of your application keyspaces. Cassandra will
            # ensure that replicas are spread across racks versus having multiple
            # replicas within the same rack. For example, let's say we are using RF = 3
            # with a 9 node cluster and 3 racks (and 3 nodes per rack). There will be
            # one replica of the dataset spread across each rack.
            racks:
            -
              # -- Identifier for the rack, this may align with the labels used to
              # control where resources are deployed for this rack. For example, if a
              # rack is limited to a single availability zone the identifier may be the
              # name of that AZ (eg us-east-1a).
              name: eu-central-1a

              # -- an optional set of labels that are used to pin Cassandra pods to
              # specific k8s worker nodes via affinity rules. See
              # https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/
              # for background on using affinity rules.
              #
              # topology.kubernetes.io/zone is a well-known k8s label used by cloud
              # providers to indicate the failure zone in which a k8s worker node is
              # running. The following example illustrates how you can pin racks to
              # specific failure zones.
              affinityLabels: {}

          # Cassandra native transport ingress support
          ingress:
            enabled: false

        stargate:
          enabled: false

        reaper:
          # -- When enabled, Reaper automatically sets up repair schedules for all
          # non-system keypsaces. Repear monitors the cluster so that as keyspaces are
          # added or removed repair schedules will be added or removed respectively.
          autoschedule: true

          # -- Enable Reaper resources as part of this release. Note that Reaper uses
          # Cassandra's JMX APIs to perform repairs. When Reaper is enabled, Cassandra
          # will also be configured to allow remote JMX access. JMX authentication
          # will be configured in Cassandra with credentials only created for Reaper
          # in order to limit access.
          enabled: true

          # -- Configures the Cassandra user used by Reaper when authentication is
          # enabled. If neither cassandraUser.secret nor casandraUser.username are
          # set, then a Cassandra user and a secret with the user's credentials will
          # be created. The username will be reaper. The secret name will be of the
          # form {clusterName}-reaper. The password will be a random 20 character
          # password. If cassandraUser.secret is set, then the Cassandra user will be
          # created from the contents of the secret. If cassandraUser.secret is not
          # set and if cassandraUser.username is set, a secret will be generated using
          # the specified username. The password will be generated as previously
          # described.
          cassandraUser:
            secret: ""
            username: ""

          # -- Configures JMX access to the Cassandra cluster. Reaper requires remote
          # JMX access to perform repairs. The Cassandra cluster will be configured
          # with remote JMX access enabled when Reaper is deployed. The JMX access
          # will be configured to use authentication.
          #
          # If neither `jmx.secret` nor `jmx.username` are set, then a default user and
          # secret with the user's credentials will be created.
          jmx:
            secret: ""
            # -- Username that Reaper will use for JMX access. If left blank a random,
            # alphanumeric string will be generated.
            username: ""

        medusa:
          # -- Enable Medusa resources as part of this release. If enabled,
          # `bucketName` and `bucketSecret` **must** be defined.
          enabled: true

          # Configures the Medusa image which is built from
          # https://github.com/thelastpickle/cassandra-medusa/tree/master/k8s.
          image:
            # -- Specifies the container repository for Medusa
            repository: docker.io/k8ssandra/medusa
            # -- Tag of an image within the specified repository
            tag: 0.9.0
            # -- The image pull policy
            pullPolicy: IfNotPresent

          # -- Configures the Cassandra user used by Medusa when authentication is
          # enabled. If neither `cassandraUser.secret` nor `casandraUser.username` are
          # set, then a Cassandra user and a secret will be created. The username will
          # be medusa. The secret name will be of the form {clusterName}-medusa. The
          # password will be a random 20 character password. If `cassandraUser.secret`
          # is set, then the Cassandra user will be created from the contents of the
          # secret. If `cassandraUser.secret` is not set and if
          # `cassandraUser.username` is set, a secret will be generated using the
          # specified username. The password will be generated as previously
          # described.
          cassandraUser:
            secret: ""
            username: ""

          # -- Enables usage of a bucket across multiple clusters.
          multiTenant: false

          # -- API interface used by the object store. Supported values include `s3`, 's3_compatible'
          # and `gcs`. For pod mapped storage, use 'local'
          storage: local

          # -- Optional properties for storage. Supported values depend on the type of the storage.
          storage_properties: {}
            # Define region for s3 / s3_compatible
            # region: eu-west-1

            # For s3_compatible option, one must define target host
            # host: 192.168.1.201

            # For s3_compatible option, port is optional
            # port: 9000

          # Is SSL used or not for s3_compatible connection
          # secure: false

          # -- Name of the remote storage bucket where backups will be stored. If using 'local' storage, set the
          # path of the local storage here.
          bucketName: /tmp

          # -- Name of the Kubernetes `Secret` that stores the key file for the
          # storage provider's API
          storageSecret: medusa-bucket-key

        monitoring:
          grafana:
            # -- Enables the creation of configmaps containing Grafana dashboards. If
            # leveraging the kube prometheus stack sub-chart this value should be
            # `true`.
            provision_dashboards: false

          prometheus:
            # -- Enabes the creation of Prometheus Operator ServiceMonitor custom
            # resources. If you are not using the kube prometheus stack sub-chart or do
            # not have the ServiceMonitor CRD installed on your cluster, set this value
            # to `false`.
            provision_service_monitors: false

        cleaner:
          image: k8ssandra/k8ssandra-cleaner:618b8ff9d368

        cass-operator:
          enabled: true

        reaper-operator:
          enabled: true

        kube-prometheus-stack:
          enabled: false

          # Disable default service monitors
          coreDns:
            enabled: false
          kubeApiServer:
            enabled: false
          kubeControllerManager:
            enabled: false
          kubeDns:
            enabled: false
          kubeEtcd:
            enabled: false
          kubeProxy:
            enabled: false
          kubeScheduler:
            enabled: false
          kubeStateMetrics:
            enabled: false
          kubelet:
            enabled: false
          nodeExporter:
            enabled: false

          alertmanager:
            # Disabled for now while we build out a set of default alerts
            enabled: false
            serviceMonitor:
              selfMonitor: false

          prometheusOperator:
            enabled: false

          prometheus:
            # -- Provisions an instance of Prometheus as part of this release
            enabled: true

            # -- Allows for tweaking of the Prometheus installation's configuration.
            # Common parameters include `externalUrl: http://localhost:9090/prometheus`
            # and `routePrefix: /prometheus` for running Prometheus resources under a
            # specific path (`/prometheus` in this example).
            prometheusSpec:
              # -- Prefixes all Prometheus routes with the specified value. It is useful
              # for ingresses which do not rewrite URLs.
              routePrefix: /

              # -- An external URL at which Prometheus will be reachable.
              externalUrl: ""

            ingress:
              # -- Enable templating of ingress resources for external prometheus
              # traffic
              enabled: false
              # -- Path-based routing rules, `/prometheus` is possible if the
              # appropriate changes are made to `prometheusSpec`
              paths: []

            serviceMonitor:
              # Disable monitoring the Prometheus instance
              selfMonitor: false

          grafana:
            # -- Provisions an instance of Grafana and wires it up with a DataSource
            # referencing this Prometheus installation
            enabled: true
            ingress:
              # -- Generates ingress resources for the Grafana instance
              enabled: false

              # -- Path-based routing rules, '/grafana' is possible if appropriate
              # changes are made to `grafana.ini`
              path:

            # -- Username for accessing the provisioned Grafana instance
            adminUser: admin

            # -- Password for accessing the provisioned Grafana instance
            adminPassword: secret

            serviceMonitor:
              # -- Whether the Grafana instance should be monitored
              selfMonitor: false

            # -- Default dashboard installation
            defaultDashboardsEnabled: false

            # -- Additional plugins to be installed during Grafana startup,
            # `grafana-polystat-panel` is used by the default Cassandra dashboards.
            plugins:
              - grafana-polystat-panel

            # -- Customization of the Grafana instance. To listen for Grafana traffic
            # under a different url set `server.root_url: http://localhost:3000/grafana`
            # and `serve_from_sub_path: true`.
            grafana.ini: {}

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-rc.0", GitCommit:"e338cf2c6d297aa603b50ad3a301f761b4173aa6", GitTreeState:"archive", BuildDate:"1980-01-01T00:00:00Z", GoVersion:"go1.16", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:47:43Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

minikube with kvm2

Additional context

This is most likely an upstream issue, however since this Chart explicitly switches between local and all else providers I think it is correct here.
Workaround is simply to add a bucket_name in additional storage options

medusa:
  enabled: true
  storage: local
  bucketName: /tmp
  storage_properties:
    bucket_name: backups

Interestingly enough: The storage-secret needs to be set (and available) even when using local, probably also not intended.

[jsanda]

Hi @Oro

Thanks for creating this ticket.

I think you are right about bucket_name being required is an upstream issue. I will create a ticket for Medusa to address this and link it back here.

We definitely should not be requiring the secret to be set for local storage.

Also keep in mind that for local you will need a shared storage solution.

[jsanda]

I created thelastpickle/cassandra-medusa#299 for the bucket_name property being required with local storage.

Also note that with local storage we need to set the base_path property in medusa.ini.