Error when specifying a storage engine (rocksdb)

[de-code]

I'm trying to use another storage engine, RocksDB. I'm passing:
--server.storage-engine rocksdb

Which results in:

FATAL {startup} content of 'ENGINE' file '/var/lib/arangodb3/ENGINE' and command-line/configuration option value do not match: 'mmfiles' != 'rocksdb'. please validate the command-line/configuration option value of '--server.storage-engine' or use a different database directory if the change is intentional

Which is I believe because that argument is not passed through when the database is initialised:

arangodb-docker/jessie/3.1.8/docker-entrypoint.sh

Lines 41 to 46 in 4f5bb42

	echo "Initializing database...Hang on..."
	arangod --server.endpoint unix:///tmp/arangodb-tmp.sock \
	--server.authentication false \
	--log.file /tmp/init-log \
	--log.foreground-tty false &

[dothebart]

Hi,
you have to pass the environment variable ARANGO_STORAGE_ENGINE with the content rocksdb to start an instance with rocksdb.

[de-code]

Thank you. That seem to work.

It doesn't seem to be documented?
The docker image documentation itself refers to the following command for the supported options:
https://hub.docker.com/_/arangodb/

docker run -e ARANGO_RANDOM_ROOT_PASSWORD=1 arangodb arangod --help

Might be worth updating it?

[dothebart]

docker-library/docs#1037 yes.
;-)

[haseebnaseem]

facing the same issue while deploying on kubernetes using the docker image. Setting the environment variable does not resolve the error.

choosing Rocksdb storage engine
2017-11-23T20:51:27Z [1] INFO ArangoDB 3.2.8 [linux] 64bit, using jemalloc, VPack 0.1.30, RocksDB 5.6.0, ICU 58.1, V8 5.7.492.77, OpenSSL 1.1.0f  25 May 2017
2017-11-23T20:51:27Z [1] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-23T20:51:27Z [1] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2017-11-23T20:51:27Z [1] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2017-11-23T20:51:27Z [1] FATAL {startup} content of 'ENGINE' file '/var/lib/arangodb3/ENGINE' and command-line/configuration option value do not match: 'mmfiles' != 'rocksdb'. please validate the command-line/configuration option value of '--server.storage-engine' or use a different database directory if the change is intentional

[de-code]

@haseebnaseem did you clear the data after changing the storage engine?

[haseebnaseem]

@de-code yes. Fresh install.

[dothebart]

@sbaugher can you have a look at this?
@haseebnaseem we have seen cases where docker wouldn't pass on the password environment variable into the containers until you'd purge the container and re-download it

See these github issues:
arangodb/arangodb#3074 arangodb/arangodb#2712

[dothebart]

As an additional workaround you could try to specify it as commandline argument - --server.storage-engine rocksdb - see arangodb/arangodb#3634 for more details.

[haseebnaseem]

@dothebart I am doing both of these things. Will share the configuration files as well if you want.

[dothebart]

Do you mount in /var/lib/arangodb3 from a storage volume that keeps state across restarts from docker engines? Does that keep the data from the previous run with the other storage engine?

[haseebnaseem]

I deleted the previous installation and did a fresh install. Here is the config file:

---
apiVersion: v1
kind: Service
metadata:
  name: arangodb-agents
spec:
  ports:
    - port: 8529
      targetPort: 8529
  clusterIP: None
  selector:
    name: arangodb
    role: agent
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: arangodb-agent
spec:
  serviceName: "arangodb-agents"
  replicas: 3
  template:
    metadata:
      labels:
        name: arangodb
        role: agent
    spec:
      containers:
        - name: arangodb
          image: arangodb/arangodb:latest
          env:
            - name: ARANGO_STORAGE_ENGINE
              value: "rocksdb"
            - name: IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: ARANGO_NO_AUTH
              value: "1"
          ports:
            - containerPort: 8529
          volumeMounts:
            - mountPath: /var/lib/arangodb3
              name: arangodb-agency-data
          args:
            - --server.storage-engine
            - "rocksdb"
            - --server.authentication
            - "false"
            - --server.endpoint
            - tcp://0.0.0.0:8529
            - --agency.activate
            - "true"
            - --agency.size
            - "3"
            - --agency.supervision
            - "true"
            - --agency.my-address
            - tcp://$(IP):8529
            - --agency.endpoint
            - tcp://arangodb-agent-0.arangodb-agents.default.svc.cluster.local:8529
  volumeClaimTemplates:
    - metadata:
        name: arangodb-agency-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: arangodb-coords
spec:
  ports:
    - port: 8529
      targetPort: 8529
  type: LoadBalancer
  selector:
    name: arangodb
    role: coordinator
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: arangodb-coord
spec:
  serviceName: "arangodb-coords"
  replicas: 2
  template:
    metadata:
      labels:
        name: arangodb
        role: coordinator
    spec:
      containers:
        - name: arangodb
          image: arangodb/arangodb:latest
          env:
            - name: ARANGO_STORAGE_ENGINE
              value: "rocksdb"
            - name: IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: ARANGO_NO_AUTH
              value: "1"
          ports:
            - containerPort: 8529
          volumeMounts:
            - mountPath: /var/lib/arangodb3
              name: arangodb-coords-data
          args:
            - --server.storage-engine
            - "rocksdb"
            - --server.authentication
            - "false"
            - --server.endpoint
            - tcp://0.0.0.0:8529
            - --cluster.my-role
            - COORDINATOR
            - --cluster.my-local-info
            - "$(IP)"
            - --cluster.my-address
            - tcp://$(IP):8529
            - --cluster.agency-endpoint
            - tcp://arangodb-agent-0.arangodb-agents.default.svc.cluster.local:8529
  volumeClaimTemplates:
    - metadata:
        name: arangodb-coords-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: arangodb-dbs
spec:
  ports:
    - port: 8529
      targetPort: 8529
  type: LoadBalancer
  selector:
    name: arangodb
    role: dbserver
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: arangodb-db
spec:
  serviceName: "arangodb-dbs"
  replicas: 3
  template:
    metadata:
      labels:
        name: arangodb
        role: dbserver
    spec:
      containers:
        - name: arangodb
          image: arangodb/arangodb:latest
          env:
            - name: ARANGO_STORAGE_ENGINE
              value: "rocksdb"
            - name: IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: ARANGO_NO_AUTH
              value: "1"
          ports:
            - containerPort: 8529
          volumeMounts:
            - mountPath: /var/lib/arangodb3
              name: arangodb-db-data
            - mountPath: /var/lib/arangodb3-apps
              name: arangodb-db-apps
          args:
            - --server.storage-engine
            - "rocksdb"
            - --server.authentication
            - "false"
            - --server.endpoint
            - tcp://0.0.0.0:8529
            - --cluster.my-role
            - PRIMARY
            - --cluster.my-local-info
            - "$(IP)"
            - --cluster.my-address
            - tcp://$(IP):8529
            - --cluster.agency-endpoint
            - tcp://arangodb-agent-0.arangodb-agents.default.svc.cluster.local:8529
  volumeClaimTemplates:
    - metadata:
        name: arangodb-db-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi
    - metadata:
        name: arangodb-db-apps
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

[dothebart]

1. can you check in any way whether these volumes you mount are empty?
2. you need to start all nodes (the agents too) in rocksdb mode.

[haseebnaseem]

@dothebart I am starting all the nodes with rocksdb mode. I can't check if they are empty because the container doesn't start due to the fatal error so no way to access the shell but since the container does recognize the "rocksdb" mode as seen in the choosing rocksdb as the storage engine echo from the container, it doesn't seem to be the case.