RocksDB causes error when mounting a tmpfs to /tmp

[jcaesar]

Adding a tmpfs: /tmp entry to a docker-compose file will make the RocksDB backend fail to load.

Example compose file:

version: '2.0'

services:
  flank:
    image: flink:1.5.1-hadoop28-scala_2.11
    entrypoint: |
      bash -c "
        echo 'state.backend: rocksdb' >> conf/flink-conf.yaml;
        echo 'state.backend.fs.checkpointdir: file:///tmp/checkpoints' >> conf/flink-conf.yaml;
        taskmanager.sh start-foreground local &
        jobmanager.sh start-foreground local &
        sleep 10;
        flink run examples/streaming/WindowJoin.jar
      "
    ports:
     - "8081:8081"
    tmpfs: /tmp

The output will contain an error like java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-[...]/librocksdbjni-linux64.so: /tmp/rocksdb-lib-[...]/librocksdbjni-linux64.so: failed to map segment from shared object

I'm guessing that this is unrelated to #14 / #37.

I'm not all that familiar with JNI and shared object loading under linux, so I'm not sure this is intended to be a possible configuration.

[patricklucas]

A search for tmpfs "failed to map segment from shared object" yielded this discussion: docker/compose#1339

Apparently docker-compose (or Docker?) mounts tmpfs's with the flag noexec for security reasons, which means libs within the mount can't be dynamically linked. The discussion above suggests a few workarounds, such as remounting /tmp with exec, or configuring the application to use an alternate tmp dir that is executable.

As a PoC, I modified your docker-compose.yaml like so and the example job ran successfully:

diff --git a/docker-compose.yaml b/docker-compose.yaml
index 486ae15..5ba9cd0 100644
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -3,8 +3,12 @@ version: '2.0'
 services:
   flank:
     image: flink:1.5.1-hadoop28-scala_2.11
+    environment:
+    - JVM_ARGS=-Djava.io.tmpdir=/foo
     entrypoint: |
       bash -c "
+        mkdir /foo
+        chmod 0777 /foo
         echo 'state.backend: rocksdb' >> conf/flink-conf.yaml;
         echo 'state.backend.fs.checkpointdir: file:///tmp/checkpoints' >> conf/flink-conf.yaml;
         taskmanager.sh start-foreground local &

[jcaesar]

Thank you for assisting with my insufficient google-fu.
The solution seems to be to specify the tmpfs as

    tmpfs:
      - /tmp:exec,mode=777

(cf. docker/compose#3425) and this is simply a configuration error on my part.

Playing around a bit with

docker run --rm -ti --entrypoint=bash --tmpfs=/tmp:exec,mode=777 flink:1.5.1-hadoop28-scala_2.11 -c "findmnt -fPO +noexec /tmp | wc -l"

made me think you could add a warning/check for this, but I have doubts whether this is worth it. Tell me if you want a PR.

[patricklucas]

I think the right place for a check would be in Flink itself, somewhere in the invocation scripts here. I don't have a strong opinion whether or not to introduce such a check.

Thanks for bringing up this potential issue!