Docker container is failing to load Julia AdaGram process with large data

[RamIdavalapaati]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

I have a Julia AdaGram process which loads 8 GB of data. It is working fine on my local machine.

So, I am expecting same behavior in the docker container.

Actual behavior

Julia AdaGram process is giving bus error for more than 20MB data file in the docker container.

Error

signal (7): Bus error
while loading no file, in expression starting on line 0
macro expansion at ./cartesian.jl:62 [inlined]
macro expansion at ./multidimensional.jl:429 [inlined]
_unsafe_batchsetindex! at ./multidimensional.jl:421
_setindex! at ./multidimensional.jl:370 [inlined]
setindex! at ./abstractarray.jl:832 [inlined]
#9 at /root/.julia/v0.5/AdaGram/src/AdaGram.jl:64
#600 at ./multi.jl:1030
run_work_thunk at ./multi.jl:1001
run_work_thunk at ./multi.jl:1010 [inlined]
#597 at ./event.jl:68
unknown function (ip: 0x7fe1822db16f)
jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:189 [inlined]
jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1942
jl_apply at /home/centos/buildbot/slave/package_tarball64/build/src/julia.h:1392 [inlined]
start_task at /home/centos/buildbot/slave/package_tarball64/build/src/task.c:253
unknown function (ip: 0xffffffffffffffff)
Allocations: 9661042 (Pool: 9659980; Big: 1062); GC: 16
Bus error (core dumped)

Steps to reproduce the behavior

Dockerfile

FROM ubuntu:16.04

WORKDIR /julia

RUN apt-get -y update

RUN apt-get -y install unzip

RUN apt-get -y install cmake

RUN apt-get -y install clang

RUN apt-get -y install wget

RUN cd /tmp/

RUN wget "https://julialang.s3.amazonaws.com/bin/linux/x64/0.5/julia-0.5.0-linux-x86_64.tar.gz"

RUN tar -xzvf julia-0.5.0-linux-x86_64.tar.gz

RUN mv julia-3c9d75391c/ ~/julia

ENV PATH="/root/julia/bin:${PATH}"

RUN julia --eval 'Pkg.add("JSON")'

RUN julia --eval 'Pkg.add("HttpServer")'

RUN julia --eval 'Pkg.add("URIParser")'

RUN julia --eval 'Pkg.clone("https://github.com/deep-compute/AdaGram.jl.git")'

RUN julia --eval 'Pkg.build("AdaGram")'

CMD ["julia", "/tmp/adagram_server.jl", "80", "/julia/full.embed"]

docker-compose.yml

version: "3.1"
services:
    julia:
        image: ramidavalapati/julia:v-1
        volumes:
            - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed
        ports:
            - 8080:80
        command: ["sleep", "1h"]

Next I am doing

sudo docker-compose up
sudo docker exec -it $(sudo docker-compose ps -q julia) bash
root@3748d5958f77:/julia# julia
julia> using AdaGram
julia> AdaGram.load_model("/julia/full.embed")

link to 40MBfull.embed file
link to 20MBfull.embed file

Output of docker version:

    Client:
     Version:      17.09.0-ce
     API version:  1.32
     Go version:   go1.8.3
     Git commit:   afdb6d4
     Built:        Tue Sep 26 22:42:18 2017
     OS/Arch:      linux/amd64
    
    Server:
     Version:      17.09.0-ce
     API version:  1.32 (minimum version 1.12)
     Go version:   go1.8.3
     Git commit:   afdb6d4
     Built:        Tue Sep 26 22:40:56 2017
     OS/Arch:      linux/amd64
     Experimental: false

Output of docker info:

    Containers: 24
     Running: 0
     Paused: 0
     Stopped: 24
    Images: 24
    Server Version: 17.09.0-ce
    Storage Driver: overlay2
     Backing Filesystem: extfs
     Supports d_type: true
     Native Overlay Diff: true
    Logging Driver: json-file
    Cgroup Driver: cgroupfs
    Plugins:
     Volume: local
     Network: bridge host macvlan null overlay
     Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
    Swarm: active
     NodeID: jlkmigmtyjhz6yvi3zuvkobu7
     Is Manager: true
     ClusterID: rqt03ulgvvnym235m1qm8vd17
     Managers: 4
     Nodes: 15
     Orchestration:
      Task History Retention Limit: 5
     Raft:
      Snapshot Interval: 10000
      Number of Old Snapshots to Retain: 0
      Heartbeat Tick: 1
      Election Tick: 3
     Dispatcher:
      Heartbeat Period: 5 seconds
     CA Configuration:
      Expiry Duration: 3 months
      Force Rotate: 0
     Autolock Managers: false
     Root Rotation In Progress: false
     Node Address: X.X.X.X
      Manager Addresses:
       X.X.X.X:2377
       X.X.X.X:2377
       X.X.X.X:2377
       X.X.X.X:2377
     Runtimes: runc
     Default Runtime: runc
     Init Binary: docker-init
     containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
     runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
     init version: 949e6fa
     Security Options:
      apparmor
      seccomp
       Profile: default
     Kernel Version: 4.10.0-35-generic
     Operating System: Ubuntu 16.04.3 LTS
     OSType: linux
     Architecture: x86_64
     CPUs: 12
     Total Memory: 251.8GiB
     Name: ram
     ID: 3OGG:275C:Q3IW:O4HX:DPLP:DPI3:5TIT:AG5J:EDMK:7NK3:L4UZ:BTQH
     Docker Root Dir: /var/lib/docker
     Debug Mode (client): false
     Debug Mode (server): false
     Username: ramidavalapati
     Registry: https://index.docker.io/v1/
     Experimental: false
     Insecure Registries:
      127.0.0.0/8
     Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)
Everything is working fine in my local machine. But in docker container it is not working for more than 20MB data file.

[cpuguy83]

Have you investigated this with julia yet?
All docker is doing is starting a process, it's not likely to be causing an issue here.
Are you sure you have the same version of julia locally as you do in the container?

[RamIdavalapaati]

We used the same version (0.5) of Julia in the docker container and on the local machine.

[RamIdavalapaati]

Julia is using the shared memory of the container. By default, it is 64MB. If we can increase it then the problem will be solved.