docker reports neoverse-n1 as neoverse-v1, leading to failures for Julia

[grahamgill]

It's not clear to me where I should report this so will start off here.

In my Dockerfile I have

FROM julia:1.10
...
# (1) check some system info
RUN echo uname info: kernel name, kernel release, machine hardware name, processor type, hardware platform, OS... && \
    uname -srmpio && \
    echo lscpu info... && \
    lscpu && \
    echo Julia system info... && \
    julia -e 'using .Sys, InteractiveUtils; foreach((:MACHINE, :ARCH, :KERNEL, :CPU_NAME, :CPU_THREADS)) do p println(p, ": ", eval(p)) end; versioninfo()'

# (2) this breaks the build
RUN julia -e "using InteractiveUtils; versioninfo(verbose = true)"
...

Note that Julia versioninfo() in (1) defaults to versioninfo(verbose=false).

The docker image is built on a Graviton 2 (neoverse-n1) instance on Amazon EC2. The output from (1) is the following (with AWS Codebuild line prefixes stripped out). Note the processor is listed by lscpu and detected by Julia Sys.CPU_NAME and versioninfo() as neoverse-v1.

uname info: kernel name, kernel release, machine hardware name, processor type, hardware platform, OS...
Linux 4.14.343-261.564.amzn2.aarch64 aarch64 unknown unknown GNU/Linux
lscpu info...
Architecture:                       aarch64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
CPU(s):                             8
On-line CPU(s) list:                0-7
Vendor ID:                          ARM
Model name:                         Neoverse-V1
Model:                              1
Thread(s) per core:                 1
Core(s) per cluster:                8
...
Flags:                              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs rng
...
Julia system info...
MACHINE: aarch64-linux-gnu
ARCH: aarch64
KERNEL: Linux
CPU_NAME: neoverse-v1
CPU_THREADS: 8
Julia Version 1.10.4
Commit 48d4fd48430 (2024-06-04 10:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (aarch64-linux-gnu)
  CPU: 8 × unknown
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, neoverse-v1)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
...

The problem comes when (2) runs. versioninfo(verbose=true) includes getting a CPU load average.

[11/11] RUN julia -e "using InteractiveUtils; versioninfo(verbose = true)"
Julia Version 1.10.4
Commit 48d4fd48430 (2024-06-04 10:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (aarch64-linux-gnu)
  uname: Linux 4.14.343-261.564.amzn2.aarch64 #1 SMP Tue May 7 02:23:29 UTC 2024 aarch64 unknown
  CPU: unknown: 
              speed         user         nice          sys         idle          irq
       #1     0 MHz        451 s          1 s         23 s       4097 s          0 s
       #2     0 MHz        365 s          3 s         25 s       4385 s          0 s
       #3     0 MHz        315 s          1 s         24 s       4458 s          0 s
       #4     0 MHz        276 s          1 s         19 s       4496 s          0 s
       #5     0 MHz        838 s          3 s         56 s       3660 s          0 s
       #6     0 MHz        225 s          2 s         24 s       4498 s          0 s
       #7     0 MHz        370 s          2 s         28 s       4388 s          0 s
       #8     0 MHz        333 s          2 s         22 s       4437 s          0 s
  Memory: 15.439155578613281 GB (14914.6328125 MB free)
  Uptime: 481.96 sec
  Load Avg: Invalid instruction at 0xffff91757148: 0x04a0e3ea

[7] signal (4.1): Illegal instruction
in expression starting at none:1
iterate at ./range.jl:901 [inlined]
vcat at ./range.jl:1375 [inlined]
_print_matrix at ./arrayshow.jl:205
print_matrix at ./arrayshow.jl:171
print_matrix at ./arrayshow.jl:171 [inlined]
#versioninfo#65 at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/InteractiveUtils/src/InteractiveUtils.jl:158
versioninfo at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/InteractiveUtils/src/InteractiveUtils.jl:97
unknown function (ip: 0xffff916ec04f)
_jl_invoke at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/gf.c:3077
versioninfo at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/InteractiveUtils/src/InteractiveUtils.jl:97
unknown function (ip: 0xffff921470eb)
_jl_invoke at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/interpreter.c:635
jl_interpret_toplevel_thunk at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
exec_options at ./client.jl:291
_start at ./client.jl:552
jfptr__start_82654 at /usr/local/julia/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-armageddon-0/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
unknown function (ip: 0xffff9217777f)
__libc_start_main at /lib/aarch64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x400a1b)
unknown function (ip: 0x400a1b)
Allocations: 627086 (Pool: 626483; Big: 603); GC: 1
Illegal instruction (core dumped)
ERROR: process "/bin/sh -c julia -e \"using InteractiveUtils; versioninfo(verbose = true)\"" did not complete successfully: exit code: 132

If instead of (2) I use

RUN julia --cpu-target=neoverse-n1 -e "using InteractiveUtils; versioninfo(verbose = true)"

then this command completes successfully, and reports

Julia Version 1.10.4
Commit 48d4fd48430 (2024-06-04 10:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (aarch64-linux-gnu)
  uname: Linux 4.14.343-261.564.amzn2.aarch64 #1 SMP Tue May 7 02:23:29 UTC 2024 aarch64 unknown
  CPU: unknown: 
              speed         user         nice          sys         idle          irq
       #1     0 MHz        512 s          4 s         44 s      13238 s          0 s
       #2     0 MHz        854 s          4 s         38 s      13191 s          0 s
       #3     0 MHz        268 s          4 s         30 s      13814 s          0 s
       #4     0 MHz        530 s          4 s         37 s      13532 s          0 s
       #5     0 MHz        245 s          6 s         40 s      13583 s          0 s
       #6     0 MHz        385 s          6 s         37 s      13652 s          0 s
       #7     0 MHz        214 s          5 s         25 s      13855 s          0 s
       #8     0 MHz        188 s          4 s         27 s      13895 s          0 s
  Memory: 15.439155578613281 GB (14917.72265625 MB free)
  Uptime: 1413.98 sec
  Load Avg:  1.95  0.87  0.35
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, neoverse-v1)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
...

So it seems like Julia can't detect the correct processor and Docker is making it harder to do so.

Note that before this I also experienced Julia precompilation failures during docker build in the same build environment, and now set

ENV JULIA_CPU_TARGET="generic;cortex-a57;thunderx2t99;carmel,clone_all;apple-m1,base(3);neoverse-n1,clone_all;neoverse-512tvb,clone_all;neoverse-v1,base(6);neoverse-v2,base(6)"

which allows precompilation to succeed. See this post on the Julia Discourse board for more details.

[yosifkit]

Perhaps the host is presenting the wrong CPU values? (or maybe an outdated binary/library misinterpreting the value?) Did you run lscpu on the 6g and 7g hosts within and without a container? As far as I understand, Docker doesn't change the CPU info for a container; it's not a VM and there aren't namespaces or cgroups for something like that.

It might be possible with eBPF to return different values inside the container, but definitely not something that would be in a standard Docker install.

[grahamgill]

Sorry for the delay in replying. I have to wait for one of our devops guys to be available to run anything directly on the host instance.

Thanks for the reply. Yes you're right it looks like docker is just passing the CPU info through. The confusion came because we were building with AWS CodeBuild (which itself uses docker apparently). We don't control the instance where CodeBuild runs, so we don't get to choose the processor, but then we run on EC2 instances where we do control the choice of instance and hence processor.

Then the issue looks like it could be a problem in CodeBuild or the host it's running on - as you said something outdated misinterpreting values - but I'm not sure if we're able to determine that. Maybe with an AWS support ticket. The alternative would be a problem in Julia/LLVM, issuing an incorrect instruction for neoverse-v1, which seems unlikely.

I'll close this issue.

As far as I understand, Docker doesn't change the CPU info for a container

It may have some impact on determining the CPU. For example on my M1 Pro Mac Julia reports Sys.CPU_NAME == "generic" when run from a container in the official Julia docker image, whereas run directly on my machine it correctly reports the CPU as "apple-m1". So from within the container Julia has trouble determining the correct processor and falls back to a generic (ARM) processor.