Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARM64 Support #214

Open
scosaje opened this issue Oct 10, 2016 · 41 comments

Comments

@scosaje
Copy link

commented Oct 10, 2016

Is there a version of the nvidia-docker for arm platform. specifically for the NVidia Jetson TK1 and TX1

@3XX0

This comment has been minimized.

Copy link
Member

commented Oct 12, 2016

Not right now but we will probably look into it for nvidia-docker 2.0

@3XX0 3XX0 added the new feature label Oct 12, 2016

@Gotrek77

This comment has been minimized.

Copy link

commented Oct 12, 2016

I made the same request to some Nvidia Engineers during last GTC2016 in Amsterdam. I hope that this feature will come as soon as possible. It's very interesting to use container into TX1 som.

@shiquanwang

This comment has been minimized.

Copy link

commented Oct 12, 2016

Have you succeeded running docker on TK1/TX1?

scosaje notifications@github.com于2016年10月10日 周一 22:16写道:

Is there a version of the nvidia-docker for arm platform. specifically for
the NVidia Jetson TK1 and TX1


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#214, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAsuvqY07VtRF0NwoiuS72T2xHSqpWJhks5qyki0gaJpZM4KSnMt
.

@scosaje

This comment has been minimized.

Copy link
Author

commented Oct 12, 2016

thanks guys. would it be possible to use linux "cgroups on mesos" on arm as
the alternate container in place of the nvidia-docker container while we
await the implementation of the nvidia-docker for arm?
Where cgroups will not suffice, I am willing to committing some funds for a
focused job on the nvidia-docker for arm (TK1/TX1), which can then be
donated to the community.

If anyone can, and is interested pls send mail direct to
sunny.osaje@gmail.com.

Thanks.

On Wed, Oct 12, 2016 at 11:54 AM, Gotrek77 notifications@github.com wrote:

I made the same request to some Nvidia Engineers during last GTC2016 in
Amsterdam. I hope that this feature will come as soon as possible. It's
very interesting to use container into TX1 som.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#214 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASolKwDas1Z8in_m1ACz2ZZCo6o6tsIHks5qzLxtgaJpZM4KSnMt
.

@emanuelstanciu

This comment has been minimized.

Copy link

commented Oct 16, 2016

@3XX0 Hi Jonathan, when do you think we will see ARM support?
Also, would this be able then to run the tensorflow docker images? Or do we need to rebuild?
Thanks

@3XX0

This comment has been minimized.

Copy link
Member

commented Oct 17, 2016

I can't really say, we have other priorities right now and the ARM support is somewhat tricky.
I started looking into it but our driver is dramatically different on ARM and we would need extensive changes in nvidia-docker. We also need to tweak the L4T kernel to make it container friendly.
I will update this issue if we make any progress.

@Gotrek77

This comment has been minimized.

Copy link

commented Nov 4, 2016

Hello 3XX0, there is some news on nvidia-docker support to jetson tx1?

Thanks

@vmayoral

This comment has been minimized.

Copy link

commented Nov 23, 2016

Also interested in this topic.

@3XX0, can you describe in more detail what would be the changes needed for getting support for ARM 64 bits?. (targeting TX1) It seems this is a rather popular topic. My group has some bandwidth and wouldn't mind looking into this.

Thanks,

@3XX0

This comment has been minimized.

Copy link
Member

commented Nov 23, 2016

It's pretty complicated and it's going to take a lot of time.

First, we won't support ARM until the new 2.0 runtime becomes stable and works flawlessly on x86 (hopefully we will publish it soon). Secondly, we need to work with the L4T kernel team to support containers there. And lastly, we need to reimplement all the driver logic to work with the Tegra iGPU/dGPU stack bearing in mind that it needs to support all of our architectures (Drive PXs will probably be first) and that our Tegra drivers are drastically changing.

@Gotrek77

This comment has been minimized.

Copy link

commented Nov 30, 2016

HI @3XX0, how can do to rise up priority to this feature? It's very important to us to have nvidia-docker on tx1.

@gofrane

This comment has been minimized.

Copy link

commented Dec 14, 2016

Hello guys, are there any news about the nvidia-docker version that can support Jeston TK1?.
I am so interested in it.thank you.

@Gotrek77

This comment has been minimized.

Copy link

commented Jan 20, 2017

Hello guys,
now that the 1.0.0 was released, when you think will start the work on 2.0.0 (with jetson tx1 support)? When will be possible to try (in alfa o beta) it?
Thanks in advance
Giuseppe

@Gotrek77

This comment has been minimized.

Copy link

commented Mar 15, 2017

Hello guys,
any update?
Thanks
Giuseppe

@GONZALORUIZ

This comment has been minimized.

Copy link

commented Apr 27, 2017

I just got my TX2 and I am also interested in running everything as docker containers, so an update on this will be very helpful

@flx42

This comment has been minimized.

Copy link
Member

commented May 3, 2017

@Gotrek77 @GONZALORUIZ sorry but you're on your own for now, it's not on our near-term roadmap.

@flesicek

This comment has been minimized.

Copy link

commented May 28, 2017

We are developing vision application running on top of docker environment and our customers are looking for local solution where TX2 will fit very nicely. Could you please please increase priority of this request?

@Gotrek77

This comment has been minimized.

Copy link

commented May 30, 2017

@flx42 we Need this for our product. There are different people that ask to increase priority or ask how can push your management in this direction. It very important to know a roadmap to have this feature.
Thanks

@flx42

This comment has been minimized.

Copy link
Member

commented May 30, 2017

@flesicek @Gotrek77 I understand that this is causing an issue for you. However, there is nothing I can do right now, it's in the backlog but not on the roadmap for now.
You are on your own if you want to try to make it work.

@SPPhotonic

This comment has been minimized.

Copy link

commented Jun 10, 2017

Bump, because Docker would be great. Also looking for cloud GTU platform access API.

@kgdad

This comment has been minimized.

Copy link

commented Jun 16, 2017

As an alternative to nvidia-docker until official support is available, we were able to get Docker running on the TX-2. Need to do some kernel modifications and pass in some parameters to Docker containers so they have access to the GPU, but it is working for those that want to try it.

You can check out this GitHub repo for more information, Tegra-Docker

@epifanio

This comment has been minimized.

Copy link

commented Jun 16, 2017

@kgdad Thanks for that repo, I own a TX1 and I'll work on adapting your approach for my device.
@flx42 @3XX0 once we have a docker version running on our L4T, do you have some pointers on how to adapt nvidia-docker to run on our devices? - my aim is to run minpy on TX1 from docker.

@kaisark

This comment has been minimized.

Copy link

commented Oct 6, 2017

I installed docker on my TX1 today. I ran into a make issue with CONFIG_CGROUP_HUGETLB while building the custom kernel, so I omitted the optional CONFIG_CGROUP_HUGETLB change and the Image was built successfully. Docker images (e.g. FROM arm64v8/ubuntu:16.04) can now be used on the TX1.

The Tegra-Docker solution works (I have verified), but still feels a little like a work-around. Also, I believe the docker version (1.12.6 currently) they recommend is a little dated. It might be better to build from source or use a debian package. BTW, the Tegra-Docker solution sources/references the Jetsonhacks blog (link below) for custom kernel build instructions...

Tegra-Docker:
https://github.com/Technica-Corporation/Tegra-Docker

building custom kernels: TX1/TX2
http://www.jetsonhacks.com/2017/08/07/build-kernel-ttyacm-module-nvidia-jetson-tx1/
https://github.com/jetsonhacks/buildJetsonTX1Kernel

docker-ce arm64: stable
https://download.docker.com/linux/ubuntu/dists/xenial/pool/stable/arm64/

docker-ce install instructions: ubuntu
https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/

To check kernel compatibility with docker, run the check-compatibility.sh script from docker site:
https://docs.docker.com/engine/installation/linux/linux-postinstall/#chkconfig

-K


nvidia@tegra-ubuntu:~/bin$ bash check-config.sh
info: reading kernel config from /proc/config.gz ...

Generally Necessary:

  • cgroup hierarchy: properly mounted [/sys/fs/cgroup]
  • CONFIG_NAMESPACES: enabled
  • CONFIG_NET_NS: enabled
  • CONFIG_PID_NS: enabled
  • CONFIG_IPC_NS: enabled
  • CONFIG_UTS_NS: enabled
  • CONFIG_CGROUPS: enabled
  • CONFIG_CGROUP_CPUACCT: enabled
  • CONFIG_CGROUP_DEVICE: enabled
  • CONFIG_CGROUP_FREEZER: enabled
  • CONFIG_CGROUP_SCHED: enabled
  • CONFIG_CPUSETS: enabled
  • CONFIG_MEMCG: enabled
  • CONFIG_KEYS: enabled
  • CONFIG_VETH: enabled
  • CONFIG_BRIDGE: enabled
  • CONFIG_BRIDGE_NETFILTER: enabled (as module)
  • CONFIG_NF_NAT_IPV4: enabled
  • CONFIG_IP_NF_FILTER: enabled
  • CONFIG_IP_NF_TARGET_MASQUERADE: enabled
  • CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
  • CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
  • CONFIG_NETFILTER_XT_MATCH_IPVS: enabled
  • CONFIG_IP_NF_NAT: enabled
  • CONFIG_NF_NAT: enabled
  • CONFIG_NF_NAT_NEEDED: enabled
  • CONFIG_POSIX_MQUEUE: enabled
  • CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled

Optional Features:

  • CONFIG_USER_NS: enabled
  • CONFIG_SECCOMP: enabled
  • CONFIG_CGROUP_PIDS: enabled
  • CONFIG_MEMCG_SWAP: enabled
  • CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
  • CONFIG_MEMCG_KMEM: enabled
  • CONFIG_BLK_CGROUP: enabled
  • CONFIG_BLK_DEV_THROTTLING: enabled
  • CONFIG_IOSCHED_CFQ: enabled
  • CONFIG_CFQ_GROUP_IOSCHED: enabled
  • CONFIG_CGROUP_PERF: enabled
  • CONFIG_CGROUP_HUGETLB: missing
  • CONFIG_NET_CLS_CGROUP: enabled
  • CONFIG_CGROUP_NET_PRIO: enabled
  • CONFIG_CFS_BANDWIDTH: enabled
  • CONFIG_FAIR_GROUP_SCHED: enabled
  • CONFIG_RT_GROUP_SCHED: enabled
  • CONFIG_IP_VS: enabled
  • CONFIG_IP_VS_NFCT: enabled
  • CONFIG_IP_VS_RR: enabled
  • CONFIG_EXT4_FS: enabled
  • CONFIG_EXT4_FS_POSIX_ACL: enabled
  • CONFIG_EXT4_FS_SECURITY: enabled
  • Network Drivers:
  • "overlay":
  • CONFIG_VXLAN: enabled
    Optional (for encrypted networks):
  • CONFIG_CRYPTO: enabled
  • CONFIG_CRYPTO_AEAD: enabled
  • CONFIG_CRYPTO_GCM: enabled
  • CONFIG_CRYPTO_SEQIV: enabled
  • CONFIG_CRYPTO_GHASH: enabled
  • CONFIG_XFRM: enabled
  • CONFIG_XFRM_USER: enabled
  • CONFIG_XFRM_ALGO: enabled
  • CONFIG_INET_ESP: enabled
  • CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  • "ipvlan":
  • CONFIG_IPVLAN: enabled
  • "macvlan":
  • CONFIG_MACVLAN: enabled
  • CONFIG_DUMMY: enabled
  • "ftp,tftp client in container":
  • CONFIG_NF_NAT_FTP: enabled
  • CONFIG_NF_CONNTRACK_FTP: enabled
  • CONFIG_NF_NAT_TFTP: enabled
  • CONFIG_NF_CONNTRACK_TFTP: enabled
  • Storage Drivers:
  • "aufs":
  • CONFIG_AUFS_FS: missing
  • "btrfs":
  • CONFIG_BTRFS_FS: enabled
  • CONFIG_BTRFS_FS_POSIX_ACL: enabled
  • "devicemapper":
  • CONFIG_BLK_DEV_DM: enabled
  • CONFIG_DM_THIN_PROVISIONING: enabled
  • "overlay":
  • CONFIG_OVERLAY_FS: enabled
  • "zfs":
  • /dev/zfs: missing
  • zfs command: missing
  • zpool command: missing

Limits:

  • /proc/sys/kernel/keys/root_maxkeys: 1000000

nvidia@tegra-ubuntu:~/bin$ docker info
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 6
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-179:33-2899445-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 450.8 MB
Data Space Total: 107.4 GB
Data Space Available: 27.04 GB
Metadata Space Used: 1.192 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use --storage-opt dm.thinpooldev to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.110 (2015-10-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.38-jetsonbot-doc-v0.3
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 3.888 GiB
Name: tegra-ubuntu
ID: GGQY:ZUKT:AEXF:NPL2:JDFZ:UHGJ:7XVI:PXVI:FQK2:F6MZ:PP4A:PEVD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8


nvidia@tegra-ubuntu:~/docker/dq$ docker run --device=/dev/nvhost-ctrl --device=/dev/nvhost-ctrl-gpu --device=/dev/nvhost-prof-gpu --device=/dev/nvmap --device=/dev/nvhost-gpu --device=/dev/nvhost-as-gpu -v /usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu/tegra device_query
/cudaSamples/deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA Tegra X1"
CUDA Driver Version / Runtime Version 8.0 / 8.0
CUDA Capability Major/Minor version number: 5.3
Total amount of global memory: 3981 MBytes (4174815232 bytes)
( 2) Multiprocessors, (128) CUDA Cores/MP: 256 CUDA Cores
GPU Max Clock rate: 998 MHz (1.00 GHz)
Memory Clock rate: 13 Mhz
Memory Bus Width: 64-bit
L2 Cache Size: 262144 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 32768
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: Yes
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = NVIDIA Tegra X1
Result = PASS

check-config.sh.txt

@Gotrek77

This comment has been minimized.

Copy link

commented Nov 8, 2017

Hi all,
I want to try nvidia-docker 2 in tx1. Is It possibile ti build libnvidia-docker for arm64 or there is some issue at the moment?

Thanks
Giuseppe

@3XX0 3XX0 changed the title nvidia-docker for arm platform ARM64 Support Nov 14, 2017

@stanleynguyen

This comment has been minimized.

Copy link

commented Feb 12, 2018

Any updates?

@alejandroandreu

This comment has been minimized.

Copy link

commented Mar 19, 2018

+1 for this feature.

@waldemar-szostak

This comment has been minimized.

Copy link

commented May 6, 2018

What's the point of having the docker-friendly kernel in the jetpack 3.2 if there's no docker for the jetson tx2 running the jetpack? :(

@stanleynguyen

This comment has been minimized.

Copy link

commented May 8, 2018

^

@kgdad

This comment has been minimized.

Copy link

commented May 8, 2018

Regular docker will work fine as long as you don't need GPU access. If you do need access to the GPU in your containers you need to make sure you give your containers access to some specific libraries and devices. Can read the details here, GitHub-Tegra_Docker

@alejandroandreu

This comment has been minimized.

Copy link

commented May 16, 2018

Is this a feature on the current roadmap for nvidia-docker?

Although the above comment points to a valid workaround, it doesn't have all the nice features that nvidia-docker provides... 😞

@laudai

This comment has been minimized.

Copy link

commented May 17, 2018

@flx42

This comment has been minimized.

Copy link
Member

commented May 17, 2018

Sorry, it's still not on the roadmap, given than the driver stacks are very different today.

@sundl123

This comment has been minimized.

Copy link

commented Aug 10, 2018

We use Tegra-Docker on TX2 as a workaround, but we really hope that nvidia-docker can add support for tx2 platform offically.

@haudren

This comment has been minimized.

Copy link

commented Sep 20, 2018

Is there any chance that this feature will appear on the roadmap given the recent release of the Jetson Xavier?

@sanjaybharadwaj

This comment has been minimized.

Copy link

commented Oct 12, 2018

Hi, Is such a feature planned now for the newer devices? Any roadmaps ?

@TheNishantGadhvi

This comment has been minimized.

Copy link

commented Oct 30, 2018

I am waiting too..!

@leontius

This comment has been minimized.

Copy link

commented Nov 9, 2018

I am waiting too

@FluxReality

This comment has been minimized.

Copy link

commented Nov 12, 2018

... also waiting!

@waseemkhan1989

This comment has been minimized.

Copy link

commented Dec 10, 2018

Hi,

I have created a docker image on Jetson TX2 which contains Nvidia drivers, CUDA and Cudnn libraries. I am trying to give access of GPU and CUDA drivers to this image through tx2-docker script (https://github.com/Technica-Corporation/Tegra-Docker) but no success. I think tx2-docker is running successfully which you can see below:

wkh@tegra-ubuntu:~/Tegra-Docker/bin$ ./tx2-docker run openhorizon/aarch64-tx2-cudabase
Running an nvidia docker image
docker run -e LD_LIBRARY_PATH=:/usr/lib/aarch64-linux-gnu:/usr/lib/aarch64-linux-gnu/tegra:/usr/local/cuda/lib64 --net=host -v /usr/lib/aarch64-linux-gnu:/usr/lib/aarch64-linux-gnu -v /usr/local/cuda/lib64:/usr/local/cuda/lib64 --device=/dev/nvhost-ctrl --device=/dev/nvhost-ctrl-gpu --device=/dev/nvhost-prof-gpu --device=/dev/nvmap --device=/dev/nvhost-gpu --device=/dev/nvhost-as-gpu openhorizon/aarch64-tx2-cudabase

But when I try to run devicequery inside my container, it give me the result:

root@bc1130fc6be4:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery# ./deviceQuery
./deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 38
-> no CUDA-capable device is detected
Result = FAIL

Any comment! Why this script is not giving access?

@Seanmatthews

This comment has been minimized.

Copy link

commented Jan 4, 2019

How's this feature coming along?

@tdurand tdurand referenced this issue Jan 24, 2019
8 of 8 tasks complete
@vielmetti

This comment has been minimized.

Copy link

commented May 5, 2019

Looking for this for the Jetson Nano.

@OlivierGuilloux

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.