Deployments issues on NVIDIA DGX2

[abuccts]

Background

NVIDIA DGX2 uses a customized DGX OS and has GPU drivers, IB drivers, nvidia-docker, etc. installed already. To install PAI on DGX2 server and keep all pre-installed software, there'll be some issues during PAI deployment.

Issues

• Cannot choose a list of services to start ([Deployment] Cannot choose a list of services to start #2734)
• Hadoop base image does not contain cuda library ([Deployment] Hadoop base image does not contain cuda library #2735)
• Hadoop node manager cannot get GPU status correctly ([Hadoop AI] Cannot get GPU status correctly #2736)
• Job exporter cannot get GPU status ([Job Exporter] Cannot get GPU status #2737)
• Cannot listen to InfiniBand or multiple network interfaces ([Job Exporter] Cannot listen to InfiniBand or multiple network interfaces #2738)
• Cannot start job container by nvidia-docker ([Rest Server] Cannot start job container by nvidia-docker #2739)
• YARN configuration cannot change max resource allocation value dynamically ([Deployment] YARN configuration cannot change max resource allocation value dynamically #2740)
• Docker and kubelet has many error logs with zfs storage (Docker zfs driver issues)
• Support ed25519 private keys for SSH in deployment ( [Deployment] Support ECC private keys for SSH in deployment #3332)

Workaround

This branch based on pai-0.12.y release can be used as a workaround to deploy PAI on DGX2 servers.

[fanyangCS]

@abuccts could you provide some links to explain the last issue?

[abuccts]

For Docker zfs driver issue, output in docker info:

Storage Driver: zfs
 Zpool: error while getting pool information strconv.ParseUint: parsing "": invalid syntax
 Zpool Health: not available
 Parent Dataset: storage/docker
 Space Used By Parent: ******
 Space Available: ******
 Parent Quota: no
 Compression: off

Here're related issues in Docker and its upstream Go wrapper for zfs.

But this has not affected Docker/k8s yet.

[fanyangCS]

closed and track in separate issues.