IPEX Multinode SSH Support #124

tylertitsworth · 2024-06-12T04:40:07Z

Description

Mount your custom SSH config and SSH keys to use manually, but otherwise it should generate a key and start a server for usage in k8s.

Related Issue

n/a

Changes Made

The code follows the project's coding standards.
No Intel Internal IP is present within the changes.
The documentation has been updated to reflect any changes in functionality.

Validation

I have tested any changes in container groups locally with test_runner.py with all existing tests passing, and I have added new tests where applicable.

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

github-actions · 2024-06-12T04:40:29Z

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details

Scanned Manifest Files

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

tylertitsworth · 2024-06-12T05:38:19Z

pytorch/Dockerfile


 RUN python -m pip install --no-cache-dir -r multinode-requirements.txt

+ENV LD_LIBRARY_PATH="/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH}:/usr/local/lib/python${PYTHON_VERSION}/dist-packages/oneccl_bindings_for_pytorch/opt/mpi/libfabric/lib:/usr/local/lib/python${PYTHON_VERSION}/dist-packages/oneccl_bindings_for_pytorch/lib"


prepend /lib/x86_64-linux-gnu overrides conda's installation of openssl which conflicts with dpkg

pytorch/Dockerfile

tylertitsworth · 2024-06-12T05:41:33Z

@dmsuehir @louie-tsai
Please test your BKMs with the multinode image.

pytorch/generate_ssh_keys.sh

dmsuehir · 2024-06-12T18:36:47Z

Thanks, I'm testing this multinode base with a rebuilt version of our workflow container that has the SSH config removed from our dockerfile. I'll report back with the results when testing is done.

dmsuehir · 2024-06-12T22:29:32Z

@tylertitsworth Based on my testing, this looks good. I ran on a single node and on multiple nodes with the updated container.

dmsuehir · 2024-06-13T17:32:15Z

@HarshaRamayanam FYI

sharvil10

LGTM

github-advanced-security · 2024-06-26T19:07:05Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

dmsuehir

LGTM. I tested this update with our k8s HF workflow.

…ion.

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com> Co-authored-by: sharvil.shah <sharvils@mlp-prod-clx-5669.ra.intel.com> Co-authored-by: sharvil10 <sharvil.shah@intel.com> Co-authored-by: Jitendra Patil <jitendra.patil@intel.com> Signed-off-by: ma-pineda <miguel.pineda.juarez@intel.com>

pytorch/Dockerfile

louie-tsai · 2024-06-12T23:48:47Z

pytorch/Dockerfile


+COPY generate_ssh_keys.sh .
+
+RUN cat /generate_ssh_keys.sh >> ~/.bashrc && \


how about client public/private key? generate_ssh_key.sh only cover server key

louie-tsai · 2024-06-28T20:21:39Z

pytorch/Dockerfile

+# hadolint global ignore=SC2002
+RUN mkdir -p /var/run/sshd && \
+    cat /etc/ssh/ssh_config | grep -v StrictHostKeyChecking > /etc/ssh/ssh_config.new && \
+    echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config.new && \


might also need to config ssh_config as below to have another ssh port support for sshd in docker with different port
Host * Port 2345

louie-tsai · 2024-06-28T20:24:18Z

pytorch/Dockerfile

+    cat /etc/ssh/ssh_config | grep -v StrictHostKeyChecking > /etc/ssh/ssh_config.new && \
+    echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config.new && \
+    mv /etc/ssh/ssh_config.new /etc/ssh/ssh_config
+


might also need to set sshd_config to use different sshd port among all instances.
"Port 2345 >> /etc/sshd/sshd_config"

louie-tsai · 2024-06-28T20:26:19Z

pytorch/Dockerfile

+    cat /etc/ssh/ssh_config | grep -v StrictHostKeyChecking > /etc/ssh/ssh_config.new && \
+    echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config.new && \
+    mv /etc/ssh/ssh_config.new /etc/ssh/ssh_config
+


I also set below sshd config to have have Root login without password.
not sure whether we also need it or not
RUN sed -i'' -e's/^#PermitRootLogin prohibit-password$/PermitRootLogin yes/' /etc/ssh/sshd_config \ && sed -i'' -e's/^#PasswordAuthentication yes$/PasswordAuthentication no/' /etc/ssh/sshd_config \ && sed -i'' -e's/^#PermitEmptyPasswords no$/PermitEmptyPasswords yes/' /etc/ssh/sshd_config \ && sed -i'' -e's/^UsePAM yes/UsePAM no/' /etc/ssh/sshd_conf

louie-tsai · 2024-06-28T20:27:30Z

pytorch/README.md

+> User also need to append content of id_rsa.pub in `/etc/ssh/authorized_keys` in the SSH server container.
+> Since the SSH key is not owned by default user account in docker, please also do "chmod 644 id_rsa.pub; chmod 644 id_rsa" to grant read access for default user account.
+> Users could also use "/usr/bin/ssh-keygen -t rsa -b 4096 -N '' -f ~/mnt/ssh_key/id_rsa" to generate a new SSH Key inside the container.
+> Users need to mount a config file to list all hostnames at location `/root/.ssh/config` on the SSH client container.


it could be handled by docker image if we config /etc/ssh/sshd_config and ssh_config well.

louie-tsai · 2024-06-28T20:29:29Z

pytorch/README.md

+    ```
+
+2. Add hosts to config
+


this step could be further simplified if you config /etc/ssh well during docker build. we don't need to put those efforts on users, and the manual configuration is also error-prone.

Tyler Titsworth added 2 commits June 11, 2024 21:39

Update Dockerfile

d6b01d2

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

Create generate_ssh_keys.sh

3726765

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

tylertitsworth added the WIP Work in Progress label Jun 12, 2024

tylertitsworth self-assigned this Jun 12, 2024

tylertitsworth requested review from jitendra42 and sharvil10 as code owners June 12, 2024 04:40

Tyler Titsworth added 8 commits June 11, 2024 21:45

Update generate_ssh_keys.sh

3d774ed

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

Update Dockerfile

79fec62

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

Update Dockerfile

0c8e2d1

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

Update Dockerfile

60d2b4d

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

Update Dockerfile

9515a31

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

Update Dockerfile

6d0c6b8

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

Update Dockerfile

d704bac

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

Update README.md

43be2a9

Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com>

tylertitsworth mentioned this pull request Jun 12, 2024

Enable Passwordless ssh connection for Multinodes docker #61

Closed

4 tasks

tylertitsworth requested a review from ashahba June 12, 2024 05:37

tylertitsworth commented Jun 12, 2024

View reviewed changes

pytorch/Dockerfile Outdated Show resolved Hide resolved

tylertitsworth commented Jun 12, 2024

View reviewed changes

pytorch/Dockerfile Outdated Show resolved Hide resolved

config sshd

94b3ee7

tylertitsworth commented Jun 12, 2024

View reviewed changes

pytorch/generate_ssh_keys.sh Outdated Show resolved Hide resolved

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

9c7bd0d

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

ec8e64f

Tyler Titsworth added 2 commits June 13, 2024 11:45

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

0f9b095

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

c713b43

update README.md

00d8147

tylertitsworth force-pushed the tylertitsworth/ssh-ipex-multinode branch from 630f798 to 00d8147 Compare June 17, 2024 23:05

tylertitsworth added 2 commits June 17, 2024 16:07

add a note to cover our butts

63159df

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

6db3ecb

tylertitsworth added Review and removed WIP Work in Progress labels Jun 18, 2024

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

eb5a0a0

sharvil10 previously approved these changes Jun 18, 2024

View reviewed changes

tylertitsworth requested a review from sramakintel as a code owner June 18, 2024 17:01

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

4c1a917

jitendra42 previously approved these changes Jun 26, 2024

View reviewed changes

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

750bc09

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

7a565fa

dmsuehir previously approved these changes Jun 27, 2024

View reviewed changes

tylertitsworth assigned sharvil10 Jun 27, 2024

sharvil10 added 2 commits June 27, 2024 14:00

Added an entrypint to set startup commands like SSH Host keys generat…

65fe3f1

…ion.

Merged update

12c2f53

sharvil10 dismissed stale reviews from dmsuehir, jitendra42, and themself via 12c2f53 June 27, 2024 21:10

sharvil10 and others added 3 commits June 27, 2024 15:29

Extra spaces removed

37f8cca

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

c4a7828

Merge branch 'main' into tylertitsworth/ssh-ipex-multinode

0f1239a

tylertitsworth merged commit 6006b33 into main Jun 28, 2024

tylertitsworth deleted the tylertitsworth/ssh-ipex-multinode branch June 28, 2024 16:28

louie-tsai reviewed Jun 28, 2024

View reviewed changes


		RUN python -m pip install --no-cache-dir -r multinode-requirements.txt

		ENV LD_LIBRARY_PATH="/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH}:/usr/local/lib/python${PYTHON_VERSION}/dist-packages/oneccl_bindings_for_pytorch/opt/mpi/libfabric/lib:/usr/local/lib/python${PYTHON_VERSION}/dist-packages/oneccl_bindings_for_pytorch/lib"


		COPY generate_ssh_keys.sh .

		RUN cat /generate_ssh_keys.sh >> ~/.bashrc && \

IPEX Multinode SSH Support #124

IPEX Multinode SSH Support #124

Uh oh!

Conversation

tylertitsworth commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Changes Made

Validation

Uh oh!

github-actions bot commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency Review

OpenSSF Scorecard

Scanned Manifest Files

Uh oh!

tylertitsworth Jun 12, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tylertitsworth commented Jun 12, 2024

Uh oh!

Uh oh!

dmsuehir commented Jun 12, 2024

Uh oh!

dmsuehir commented Jun 12, 2024

Uh oh!

dmsuehir commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sharvil10 left a comment

Choose a reason for hiding this comment

Uh oh!

github-advanced-security bot commented Jun 26, 2024

Uh oh!

dmsuehir left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

louie-tsai Jun 12, 2024

Choose a reason for hiding this comment

Uh oh!

louie-tsai Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

louie-tsai Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

louie-tsai Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

louie-tsai Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

louie-tsai Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tylertitsworth commented Jun 12, 2024 •

edited

Loading

github-actions bot commented Jun 12, 2024 •

edited

Loading

dmsuehir commented Jun 13, 2024 •

edited

Loading