Skip to content

Commit c9b2c44

Browse files
JiangJiaWei1103Future-Outlier
JiangJiaWei1103
and
Future-Outlier
authored
[Docs] Add the Slurm agent example usage (#1781)
* docs: Add the Slurm agent fn task example usage Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * docs: Add example usage for SlurmTask and SlurmShellTask Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * docs: Add install command Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * update host name Signed-off-by: Future-Outlier <eric901201@gmail.com> * refactor: Define unique workflow names Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * docs: Add three main components for DL training wf example Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * docs: Add DL model training wf Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * docs: Complete index page Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * Lint Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * Add missing python pkgs Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * Add tqdm to requirements Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * Add the Slurm agent to integrations Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> * push Signed-off-by: Future-Outlier <eric901201@gmail.com> * update Signed-off-by: Future-Outlier <eric901201@gmail.com> * fix: Use unique task name for each shell task instance Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> --------- Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Future-Outlier <eric901201@gmail.com>
1 parent ab2e8e8 commit c9b2c44

File tree

6 files changed

+680
-0
lines changed

6 files changed

+680
-0
lines changed

‎docs/integrations/index.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,8 @@ Native backend plugins can be executed without any external service dependencies
116116
- Execute tasks on PERIAN Job Platform.
117117
* - {doc}`Sensor agent </auto_examples/sensor/index>`
118118
- Run sensor jobs in your workflows with the sensor agent.
119+
* - {doc}`Slurm agent </auto_examples/slurm_agent/index>`
120+
- Run Slurm jobs in your workflows with the Slurm agent.
119121
* - {doc}`Snowflake agent </auto_examples/snowflake_agent/index>`
120122
- Run Snowflake jobs in your workflows with the Snowflake agent.
121123
```
@@ -245,6 +247,7 @@ Memory Machine Cloud agent </auto_examples/mmcloud_agent/index>
245247
OpenAI batch agent </auto_examples/openai_batch_agent/index>
246248
PERIAN Job Platform agent </auto_examples/perian_agent/index>
247249
Sensor agent </auto_examples/sensor/index>
250+
Slurm agent </auto_examples/slurm_agent/index>
248251
Snowflake agent </auto_examples/snowflake_agent/index>
249252
```
250253

‎examples/slurm_agent/Dockerfile‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# ######################
2+
# NOTE: For CI/CD only #
3+
########################
4+
FROM python:3.11-slim-buster
5+
LABEL org.opencontainers.image.source=https://github.com/flyteorg/flytesnacks
6+
7+
WORKDIR /root
8+
ENV VENV /opt/venv
9+
ENV LANG C.UTF-8
10+
ENV LC_ALL C.UTF-8
11+
ENV PYTHONPATH /root
12+
13+
# Install Python dependencies
14+
COPY requirements.in /root
15+
RUN pip install -r /root/requirements.in
16+
17+
# Copy the actual code
18+
COPY . /root/
19+
20+
# This tag is supplied by the build script and will be used to determine the version
21+
# when registering tasks, workflows, and launch plans
22+
ARG tag
23+
ENV FLYTE_INTERNAL_IMAGE $tag

‎examples/slurm_agent/README.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
(slurm_agent)=
2+
3+
# Slurm agent
4+
5+
```{eval-rst}
6+
.. tags:: Integration, HighPerformanceComputing, Advanced
7+
```
8+
9+
## Installation
10+
11+
To install the Slurm agent, run the following command:
12+
13+
```{eval-rst}
14+
.. prompt:: bash
15+
16+
pip install flytekitplugins-slurm
17+
```
18+
19+
## Example usage
20+
21+
For the example usage of different Slurm task types, please see {doc}`Slurm agent example usage<slurm_agent_example_usage>`.
22+
23+
## Local testing
24+
25+
To test the Slurm agent locally, create a class for the agent task that inherits from [AsyncAgentExecutorMixin](https://github.com/flyteorg/flytekit/blob/cd6bd01ad0ba6688afc71a33a59ece53f90e841a/flytekit/extend/backend/base_agent.py#L3). This mixin can handle asynchronous tasks and allows flytekit to mimic FlytePropeller's behavior in calling the agent. For more information, see "[Testing agents locally](https://docs.flyte.org/en/latest/flyte_agents/testing_agents_in_a_local_python_environment.html)".
26+
27+
```{note}
28+
In some cases, you will need to store credentials in your local environment when testing locally.
29+
```
30+
31+
## Flyte deployment configuration
32+
33+
```{note}
34+
If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment.
35+
```
36+
37+
To enable the Slurm agent in your Flyte deployment, see the {ref}`Slurm agent deployment guide<deployment-agent-setup-slurm>`.
38+
39+
40+
```{toctree}
41+
:maxdepth: -1
42+
:hidden:
43+
44+
slurm_agent_example_usage
45+
```

‎examples/slurm_agent/requirements.in‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
flytekitplugins-slurm
2+
torch
3+
torchvision
4+
tqdm

‎examples/slurm_agent/slurm_agent/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)