Tutorial for SmartSim on Slurm

[Spartee]

Description

Add a tutorial for using SmartSim on Slurm based systems.

Justification

Currently, we only have two tutorials and neither cover Slurm functionality

Implementation Strategy

The following items should be covered in a SmartSim-Slurm tutorial

• Launching jobs in a previously obtained interactive slurm allocation (using SrunSettings)
• Launching batch jobs with SbatchSettings in ensembles
• Getting and releasing allocations through the slurm interface
• Launching on allocations obtained through the smartsim.slurm interface
• Creating and launching the SlurmOrchestrator on interactive allocations and as a batch

The tutorial should be implemented as a Jupyter Notebook and use nbsphinx to host as a part of the documentation.

[neuralvis]

I had started some examples based on your pointers, @Spartee - https://github.com/neuralvis/SmartSim/tree/slurm_examples/tutorials/slurm/HelloSmartSim

They still need to be cleaned-up a bit. Additional examples would be good too. But it solves the first two items on your list, hopefully !

[Spartee]

@neuralvis these look great! A couple notes:

• It looks like your using 0.2.0. You should update to 0.3.0 so you don't need to worry about the SMARTSIMHOME path and are on a stable version. You can just follow the instructions for the quick (or full if you want GPU support) install, but cloning the repo instead of building from the release. instructions for pip install will work the same from release tarball as from source. Interface should be the same for the most part so the examples don't need to change much.
• We have been shifting to doing tutorials in Jupyter and "sphinxing" them with nbsphinx. It would be great if we could put these examples in a notebook like this tutorial we did

Let me know if I can help with anything.

Thanks!

[neuralvis]

Thanks for the notes @Spartee ! I'll port these starter examples into a notebook format. Are the examples expected to work from jupyter as well ? Or are they merely "written" in notebook style to "sphinxify" them ?

[Spartee]

All the examples should work from jupyter, if you haven't setup jupyter manually on an XC (or EX :) before, I can also help with that.

[Spartee]

closing as README now has slurm examples