Few of ZeroCostDL4Mic deeplearning notebooks which are originally adapted from the GitHub repository ZeroCostDL4Mic, are customised to work in CSC environment. Researchers working in microscopy, with little or no coding expertise, can take one of these popular deep learning models and customise it to work in CSC environement.
CSC hosts notebooks on Puhti supercomputing environment. A CSC user is expected to have a user account at CSC and be a member of a project that has access to the Puhti service. Users can use either Puhti web interface or SSH client for logging into Puhti. The familiarity with the following main topics help you get started smoothly with notebooks:
- A brief primer on using Puhti computing environment
- Preparing a custom Jupyter notebook
- Creating a custom environment/module(s) on Puhti Web Interface
- Accessing notebook via Puhti web interface
- Useful CSC documentation
- Acknowledgements and citation
- Module system: CSC uses module system to manage complex application stack in supercomputing environment. Applications installed as modules can easily be used in both interactive and batch jobs. The detailed instructions on using modularised applications can be found in CSC documentation pages as well as a CSC course page
- Disk areas: CSC supercomputers have three main disk areas namely home, projappl and scratch which are visible to all compute and login nodes. Each disk area has its own dedicated use and comes with quota limits on the size of disk pace and the number of files. Default quotas and their specific use can be found in CSC user documentation
- Custom installations: You can install own software on CSC supercomputers if you cannot find it from the list of pre-installed applications or using module spider command on Puhti terminal. Typically, one can download the source code of the software, compiles the code, and installs to a location where the user has write-access, e.g. the project's /projappl directory. More about installations can be found on CSC documentation page and a CSC course pages. Puhti also supports containerised installations
- Puhti web interface: You can use web interface for Puhti to access Puhti supercomputer. The web interface greatly eases the use of complex applications that have graphical user interfaces among other uses. Read more information about the web interface on CSC documentation page
CSC provisions popular python environements as ready-to-use notebooks. You can however customise a python environement to meet your own needs and can be accessed through Puhti web interface. The customisation of notebook typically involves the following procedure:
Conda installations should not be done directly on Puhti. Tykky wrapper tool instead be used to install python packages in setting up your compute environment. The wrapper tool installs applications inside of a singularity container and thus facilitates better performance in terms of faster startup times, reduced IO load, and reduced number of files on parallel filesystems.
The example of tykky-based custom installation for three deep learning models namely, StarDist-2D-ZeroCostDL4Mic, SplineDist-2D-ZeroCostDL4Mic and EmbedSeg-2D-ZeroCostDL4Mic can be found in the individual folders.
One has to create a course environments (modules) in the directory /projappl/project_xxxx/www_puhti_modules/ to be able to see a course module in Puhti web insterface ("under Jupyter for courses") . The www_puhti_modules directory can be created if it does not exist. Please note that module_name has to be unique accross the project members.
The two files needed for setting up the course modules are:
- a <<module_name>>.lua file that defines the module that sets up the Python environment. Only files containing the text Jupyter will be visible in the app.
- a <<module_name>>-resources.yml that defines the default resources used for Jupyter.
For three Google colab notebooks, the above mentioned two files (i.e., .lua and .yaml files) are created in the GitHub repository in their respective folders. So just copy them over to appropriate place under /projappl folder.
Once you have set up modules as instructed above, a module named, notebook is visible in "Jupyter for courses" app in the Puhti web interface
-
Login to Puhti web interface
-
Login with your CSC account (or Haka/Virtu) (Users should have accepted Puhti service in myCSC page under a course ( or own) project before using this service).
-
Once login is successfull, you can click "Interactive Sessions" on the top menu bar and then click "Jupyter for courses" (or directly from pinned apps on landing page). Once the app is launched, you can see the different fields to fill in before launching a job. For this course, select the "course Project" and Working directory corresponding to course project. Then you will be able to see custom module under the "Course module" field. You can then launch Jupyter notebook whick will be launched in the interaction partition by default. You can also change the default settings by checking "Show custom resource settings".
-
Upon successful launching a job, you can click on "connect to Jupyter" to see the course notebook corresponding to your custom installations
Fisrtly, we like to thank Jacquemet and Henriques laboratories for making these notebooks open-sources. There is a long list of contributors associated with the project acknowledged in our related paper and the wiki page.
Lucas von Chamier*, Romain F. Laine*, Johanna Jukkala, Christoph Spahn, Daniel Krentzel, Elias Nehme, Martina Lerche, Sara Hernández-pérez, Pieta Mattila, Eleni Karinou, Séamus Holden, Ahmet Can Solak, Alexander Krull, Tim-Oliver Buchholz, Martin L Jones, Loic Alain Royer, Christophe Leterrier, Yoav Shechtman, Florian Jug, Mike Heilemann, Guillaume Jacquemet, Ricardo Henriques. Democratising deep learning for microscopy with ZeroCostDL4Mic. Nature Communications, 2021. DOI: https://doi.org/10.1038/s41467-021-22518-0
