Improvements to Episode 2 (Working on a cluster)

[Sabryr]

Notes for possible PRs (I haven't gone through all the material yet, so some of the following may be addressed later).

1. Why shouldn't we run long jobs on login nodes.
2. A diagram to show the login process from laptop to login node via internet and how the login nodes are used to interact with the worker nodes.
3. Elaborate more on, availability of common storage, place in one location and access from any worker node (much faster than through internet). A diagram to show all compute nodes are connected to each other and to the common storage.
4. Photos or CPU and memory and disk (to show these are not mystical stuff but physical objects)
5. When showing nproc, sinfo and free -n, show "df -h" to see available storage locations.

[psteinb]

Hi Sabyr,
thanks for opening this issue. I fully support 1 & 2. On 3. you mean to discuss the shared file system, right? On 4. I've had mixed success with similar approaches, but still think that its a good idea. On 5. this part needs to be reworked as it currently is too SLURM and site specific. Further, there is quite a large possibility to talk about concepts that have not been touched.

[Sabryr]

Yes shared filesystem. I agree with your other comments. Your earlier material had a configuration option to easily change from SLURM to other. Is it OK to start a new issue to try to transfer that here, or was there a particular reason that approach was abandoned ? .

[psteinb]

It's still the plan to implement the configuration trick you mentioned in this repo. Feel free to open an PR if you want. I already adopted lesson-outline.md in this direction, which might work as a guideline. However, at the last video call, people expressed strong interest to reiterate on this lesson-outline. So I am waiting for any more contributions until that discussion has come to a conclusion. @ChristinaLK @devbioinfoguy

[ChristinaLK]

We should probably nail down who is adding the configuration trick. I thought @aturner-epcc might be doing it, but at this point, I think anyone who adds it would be a hero. :)

[devbioinfoguy]

Agreed. I think everyone is looking forward to this.

…

-B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bob Freeman, PhD Director, Research Technology Operations @ Research Computing Services Division of Research and Faculty Development HARVARD | BUSINESS | SCHOOL Baker Library & Bloomberg Center B96 25 Harvard Way, Boston, MA 02163 617.496.6612, phone @DevBizInfoGuy http://bit.ly/whatAboutBob2 For more information – RCS: http://intranet.hbs.edu/dept/research/ Software: http://intranet.hbs.edu/dept/research/software/ Rsrch Grid: http://grid.rcs.hbs.org<http://grid.rcs.hbs.org/> Training: http://training.rcs.hbs.org<http://training.rcs.hbs.org/> From: "Christina K." <notifications@github.com> Reply-To: hpc-carpentry/hpc-intro <reply@reply.github.com> Date: Wednesday, January 30, 2019 at 11:06 AM To: hpc-carpentry/hpc-intro <hpc-intro@noreply.github.com> Cc: Bob Freeman PhD <rfreeman@hbs.edu>, Mention <mention@noreply.github.com> Subject: Re: [hpc-carpentry/hpc-intro] Improvements to Episode 2 (Working on a cluster) (#79) We should probably nail down who is adding the configuration trick. I thought @aturner-epcc<https://github.com/aturner-epcc> might be doing it, but at this point, I think anyone who adds it would be a hero. :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#79 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHO3gAtx-K9Pw3FHT2cH0rs-OxpLQbX0ks5vIcMCgaJpZM4aTVpA>.

[aturner-epcc]

It is in progress at my end - hope to have it in the next couple of weeks. Worst case by the end of Feb. I'll assign this to me.

[psteinb]

You mean this in PR #73 ?

[Sabryr]

@aturner-epcc , I could help with this if you need. Is it possible to submit a issue specific for this (which issue did you assign it to yourself ? ) and a WIP PR. (sorry for pushing I would very much like to use this material in a workshop)

[aturner-epcc]

I have stopped doing anything until #73 is merged and we make a plan for how to take this forwards. I do not want to tread on @psteinb and others toes and was only doing stuff to be useful, not to take over this, it was all Peter's ideas and he should get the credit.

I have a working version with auto-includes for PBS Pro on our system that required a few more customisation variables than in Peter's original implementation. Should we open a new issue specifically to discuss auto-includes so we can capture everything in one place going forward? I feel like we have struggled as discussion is spread across multiple threads....

[psteinb]

@aturner-epcc thanks for stepping in. I totally agree. And yes, please open an issue on auto-includes. I am curious on what you found.

[aturner-epcc]

Autoconfig now in via #101 . Need to reassess the original issues raised and split out into separate issues that we can work to address.

[tkphd]

Broke out item 1 as #250.

[tkphd]

Items 2 and 4 have been resolved by @Sabryr's contribution of original artwork.

[tkphd]

Item 5 was resolved separately (df -Th alongside nproc and sinfo exercises).

[tkphd]

Broke out item 3 as #251. This combined issue is deprecated, in favor of discussing the specific topics separately.