This repository is intended to hold a variety of resources and materials to support work being undertaken across a group of UK Research and Innovation (UKRI)-funded research projects. These projects are focusing on enhancing support for "digital Research Technical Professionals (dRTPs)" within the UK academic community.
The specific focus for the content in this repository is on enhancing skills development and technical career opportunites for current or future technical professionals working with High Performance Computing (HPC) infrastructure.
Universities and research institutions have made use of specialist HPC infrastructure to support specific areas of their research for several decades. Originally, such uses were very niche and only a very small number of people had the necessary technical skills to deploy, manage and use these platforms. As the use of HPC infrastructure has become more common over the years, more and more people have needed to develop the skills to manage and use this infrastructure and develop code to run on it.
For many current members of the research community, their first introduction to such infrastructure will have been as a Masters or PhD student, needing to undertake some large-scale data analysis, run large or complex simulations.
Academic institutions or individual groups or centres within these institutions often have access to their own computing hardware to undertake their research. In the past such resources would commonly have been hosted on-site. Given a sufficient amount of resources, these would be housed in a dedicated "machine room" or "data centre". Tours of such facilities were not uncommon as part of an institutional open day or for visiting research teams.
However, things are changing and such resources are increasingly housed off-site, in dedicated facilities that take advantage of economies of scale to more efficiently host, manage and operate resources, often for multiple clients. These resources are no longer visible to their users who interact with them excusively via a command line terminal on their laptop or desktop, or via a web browser.
Why does this matter?
Many current research HPC professionals come from a research background and were inspired to pursue a technical career in research computing after having the opportunity to gain hands-on experience of working with HPC hardware while part of a research group or while studying for a Masters or PhD. Indeed, many research groups who run their own clusters or specialist hardware or instrumentation have often relied on the support of enthusiastic research students and researchers who are keen to learn new skills and who have found themselves enjoying playing an active role in supporting the group's infrastructure.
With infrastructure moving to a model of being remotely-hosted and centrally-managed, these opportunities are disappearing rapidly, just as demand for specialist research computing infrastructure is going through a period of significant and rapid growth.
We are seeing a major shortage of people with the necessary skills to fill the roles available in High Performance Computing teams and it is the writer's belief that part of the issue here is the lack of remaining opportunities for gaining informal hands-on experience with research computing infrastructure.
We need to make research computing infrastructure visible again!
One of the key things we'll be documenting and providing information on via this repository is our work to specify and create a standard structure for a miniature, portable HPC training cluster.
Current plans are for a cluster based on Raspberry Pi 5 boards that will enable us to provide an HPC cluster-like interface to support training for users at sites that don't have access to such platforms. The hardware will also enable us to demonstrate the structure of an HPC cluster and how the key elements of the cluster are designed and deployed.
We will be developing or building upon existing training materials to support both cluster users and also to provide an introduction to the process of building a cluster.
The repository is at an early stage of development but the folders will be developed with content as follows:
-
cluster: Material relating to the specification, provisioning, deployment and use of small, lightweight HPC training clusters.
-
training: Open training materials will be developed and hosted here.
-
docs: Documents and papers developed by the project team will be hosted here.
Feel free to get in touch with Jeremy Cohen who is leading this work.
The text-based content in this repository is licensed under a CC-BY-4.0 Attribution licence. See the LICENSE file.
Any code developed by the team that is hosted in this repository is available under a BSD-3-Clause licence unless otherwise noted. See the LICENSE.code file for details.
Work being undertaken within this repository includes materials being produced as part of, and with support from, the following projects:
- UNIVERSE-HPC: Understanding and Nurturing an Integrated Vision for Education in RSE and HPC (Funder reference: EP/W035731/1)
- STEP-UP: a Strategic TEchnical Platform for University technical Professionals (Funder reference: EP/Y530608/1)
- DRIFT: DRI-focussed training for research facilitators and teams
We thank UK Research and Innovation (UKRI) for their support in funding these projects.
