Containers Working Group for HPC Survey

[vsoch]

This should be discussion around the Containers Working Group for HPC Survey idea.

[vsoch]

@ArangoGutierrez let's put your follow up ideas/discussion here?

[ArangoGutierrez]

Idea 1:

• I think we should partner with HPC/Research related marketing orgs like insideHPC , HPC wire and related to sponsor and help us expand our reach when going GA with this poll. they will benefit from the final reviewed outcome for sure. so is a win:win they help us reach more users, and we will provide them a nice blog post analyzing the results, Thoughts?

[vsoch]

I think we already agreed yes on this one? But partner scoped to reaching out to them to advertise taking it.

[ArangoGutierrez]

Yes, I just wanted to bring the Slack conversation here, to the open

[sk8forether]

Hi there! I'm adding a few ideas I shared in slack initially and wanted to further discuss here. I think these are the 3 major questions we need to answer - following good statistics practices - before we are sure we are framing good questions.

What is our target population?

• Are we focused on RSEs? Sysadmins? Anyone/everyone using containers for HPC? Anyone using containers for any type of research (i.e. including not for HPC)? (I see that we ask what the individuals’ job function is, but curious what our target population is) Our questions right now seem to focus on the users of HPC systems, I think.
• Do we intend to focus on results pertaining to HPC? As in, maybe we collect results from anyone who is willing to fill it out, but we are really concerned with folks using containers for HPC in some way. Said another way: Are we interested in results from folks who have only ever used containers for non-HPC tasks, and thus may have a biased opinion about HPC containers?
• Do we care about people using containers in the cloud for HPC-style workloads? For example, Docker in kubernetes cluster, deployed by the user (so they are also the sysadmin/DevOps), etc.
• Do we want to reach people worldwide?

What are our sampling methods?

It can be tempting to just throw a survey up online and hope that we will get enough respondents, but there are benefits to being intentional about how we approach it. Here is a good summary on Sampling Methods for Web and E-mail Surveys that might help us to decide on our approach once we know what our target population is. Of particular note for why we should care about sampling methods:

"... surveys based on large convenience samples that are likely to yield less accurate information than a well conducted survey of a smaller sample."

Ideally, we want a representative sample. Some questions I had were:

• Are we presenting this as an "opt-in" survey that anyone can fill out? (Non-probability/convenience samples) I'm assuming we are for the following questions.
• If so, how do we plan to handle the sampling? Being aware of which sampling method ahead of time can help us reduce and/or account for the bias generated.

What is our plan for analyzing the data?

• What are our objectives of the survey? What do we want to determine? We could put this into a nice neat list.
• For each objective, we should define the data type/level of measurement of the result (nominal, ordinal, ratio, and interval). Then each question should be designed to gather this measurement.
• We can consider how to aggregate, visualize, and analyze the data in a more streamlined fashion if we have this info.

I would be happy to be involved in the analysis side of this, if it is helpful. 😄 Depending on what folks think about this, I could start putting together a list of objectives with measurement levels based on questions already listed (we're almost there, I think).

[vsoch]

I was just going to visualize it. I don't think there is much analyzing to do.

[sk8forether]

I guess it depends on what is meant by "analyzing," but I tend to include accounting for bias as part of analysis. If we simply drop a survey into the void and visualize whatever results we get, then we are inviting biased conclusions that do not represent the population well.

[vsoch]

I think we just have to be transparent about who is taking the survey (e.g., show the question about people's roles to show the bias). Akin to https://stanford-rc.github.io/stanford-software-survey/results/job-title/ and https://stanford-rc.github.io/stanford-software-survey/results/are-you-an-rse/

[sk8forether]

For continuity of the conversation here, reposting the following from slack.

That survey’s goal is:

Assessing the heartbeat of open source software at Stanford University for awareness, understanding, and support.

I am unfamiliar with the Stanford Software survey, but there are a couple differences I see in the goals:
The target population is much smaller, and more measurable. There are clearer bounds on who is and is not part of the community. It is also easier to sample via opt-in methods and be confident you’ve achieved a representative sample (or at least have some idea when you have not).

Additionally, I think the desire to show what our community is made of adds complexity beyond just reaching the population. Characterizing such a heterogenous population (defined in my mind as people who use containers for any type of research - based on your earlier response) is non-trivial compared to a subset of a campus population.

[ArangoGutierrez]

I think we should aim for something more than visualizing some results, I think many are interested in this results. I Agree with @sk8forether point of view

[kfox1111]

some more options / writeins for registries. gitlab.com, private gitlab, harbor are more options.

[sk8forether]

Just in case folks are looking here, I opened PR #40 on this topic. Planning to make some edits later today/tomorrow after folks have had a chance to review it.