[UI/UX support for attestations] Stakeholder Engagement & Research - Usability/User Test research anonymised notes

[Erioldoesdesign]

Post recruitment, interview and notes clean-up and approval from individuals that participated. This issue should collect, in a single comment to the issue the notes that were approved for public release by the participants in order to maintain transparency of the design process that follows into the style guide.

#78 <- The interview protocol and guide

[Erioldoesdesign]

Combined notes for open publishing

{Q. Can you tell me a little about what you use programming languages for?}

Data analysis and pipelines tools and Bioinformatics which is the use of computational tools to analyse biological data with lots of sequencing data, data analysis. Anything from making graphs, building big pipelines e.g. taking information from DNA sequence, aligning it to reference genome.

I build static websites / literate programming artefacts and have contributed to an r package or two and working on one within my current role. I also configuring my own systems like self hosting and work in nix.

I have a side interest of reproducible computation in the life sciences and I think a lot about the provenance of information and making sure scientists document how they produce their data, so others can reproduce.

Started off as a front end developer, used to build websites (landing pages), worked with a few orgs. My current role is a technical writer with a (company type redacted) company in (location redacted). I work with developers to write their documentation, do some indepth programming architecture - talk about languages, apis, etc.
Languages most familiar with are javascript and a little react.

In professional capacity, I work with javascript and develop a hardware testing framework at a company. We use open source testing libraries as well. This is a base for testing, we use [Test Anything Protocall](https://testanything.org/).

I use utility libraries in IOT work, propriatary tools but we’re trying to open up and become more open source. Be an open source citizen. I maitain hardware testing framework to make hardware testing easy - buildng the [open-source hardware in the loop pipeline](https://blog.balena.io/migrating-balenaos-cicd-to-github-actions/). These tests run directly on the device to give you feedback without needing to touch hardware devices in the ‘field’. As replacement for your fingers to interact with the device - it all happens on the device and feedback gets sent to devs right in the pull request. A 6 week OS release pipeline now takes a couple of hours.

I used pythondjango drf? framework to build robust applications. Expressjs to create API’s it expressjs. Front react, and another framworkwork. I work in QA mostly.

it is a broad topic. there’s many distribution channels. How one org can work on these and influence the ecosystem sounds challenging that’s why I wanted to know who is funding this project. I was an android crypto wallet developer. Wallets need to be really secure and do what they are meant to do as the incentives to backdoor them are huge. Any supply chain issues are really relevant to those projects. I did code review for our team and made sure the releases were secure. Later, (I) started my own project where I looked at third party bitcoin wallets on all platforms. Mobile, Linux, Windows, custom firmware, ... Look at the distribution platforms and analyse if those products are transparent. Check if the binary is compiled from the publicly available code or if the binary is compilable at all. (I) Do not do code review for these third party products. Just transparency. If we attest to the reproducibility of a product, a security researcher could review the product by looking just at the public source code.
We are using whatever the bitcoin wallets are using as per their explicit or implied build instructions, (so if they use Python, we use) python. (With the) wallets we look for compile instructions (to) see if we get the same binary out. If we don’t (get the same binary out) or have (useful) info we take a guess. Most projects (and wallets) that publish source code are not pinning the artefacts that they are using. They don’t pin it by any hash or anything. Security is really lax. Most providers don’t have any idea about reproducibility. (We are) fighting an uphill battle (for more visible binaries/security info etc). (I want the project to) have the binary transparency for the entire product and not just a small part of it.

{Q. Does hardware testing need software that recognizes if physical component is failing?}

Yes and more. These an interface layer between hardware and software (eg lights that show the software is working, sensor coding, etc) and Software layer itself. We make IOT feed mgmt easier for consumers - eg CO2 sensors, satellite info. People who make javascript environments in containers and deploy in hardware - can it run in eg 2 gigs, is my hardware heating in a hot environment, etc. The company is Amazon Web Services for IOT basically, we manage networks like this; we manage friction from IOT management; a lot of javascript components. My part is this hardware testing. Python - I work more on open source; used to work with Python Software Foundation as part of Google Summer of Code; python for scripting and contributing

{Q. Can you think of a recent project and tell me a bit about it? What was its goal? What technologies or programming languages were you using?}

I don’t have anything that is “high security / risk”. Most of my stuff is working with open data and going to be open anyway, so not a lot of security concerns.

Even for my full-time role as a docs writer, i have to use npm. I recently had to install npm about 2 months ago when I was working on design system for my org. To streamline the entire progress with reusable components.

With QA there are methods for software dev lifecycle. Test scenario and use cases are in spreadhseet (delivered by product manager) and then I write a test framework for those. I then compare design and framework to the spreadsheet information to assess the quality of the output and also the code.

For automation, I use various languages and several tool. What I’m working on now is built using [python](https://www.python.org/) and [selinium](https://www.selenium.dev/) and [wordpress](https://wordpress.com/) [[playwrite](https://playwright.dev/)](https://playwright.dev/). Oh I also use for the application [npm](https://www.npmjs.com/) and [firebase](https://firebase.google.com/) api tool.

This work is done as a contractor for a business. A project working on QA enginering for legal automative industry. I work for lots of different companies that ask for dev tech work for just one company at the moment and then will move on to another.

(the project is making sure to) assess the safety, security parts of the products. We try to convince people that public source code is not enough (and that) transparency is important. Most people don’t understand that even if the wallet is open source, using a pre-compiled binary doesn’t guarentee that the wallet was compiled from that open source. So we’re trying to advertise reproducibility for binary transparency so a backdoor would need to be added in the open aka public source code.

I did security code reviews for one wallet and it was a full time job. With LLMs, it might be feasible to scan all publich source products for backdoors but currently we can’t provide code reviews for hundreds of products.
It is hard to decide what’s okay and what not when it comes to integrating libaries. My boss might want a (UI) carousel in the wallet interface and the library) is megabytes of external code. A designer might think about using this rarely used library for design reasons and I would prefer not to have this as any dependancy is a security risk. I did this in my prior job but now i don’t have the insights or the leverage to analyse the dependencies of the the many wallets we monitor. I wish we had the resources to do that. But we are developing tools to accommodate for more external reviewers.

{Q. Did you install other packages for that project?}

no, mostly npm but when i was doing stuff for CHAOSS (a community OSS project), i installed pandas (PyPi).

{Q. Tell us more about react development?}

What exactly would you like me to speak on? {answer: what you’ve built with react}

When i was starting out with tutorials and everything, it was personal use. A website where users could sign up, select pricing and purchase a product from an application. There was a login and sign-up form. It was software as a service.

{Q. is there a project where there needed to be careful consideration around legitimacy? i.e where institution would want in?}

Bioinformatics pipeline project would fit this description

{Q. Is your project open source?}

Yes

Yes

Yes

Not the client or overall project but the tech/languages/packages they use is OSS. [Django](https://www.djangoproject.com/) is in current stack.

{Q: Thinking about that project, can you recall the last time you had to search for and add a new dependency or package? Tell me about that. What did you do? What packages did you install? From where?}

Do the dependencies do the things we want? Does it implement the statistical method we want to apply? In the r ecosystem packages come from [Cran](https://cran.r-project.org/), [Bioconductor](https://www.bioconductor.org/) - both have curation / review policies, so I tend not to worry about their basic security considerations, as they have to pass a review process to get into these repos, which is different from PyPI which is open to anyone.

(When adding a new dependancy) I probably would ask chatgpt what is the best, then I would look out for if the package has been maintained; no one wants to install a package that hasn’t been maintained in a while. Is there regular communication or documentation about the package? When was the last commit. I would check the documentation (because i’m a technical writer now, so i understand good documentation) and make sure it’s simple, clear to understand. (Other ways are) If people know about it and If a lot of people use it, a package would have very low security risk and if it is not an entirely new package.

I run weekly maintenance meeting. I’m also docs lead at my company and I update a lot of these packages for the website (react) lots of dependencies come with it. I use small extension which looks at json package and gives me next upgrade for each package next to it; i don’t have to look at the npm interface itself. Where I can see package updates coming up with click of a button. We can do patch versions if we don’t update the entire package. Can check for packages that don’t break anything. Can see if current security issues have been updated during NPM install (npm has security plugin that’s helpful). We’re trying to fix our docs framework (javascript based). Process is how I become aware of security issues in packages I use and how do I securely update them.

For something new I look for the Python code of conduct, and ask myself ‘does the project have this?’ Like with PyPi, I go to the main website to check it’s legit package I want to install and go to python software foundation to check if it’s legit. Then I look for active maintainers and check on linked in whether they are the same positions on linkedin (as the package website on official channels). I also check their work history (to understand more about their credentials/experience) and do a security check on the actual package/code/project like, does it have vulnerabilities that might affect my project?
I (also) organise events (so I look for legitimate people from my known community)
There’s communities on discord and slack that are where security (minded people) track code and where people list their own issues in PyPi and other packages. I also check on the docs quality and compare that with what I think good docs need to include.
{Where do you look at the dependancies of the packages?}
I check on the main website (PyPi main website) and then i check on contributions github or gitlab - i then try to run it and see if it comes up with errors and if it’s my environment or the package. I’ll then look at comments on the readme and stackoverflow and resolutions made by others.
I check all the channels I’m in (slack etc.) and check the network that I have. For example, since I belong to the python software foundation group (slack), I go to the channel for PyPi and because I know the infrastrcuture person, I check his updates and check on issues there with him (via @ing or direct messaging him). Because It’s faster!

It depends on the platform you’re developing for. If you develop on android then you use google for the options then you judge if the repository looks solid and well maintained. If it has 100’s of contributors or 1000’s of users/viewers (that looks good). If it doesn’t look good then better keep looking. If it’s not well maintained then it might get obsolete quickly so we look for recent releases etc. If it doesn’t have many users then nobody is looking at the security of it. I gotta go with the gut feeling.
You cannot prove that something is secure until it’s not. It’s hard to decide with the information that is easily available. I prefer using libraries by big orgs like Apache, Github, Google. Not trusting this is pointless if it’s on 1000’s of devices at the root level. With small libraries we also look into the source code and judge by that. A nightmare/bad situation was when my boss needed to integrate another crypto token into the crypto wallet. It would get access to the private keys of the user and we had to make sure that it wasn’t abusing keys it had no business touching. Android is great for isolating one app form the other but in some wallets you have multiple libraries from multiple cryptocoin developers that all have access to the keys. My boss only integrated relatively few crypto tokens. I had to sign off on those integrations. I had to get comfortable with it and had to review their code as there was no other way for me to be comfortable with letting their code anywhere near our users’ funds.

{Q. Other ways to trust package if it’s very niche or hard to find info for?}

Scenario: We use a small tool - multiplexor to switch SD cards from one device to another; raspberry pi is powered through sd card which flashes through my laptop so you don’t have to use your hands to change SD cards. The package that does this is a very small package. So some people used packages and there is just 1 option.

What we do to assess is first, we look into licensing and then we copy the code out - has to be done incase code is lost or they do update and something is lost/changed.

• Sometimes we only need a part of the code
• Forking to isolate the code and retain copies for our use cases
• In this use case, doesn’t work if we use npm package install
• Example scenario: raspberry pi releases api hosted in google drive but made my system fail
• Library is top concern for reliability and uptime
• Algorithmic way to check if very big file is bizip to test later in the pipeline if its corrupt or not

{Q: Do you share back process and assessments to other communities?}

Current convo we’re having in my communities: niche hardware in the loop pipeline devs

• https://openhil.github.io/
• Happens way to much - people keep rebuilding the wheel. For good reasons, but companies aren’t very open about what small packages are and there’s hesitancy to figure out what small modules are.
• Company effort - trying to figure out way to give back; our code is open source and we write blogs to communicate about the general work
• e.g. When looking into this company that made this project, can I find this one part I need
• Documentation can account for minute detail or process problem
• User journey: Testing hardware page → open source hardware page
• Company doesn’t benefit from this discovery, so not a big business arm, especially for the hardware aspects
• We’re at least not limiting someone
• Internal seminars to help people find things

{Q: How would use access if a lot of people are using a package?}

Would check reddit, stack overview and the (github) stars would be a good thing to check, but that’s not really always the best. Sometimes i like a project but i don’t necessarily star them.

{Q. have you ever looked that integrity process from Cran?}

I don’t recall because it’s been a while since I looked. It’s mostly technical review, does it comply with their guidelines. They look at content, they check it builds, etc. Not a detailed security audit of package content. A “sniff test”. A malicious actor might be able to get something in (note - more likely attack vector is compromising an existing package which might then get less scrutiny). Most of the packages I install are quite well known, so one would hope that community would see problems.

{Q. Can you tell us briefly about the role/function you are in/perform in relation to your understanding of validating the security, legitimacy or safety of software contain in packages and/or the packages themselves.}

I work in the financial sector so we deal with people’s money and financial info; you do not want to ever be at risk of installing that would cause vulnerabilities on your system or allow people to spy on your system.

My steps are:

• First search is always to go through an LLM
• The LLM shortlists/recommends a package/s which I usually not go with because they don’t do comprehensive testing
• Instead of leaning into suggestion, use it to search
• If I’ve already heard of it and validated e.g. [Lodash](https://lodash.com/), I can go with it
• Can use to find packages and create a list
• Then Human validation - visit github or npm page
• I look at metrics - github stars (doesn’t really matter if package is niche or small), latest comments, used by element
• How many people use it
• Downloads is a metric but not really, bc CICDs contribute to download metrics so may not reflect
• Comment frequency is quite influential - to see if new features being added
• I also look at Issues, specifcally for any security flaws or security discussion generally, ask questions about the repo using github co-pilot and/or available comms channels
• If package is really important to production, look through [Snyk](https://snyk.io/) / socket.dev or other platforms that analyze and provide metrics on npm packages
• https://socket.dev/search?e=npm&q=lodash
• https://security.snyk.io/package/npm/leviathan-worker
• Everything I’m saying I would be applicable to python packages as well

Summary

1. I know I need to use a package
2. Make a list using LLM search
3. Validate using human factors
4. Extra validation from Snyk/socket.dev to understand security risks

{Q. Has there been a scenario where you have looked at a package in a less secure setting?}

If someone has put something up on github, not in a main package repo. I can install it with existing package tooling like any other package.

In this case I go to github repo and do these steps:

1. Look is it active?
2. Maintained?
3. Up to date? (updates, releases etc.)
4. For high security requirement - Looking for things like
5. Signing git commits
6. Is the contributor graph - one person? Multiple people? What is the proportion of commits of each contributor?
7. Affiliations listed? Funded or associated with reputable institution?

{Q. I’m curious about “high security requirements” are things that are happening in CLIs? Signifiers on a web page or landing page? Different processes between UI and CLI?}

If I am installing a system package (e.g. Ubuntu dependency for R package), it will be coming from the main package repo run by Ubuntu. Ubuntu build server → Ubuntu Signs Binary → Ubuntu Creates Attestation → when you install, installer automatically checks the signature matches the binary. Its automated in the tooling, but it does depend on the trust architecture, you are choosing trust, e.g. Canonical.

There are other models, like [trustix](https://nix-community.github.io/trustix/)’s approach where multiple people can build binaries, and you can say “as long as this number of trusted people built the same thing I can trust it too.”

Things like signing of commits in repos is not well represented in GitHub UI.

How to find: look at commit history and then you can see verified badge associated with a commit.

(User shares screen to demo signed commits on Github) My steps:

1. See verified badge
2. If you click on it, you can see my gpg key signature
3. But it still shows up as verified if your email on the git commit signature matches your github email - in that case github signs it with their gpg key
4. When it comes from me, it is time stamped by open time stamp service - says I signed it at a specific time, recorded by an independent authority
5. Important if I ever revoke my key, if my key was leaked. You could tell when it was signed and whether my key was leaked at that point
6. These things under displayed visually in UIs

{Q. when you see this kind of information, what choices does it help you make?}

If I see a package has all signed commits- they are taking security seriously. It takes effort to set gpg keys up properly!! There is a high barrier to entry. Does not give you full custodial information.

Project with security model most compelling: guix project. Lunix distro, based on functional package management model When you describe a software package, it explicitly list dependencies, incl. hash. Functional package management: nix and guix https://guix.gnu.org/

Complete description of dependency tree in the package manager. Full source bootstrap - can rebuild any package in their distro entirely from source. If you dont trust the binary cache, you can rebuild it yourself. You can challenge individual packages - guix challenge command - will check if your local build matches remote mirror. You want to be able to check it on the command line with a simple tool, even if you see the information visually on a UI.

Can I verify what the UI is telling me with a simple CLI? I would want to see “This is how you can check it with the CLI” instructions on any UI - so I don’t have to place too much trust in the web page itself (which could be compromised).

Why functional package management would work nicely in the life sciences: Managing dependencies when you are outside the main language e.g. R package has to install system C dependency. It’s a universal package manager tool, instead of language specific. Still has lock file equivalent In r - .renv (equivelent of .gemfile)

Guix - sources are archived in software heritage (archived in france). So even if the main source is lost you can rebuild it from the archive

[IPFS](https://ipfs.tech/) - the way that works - it is a peer to peer network for sharing files. If you put your source archive in IPFS, the individual package source gets an address based on its content (hash based naming scheme). You have a guarantee that if you request something, you get back the right thing as long as someone is pinning a copy, it is still there. This is resilient to large archives going down due to distributed mirrors which are nice properties for resilience.

{Q. What i am hearing that its complex mental model for how you trust something, including security models, human aspects, etc.}

IRL example: contrast between google play vs [fdroid](https://f-droid.org/). Issues with attestation API, they won’t work because they are not trusted by Google but I might trust it, but as there is an intermediary, I cannot use it.

There are two things to think about:

1. Do I trust the technical architecture / delivery model?
2. Do I trust the developer and the process of how this code was produced? (which is the far more important security question)

{Q: What in your experience are the most important things to consider when choosing packages to install?}

Is it actively maintained? Getting updates from someone? (Assumes good faith maintainer.) Is it getting security updates, version updates? Theoretical esoteric threats are less probable. A package being behind on security updates is much more likely to be a problem.

Big orgs using the packages and they are reproducible and social proof

{Q. When the fingerprint isn’t enough, when you have to really dig into the security of these. what are you looking for?}

When i know the developers and i know they are good actors with high responsibility and they know about reproducible builds and providing it for many years, then i feel like i can endorse their projects. If i feel they have fake profiles then i move on. Is it mainly technical people or some management people that hired somebody that change their developers every other week?

I think that one employee can corrupt a project. The product needs a solid culture of code review to keep it secure.

I wasn’t always happy to integrate non-financial libraries in our bitcoin wallet. A bitcon library is targeted to do something with bitcoin. a button library may not have such a security focus. It’s difficult either way. in the end it’s always gut feeling and trying to get as much information as possible to evaluate how security minded are they.

There are projects that are proudly anon. I would stay away from them if the provider isn’t declaring who he his. It is possible that they have a backdoor as they have no incentive to take responsibility should they get hacked or hack their users themselves. A big company that has a vested interest in not messing things up is more trustworthy.

{Q. Can you tell us why, if at all, the security of your dependencies is important?}

If I work with PII (happens sometimes) - a leak of that sensitive information would be worst case scenario. E.g. if I am responsible to protect it, e.g. patient data. Usually I don’t work with data that is that sensitive.

What we build is critical infrastructure for IOT and hardware.

How do I know If it’s safe? I rely on the community a lot and what people talk about. I will also google it and do research, is it the correct updated version? and check the application i’m working on has that version. Do I need to upgrade my python version? At some point the code quality will detect any problems. I’m not a pro, i don’t know this, but I feel I can rely on online tools to check on it and due diligence like using [coderabbit](https://www.coderabbit.ai/) to check code quality.

For programming languages verses the code of git a packet, if i push my code to gitlab or bit packet and i have coderabbit integrated into my push system this then creates a Pull Request and coderabbit catches bugs and whether code is redundant and then I can use a different code method and also a CI/CD pipeline.

{What about the security of packages not the code you write? Do you use dependabot or what other things do you check with/for? Specifcally for the security of dependancies ?}

I rely on the security team for the project i’m on and a dashboard with installed [OWASP](https://owasp.org/) to check on the security of the dependancies that are being used. It checks the database and its checks the dependancies for vulnerabilities in the project.

For npm i run the audit command. I run this all the time to check npm and it is specific to js. For python - use pip and use the word I need and the flags when i run a command in the CLI. This helps.

Access to the user keys in these crypto projects. A bitcoin focused screensaver or game for example - i would be very careful about using it. It is definitely targeted at Bitcoin holders. How are they making money? are they waiting for zero-day exploits? Those are not OSS generally. When i understand how they are making money then i’m already more relaxed. Accountability and transparency is important, if there’s none I am not touching it.

{Q. Do you look at their dependencies?}

Very yes!

{Q. Have you tried to talk to the developers to find information if it wasn’t available?}

Our project works off the publicly available information so we might ask them to make missing information publicly available.

{Q. Worst case events and series of events to stop them from happening?}

I’m always waiting for the rug pull. i try to sensitise user to this, and personally to use products where this is not possible. When it’s closed source - i have no idea how anybody would trust that wallet. But most do.

For my project, the worst case scenario would be to get the page defaced. They might trick our visitors to enter their passwords. i don’t consider our project a high profile target for any backdoors. I’m nervous about running 100’s of build instructions on my developer desktop. I use containerisation and try to be as defensive as possible against something like that. For our projects dependencies there isn’t really a worst case scenario. we would just start over and i would be back in 2 hours. No problem.
In my prior job if i would have noticed something weird going on, if a dependency was leaking private keys to some rogue server then i would be in a very difficult situation. I could try to leak keys to my server so we can kind of steal the funds before the evil guys. Then we can figure out the rightful owner later but that wold be such a great mess. is this white knight stealing is something one should do? inevitably i would steal funds from people. It would definitely end in court.

{Q. Could you please share your screen and walk me through how you would evaluate this package?}

https://pypi.org/project/pytz/

https://www.npmjs.com/package/axios

https://rubygems.org/gems/rake}

{Q: Familiarity with website? What does it do?}

I don’t think I’m familiar, I can see it’s npm. Basically, this is where you can search for different packages or have packages that you can use when building javascript projects.

I do most with NPM. our website is javascript based.

Q. How would you determine if you should use this package?

First thing I want to do is see the source code and undrstand what is the dev process?

Authors email is there, it’s nice to have human name associated with a package. Verified details thing with a checkmark, it goes to docs which means that the URL for the package is under the control of the maintainer. Relationship between package and maintainer has some degree of verification.

I’ll look at release history, there’s a long standing history back to 2004 which is a good indicator that it has an established status.

I’m struggling to find the github repo. (Finds personal website of author, Finds files, Finds Licence, Finds attestations). The package does not have github link, but I found it via attestations UI element.

The user Inspects github and sees lots of upstream & downstream dependencies and looks at contributor graph.

First, i’d check the last published (saw that it was a month ago, which is good). The size (file size/package size) is really nice, no one wants to install something that will slow down. It has a lot of weekly downloads, which means people are actively using it and it’s being maintained.
Check the github repository - has a lot of stars. I think this would be a nice package to install. I’m sure that it’ll be maintained for a while and it will be used for a while.

I have already seen this package before. It’s a good sign it was published 1 month ago. I go to the repo - i like github more bc this whole page (on npm) is a readme.

Download number doesn’t matter anymore in our industry (especially if we can assess the reliablity another way). It’s a good metric but not for security esp bc big packages can also be compromized.

On github I look at:

• Stars
• Lots of activity on the page - big numbers
• Most recent commit is last week (different from npm publish date)
• Used by number metric in the GitHub Right Sidebar
• Opencollective page - i trust the sustain community and they talk about open collective; i’ve known them for a while
• Check out the package website
• Readme is a good place for me
• Will it work on my system? (checks beyond security)
• Does it have a mac os situation
• CI/CD library?

I will also look at issues

• Before i would just type in security in the (issue) search to see how the community handles any issues; what issues pop up
• Do they have a security page
• See how maintainers respond
• Everything has a security flaw so I want to see that things are being reported and responded to - it’s a good sign

Chat with Copilot

• “Can you find security issues with this package that people reported in production?”
• I don’t fully endorse any tool - sometimes doesn’t work well, not the most intuitive but correct prompts can give helpful answers
• Faster way to go through the issues search

I’m gonna open another tab and check if it’s the correct package (by searching) - i’ll go through the docs (for the project) and see if i know about it or don’t know it. I’ll look at what the steps are to install it and is it compatible with the project i’m working on (from my current knowledge)?
Docs first and then security information. I’ll either google the info or use the tool/website names i mentioned previously. I’ll ask an ai ‘is this the best timezone conversion (package)’ for my code? ai is last option to ask. I would also check if there’s any issues/discussion on stackoverflow and I’ll see the answers and up votes and how many people have used it.

The name rings a bell. I’m not a ruby developer. There’s 12m downloads. It might be important for security community.

First thing to instil trust was (the name) ‘Rake’ - have used it before. I’m not a ruby developer so i’m not sure how it works but it seems familiar. I don’t know the creators/owners. Weird that there’s only avatar and not the names (usernames). I would like to see the names of creators/owners/maintainers.

I don’t know what ‘pushed by’ means. Does it mean ruby gems get’s updates via GitHub? It’s only a logo (without more information avialable via a link or text or hover). If i didn’t know that Octocat was the GitHub logo, it wouldn’t have been clear. If anyone has access to the GitHub, they could also be pushing. This sounds like the security depends on GitHub not being compromised and ruby gems not being compromised. There’s some checks that are advertised. If i do my job i would go look at that gem and check the hash and review it. I’m not doing it for ruby gems i’m doing it for bitcoin wallet so perhaps someone else is doing it

{Trustworthy elements of the page}

Top right of the page. It has relatively recent release that got 12m views. 1B total downloads. 12m downloads is maybe not 12m users? Each dependent project might be counted 100’s of times depending on who downloads and when. But perhaps there are millions of users. Tragedy of the commons - if anybody can look into it, does anyone look into it?

It looks like a solid project. i would be ready to try it out if i needed it for my project.

{Q. Is there any information you're looking for that you can't seem to find?}

Source code / code forge where the projects is managed, issues, metadata etc. (github link)!

{Researcher note: this is a problem specific to this package, most have github links}

I think they covered a lot of things. I check documentation to see if it’s good and I think it is good, they explain how to install it and how to get things running

Was hoping metrics could show up for npm audit so you don’t have to download the package itself to see

A UI feedback is that it could go right under last published

• Github does have a security page
• More official declaration you have npm provide a report card or something (like the openssf report card)
• So people can reliability get big picture for whatever package they’re looking for
• Axios page shows it monitored by snyk but this is more easily accessible than a lot of other pages
• NPM could also have security tab now (next to versions)
• A way to report security issues
• NPM has report malware but i’ve never seen this before
• Feels like I’m just contacting the npm team for something generic
• I wouldn’t use this page to report issues
• Let maintainers take control of what “report malware” button does - could go to report issue template
• There’s lot of intermediate things before malware - malware is usually the least likely
• Attackers or data leaks are more likely than malware
• PIP
• Verified by pypi button is nice - see what it means and if its a metric i need
• Built and signed on github actions maybe doesn’t mean much bc anyone can be verified

(When looking at the Provenance section)

I’ve never seen this section - it’s a new discovery. Clicked to it from the Verson → Built and signed on github actions → view more details

• Built and signed on github actions means almost nothing
• Low accessability
• Lots of people don’t know terms like provenance and governance
• Security name could be better - security team or security documents
• This section provides less info and can be improved. Need to explain what the things are. Communicate trust component.
• I can look through the 3 links
• Public ledger is quite nice. never saw it before.
• We use github pipelines to store ours. would be more accessible to store everything together (can show later)
• Can prove this package is validated but this is something I don’t do that deeply
• Nice bc these are computer admins - proves that what the maintainers built —> got built and was published correctly
• Hard to find, probably don’t come down here often
• Checkmarks are very misleading - I wouldn’t expect it to be about security, rather they would be verification.
• Docker uses the same situation; official tag doesn’t mean much bc they’re following supply chain guidelines but it doesn’t improve security confidence for using the package
• Lock icon would be better
• Differentiation between supply chain security versus package security - insecure packages can still follow supply chain security

Summary

• There are common ways to understand how good a package are (eg weekly downloads, etc)
• Big numbers overpower some other useful information (eg provenance)
• Provenance gave confidence that there are artifacts and ways of interrogating the package

{Q. What didn’t you find on PyPI?}

The source repo

The stuff that would raise red flags is information about the dynamics of its development:

• Created by a user with a new account
• Not long history
• Dependencies
• No dependents
• Number of contributors

Red flags are very context dependent though:

• if it is community / independent open source - perhaps expect fewer major contributors, (might have a lot of ‘drive by commits’), than a package with a corporate or institutional affiliation.
• if it is a niche package - perhaps less active, etc.

{Q. What is your level of confidence in using this package right now, and why?}

It passes “the sniff test”. If I was doing an evaluation in a high stakes security project, I would inspect further.

With this one, I mean it’s a pretty popular package. I’ll give it a 100%. The sponsors are people supporting it and they are big names.

Using Axios as the example, If I see something that was last published even a year ago, I probably wouldn’t use it. If I see 2 years ago, I wouldn’t use it. Idk if people are actively maintaining it and if it will be difficult for me to understand it. If I don’t see many downloads, that would be pretty tricky. Going back to github, if there are no stars, no new releases, no proper licensing.

I wouldn’t install this package because It doesn’t have a readme. I didn’t mention this earlier but every good project since I’ve been doing open source work, I’ve noticed the prescense of a good readme. That should be the first thing people see. It was published two years ago and I don’t think i’ve seen any activity on it since. It has very low downloads.
I’m unable to find source commit to this package. It means they (the site) can’t find the source and it’s probably not available on github anymore. Anyone installing this would be playing a risky game also the size (919 bytes) and 2 files (which is big).

{Q. Thinking about the page we just looked at, how would the information presented here fit into your organisation's compliance or vetting process? if they have an organisational process they are legally able to share}

I find web of trust very interesting. The username or avatar is not unique. I don’t know if it’s unique. Maybe the username is not unique and the avatar is not unique. Some look like letters also called rake (typosquatting). They do some SEO to get people into the copy rake and not the actual rake. It’s not very solid but i have to move on. I don’t have infinite time to investigate. I cannot bootstrap every library. I have to trust some level of pre-existing tech and go from there.

I would love to see how many people i follow endorse these accounts. I’m excited by [nostr](https://nostr.com/) (a social protocol). you can have followers in different products. That would give me confidence. Engineers that i follow on nostr, i trust them.

NPM is not doing it better. it’s actually worse. 42-cli which was last published 10 years ago ... if i see last published 10 years ago, forget about it.

https://www.npmjs.com/package/42-cli. On click i would get user profile on npm js. They aren’t showing stars on GitHub. GitHub de-platforms people too. In some countries people cannot use Github like Iran. I don’t think it’s the best solution, but it is what it is. if i see 10K stars then i trust it. I might not trust every release, like if there was one 12s ago so i might hold off (on using the package).

There have to be some people checking (the package/project/source-code etc.)

There are people that are more security minded than others. I always recommend people to stop auto-update by default on Android. Any bitcoin wallet provider could push an evil update that would immediately leak the keys for later rug-pulling and by the time, Google play store removed them, they would still get 10000’s of wallets.

{Q: What would you look for to determine the safety of this package?}

(in regards to safety of packages) Personally I don’t think I pay that much attention.

Before gemini ai i would go to google and ask is the package safe? or ask on reddit and stackoverflow and go to the Python Software Foundation website. From my understanding the command is safe and i know it’s safe because i know maintainers.
Use the url to check and I will try to run in vscode in my project with pip run and see if it gives errors. And then figure out if it is my code or the package? Using console f12 and see if theres an issue or not. I would then go to a community page and post issues and see if someone replies.
Ask for expert feedback from maintainer to check if package is safe. After i’ve explored all the options. first I try it myself and last resort is asking a person.

When I was doing full software development work. I often got blocked because there was a vulnerability in my side. I was working on an essential safety app for kids and parents to communicate. Someone (another dev) put in a code from another source and it impacted our project. It was an SMS function. We had to had to change the application to meet the needs better but this was a code error and not a dependancy so the code was copied badly and was not an insecure dependancy.
I forked a project on github and was using python v2 and couldn’t support what I needed.
I’ve never found a package that is not secure. I just go to documentation and see it’s secure and I’ve never had to remove a package. Only if the package wasn’t correct in doing what I needed it to do for the project.

{Q: Does this give this package any authenticity?}

Makes me feel more confidence, because github is a universally known hosting site. Or should I say it doesn’t really matter. It could be that other projects are really safe, but for this particular project, it being on github improves authenticity (because of the other factors). Other projects being on github wouldn’t necessarily mean they’re more authentic.

{Q. if you wanted to interrogate it more? you’d look for gpg?}

An enforced all commits are gpg signed policy is a kind of an indicator that the authors take security seriously. Unless it is really security critical package, I wouldn’t expect to see it. (even better would be something like guix git authenticate where you can easily check all commits where signed by someone with the permissions to be a commiter at the time they were signed)

{Q: How did you learn how to work out if a package is a good package? (eg tutorials, experience, guidance, etc)}

I don’t think I got it from any tutorial but It’s a matter of as you grow and learn more, you know things better. Maybe someone pretty new to development might not check for certain things. When you’ve spent a number of time in a particular field, then you know things that are a red flag. There are some things that are glaring. In one of these packages, there are a lot of issues. Even if you’re a beginner and you pay attention, you should be able to tell that this might be a risky package.

{Q. What are you looking for? What do you need re: supply chain security?}

The supply chain, once you have established the project is trustworthy, I need to establish that its not getting intercepted (man in the middle attack). It’s a holistic thing - it needs to go all the way back to the authors, distribution methods, etc.

One of the things that helps, is reproducible binaries and being ‘Signed’. That way I can check it corresponds to signed version, the originator sent out. If you are using a package mirror, you don’t even need it to be an HTTPs connection, as long as the item is being checked by an independent protocol. I know the binary is right as it was signed by the key that I trust.

I could imagine a (UI) dashboard to see these pieces of information when I go “poke at a project”

(That UI could Tnclude) hings like:

1. How long the projects existed
2. Contributors - total commit, etc
3. Institutional affiliations
4. Proportion of commits are signed by gpg not from github

{Q: Software supply chain - what does that mean?}

I don’t think I’ve heard of that. Is it like software development lifecycle?

{Q. Can you tell/show us what you understand of the terms 'SBOM' or 'provenance' or 'attestations' are? If you have examples of where you've seen these terms or encounters these terms that'd be helpful.}

SBOM is a Bill of materials. What are all the dependencies? What am I taking on as an attack surface? A smaller list of things in the SBOM is less attack surface, less concern for dependencies breaking. But functional package managers don’t need a seperate SBOM because you can query it (better UX for doing this is needed though). If you are creating an SBOM for something that does not behave that way, I have to trust the process that made the SBOM. They mean different things based on how they were made, for example if it’s manual, I want to know how was the SBOM made? for automatic, it’s more integrity.

Provenance is where did the thing come from, what are its origin and the process which produced it.

Provenance doesn’t mean anything to me. Feels like it’s where it came from; I know it’s hosted on github. But doesn’t have anything when I search.

Attestation/s is someone or some system is signing their name that guarantees a piece of information. The fact that this binary is what they got when they compiled. This term is used in many different context. e.g. gpg signing, crypto time attestations.

Attestations are very useful but often used in a flattening sense. When a project says ”We attested this thing” I want to know, what is being attested? and who is attesting it? You can’t put a green checkmark without having the context of what it means. (follow up note – ‘verified’ badges can ben an issue if it is not very clear what is verified and by whom as they can give false confidence if gamed/compromised.)

SBOM, I’ve heard about it, it’s a security feature that helps check on the components and vulnerabilities in dependancies it does the best things that you can use and looks at the libraries and checks on the software applications.
Provenance = not heard about it. Does it mean it’s quality? it doesn’t mean much to me in a developer context but the word might mean something like ‘to own’ something.

{Q. What do you understand about package attestations? Do different types of attestations hold different weight for you and/or your organisation?}

We try to do attestation of reproducibility. The bitcoin core project attests to bitcoin. In my project i do attest to wallet binary reproducibility. It is kind of a singed message, attested to certain binary being reproduced. I know that bitcoin core does that a long time.

You attest to it being either what the source code says it does or you attest to it not talking to the wrong servers. it’s just about the transparency. we attest only to the transparency.

{Q. Is that knowledge of attestations automatic - when it’s signing. Info after running commands. is that the way you’ve interacted with them before?}

We do attestations where we deliberately create a message that we attest to the reproducibility of this product and this is how we reproduced it. We add a full process log to it. We show how the command line worked. We say whether it failed to built from source. It is a very long, signed nostr message.

{Q: What does the word NPM mean to you? How would you explain npm to someone else?}

Honestly I just know it to be npm. I’ve never checked for the meaning of npm. Whenever we say npm, we just say npm install. I think I would tell them that when you see npm, it’s like a popular to install things into your code or whatever project you’re running. it’s how you can bring things in

{Q: How would you describe packages?}

External tools that can in whatever you’re building. They can edits. Reusable tools that have already been made by people and you don’t have to start fresh.

{Q: What’s a dependency?}

Things that your project would need to run to work. It depends on it to run (correctly).

{Q. Any questions or last thoughts before we finish up?}

One thing I’m not clear on - UI design project? A UI for What?

{Researchers clarify that a styleguide to be piloted with different programming ecosystems}

{Q: Can you tell me about a guide/documentation, that you've used in your work/role that you find easy to understand and useful?}

Here are some links, papers, & documentation. A lot of this is quite technical and not a lot is very visual I've been collecting sources for some writing I've been doing on computational reproducibility related to a project create checklists for research software producers https://rsspdc.org/ so I've got a bunch of references ready to hand. If I think of anything more visual I'll let you know.

Reproducible computation

A recent masters thesis which is quite readable and an excellent overview of the subject https://zenodo.org/doi/10.5281/zenodo.15315531

what are the prerequisites for making software reproducible? the four possibilities https://doi.org/10.59350/81ba8-dse91

Draft currently under review of a contribution that I'm making to the turing way about this subject

text: https://deploy-preview-3741--the-turing-way.netlify.app/reproducible-research/renv/renv-considerations-limitations

review: the-turing-way/the-turing-way#3741

Functional package management

"The Purely Functional Software Deployment Model" Eelco Dolstra's PhD thesis on Nix, whilst this gets technical the introduction is quite accessible and a reasonable statement of the problem https://edolstra.github.io/pubs/phd-thesis.pdf

Does Functional Package Management Enable Reproducible Builds at Scale? Yes https://arxiv.org/abs/2501.15919

GUIX's supply chain security model http://arxiv.org/abs/2206.14606

using guix git authenticate in practice: https://guix.gnu.org/en/blog/2024/authenticate-your-git-checkouts/ I employ it in my personal webpage to attest to it's validity https://richardjacton.net/

Nix & IPFS https://blog.ipfs.tech/2020-09-08-nix-ipfs-milestone-1/

trustix - Distributed trust of binary builds https://nix-community.github.io/trustix/about.html

Full source bootstrap

https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/

Challenges for open source 'suppliers'

"I am not a supplier" https://www.softwaremaxims.com/blog/not-a-supplier

Source archiving

Source Code Archiving to the Rescue of Reproducible Deployment https://dl.acm.org/doi/10.1145/3641525.3663622

How Software Heritage ensures reliable Guix deployment https://www.softwareheritage.org/2025/05/21/software-heritage-guix-deployment/

Cryptographically attested timestamps from independent sources https://opentimestamps.org/

We didn't touch on quality / correctness as much in the discussion - automated testing is something I'd look for there as an indicator too

I would say mine, but…Github is a big enterprise, so I think they’ve gotten their documentation right. Arrangement of documentation page is really good - break things down. You already know what you’re looking for and where you’re going. For example, I’m using github copilot (we use this a lot in CHAOSS). One thing with good documentation is can people get what they need to know w/o reading through 100’s of pages. With copilot page - there are different articles I can use get started. Good organization of information; don’t have to read through info that’s buried deep in article.

Depends on the type of documentation, a general site you mean? Selenium is great, i use it a lot and I can’t cram all the functions and commands into my brain so it’s great as a reference. Useful because it has full reference commands to use.

A roadmap documentation is [roadmap.sh](http://roadmap.sh/). It’s beautiful breaks a subject all down for me with good references. It has everything that i need and lots more topics as well.

[Erioldoesdesign]

This shouldn't need reviewing as the doc has already been reviewed internally before publishing here