Add new spec for go package URLs
#338
Conversation
There was a problem hiding this comment.
This is a breaking change that affects all software utilizing PURL for Go. Personally, I don't think there's anything fundamentally wrong with pkg:golang except that the description is outdated, and I'm sure it can be fixed without making this level of breaking change. Maintaining the separation of namespace and name and putting the entire Go package ID into the PURL name makes PURLs difficult for human users to work with.
PURL-TYPES.rst
Outdated
| ------ | ||
| ``go`` for Go modules: | ||
|
|
||
| - The ``namespace`` field is empty and implies the go mod proxy. |
There was a problem hiding this comment.
Is the field empty or does it imply the go mod proxy? It can't be both.
There was a problem hiding this comment.
Should be done now, see the new commit (sorry for that).
PURL-TYPES.rst
Outdated
| - The ``name`` will be the full module path. | ||
| - The ``subpath`` will represent the package path within a module. | ||
| - The ``version`` will be a valid go version or pseudoversion, or empty. | ||
| - Additional Build information for binaries can be included as ``qualifiers`` (i.e VCS info, go version info, GoArch/GoOS info etc) |
There was a problem hiding this comment.
The additional information should be explicitly defined here.
There was a problem hiding this comment.
exactlty. be specific in the spec, so we all are on the same page.
There was a problem hiding this comment.
PTAL in the new commit (sorry for that).
PURL-TYPES.rst
Outdated
| ``go`` for Go modules: | ||
|
|
||
| - The ``namespace`` field is empty and implies the go mod proxy. | ||
| - The ``name`` will be the full module path. |
There was a problem hiding this comment.
This should probably specify that it is case sensitive. pkg:golang incorrectly states that it is not case sensitive and must be lowercased.
There was a problem hiding this comment.
Exactly. this is what the whole #308 is about.
Please don't repeat the mistakes from the past.
There was a problem hiding this comment.
Should be done now, see the new commit (sorry for that).
|
Thanks for pointing at these! This is essentially a combination of #196 and #308 (with the addition of Say you have a module with the path |
I think the better solution to this problem is that However, "a go module cannot be represented by different PURLs" is not generally the case:
|
jkowalleck
added
Proposed new type
type: golang
I'd disagree. In fact, it is non-breaking, as it adds a completely new purl type. Therefore, no breaking changes are introduced. |
|
It is breaking because no existing PURL software expects |
this is true to every newly proposed PURL Type :-) this PR is trying to add a new type |
|
The problem is that this is not a new type. The |
it is not? Could you point me to the existing
I wonder how you come to this conclusion. |
From the PR description:
|
which is a behavioural change in a downstream application. This is out of scope of this spec, and not in our hands at all - we have no authority there.
exactly this paragraph makes it clear: this is a non-breaking change. Causing no breaking change is the whole point of introducing a new purl type, instead of modifying an exising one. |
i don't think so. #308 makes this clear: the existing spec has flaws that require breaking changes to fix them The only way to fix
|
|
Introducing a new type for an existing type is a breaking change to the PURL ecosystem. Implementations that use If you start writing SBOMs that have Keeping You cannot just fix a PURL type by introducing a new type. Even if PURL libraries are updated to support transparently upgrading the old type into the new type on read, any software that is comparing pre-canonicalized PURL strings will need updates.
What are the flaws that require breaking changes? #308 is about the path being incorrectly converted to lowercase, which is much more easily fixed by just not doing that. |
how? If a tool that produced purls would change it's behaviour by using the new purl-type, where they've used the other one before - this would be a breaking change in that very tool.
So?
A PR tells a story, and the effective patch gets updated along with the discussions on a PR. (PS: I review the content of the PR. and at the time of review, I saw no breaking change.
how comes?
the curerent |
PURL-TYPES.rst
Outdated
| ``go`` for Go modules: | ||
|
|
||
| - The ``namespace`` field is empty and implies the go mod proxy. | ||
| - The ``name`` will be the full module path. |
There was a problem hiding this comment.
| - The ``name`` will be the full module path. | |
| - The ``name`` is the full module path. It MUST be unmodified, and follow the `Go Module Reference <https://go.dev/ref/mod#go-mod-file-ident>`_. |
this change would close #308
There was a problem hiding this comment.
- - The ``name`` will be the full module path. + - The ``name`` is the full module path. In case of an URL: protocol MUST be lowercased; host-part MUST be lowercased; path-part MUSTbe unmodified, as it is case-sensitive.this change would close #308
I don't think this is correct.
- I don't think it's legal to include a protocol in the module path. Go makes some HTTPS requests to resolve a VCS URL to download the package from (usually this is delegated to the proxy).
- The host part is also part of the case sensitive module path. It should not be lowercased. Uppercase characters are currently forbidden by Go for modules. I don't think it's worthwhile or really correct for the PURL spec to be specifying how to convert an invalid module path into a valid module path, I don't think it's worthwhile for the PURL spec to be specifying how to validate Go module paths, this doesn't cover all the restrictions, and this may cause problems if Go ever changes the restrictions for some reason.
There was a problem hiding this comment.
re 1: I see. i was wrong there. Adjusted my suggestion for the protocol.
re 2: the host-part is, per URL-spec case-insensitive, and is normalized to lowercase.
There was a problem hiding this comment.
As far as Go is concerned, it's usually a host-part but it has additional restrictions and it is case sensitive: https://go.dev/ref/mod#go-mod-file-ident
There was a problem hiding this comment.
I see. I will modify my change-suggestion accordingly. does it fit better, now?
PURL-TYPES.rst
Outdated
|
|
||
| - The ``namespace`` field is empty and implies the go mod proxy. | ||
| - The ``name`` will be the full module path. | ||
| - The ``subpath`` will represent the package path within a module. |
There was a problem hiding this comment.
| - The ``subpath`` will represent the package path within a module. | |
| - The ``subpath`` is the unmodified package path within a module. |
PURL-TYPES.rst
Outdated
| - The ``namespace`` field is empty and implies the go mod proxy. | ||
| - The ``name`` will be the full module path. | ||
| - The ``subpath`` will represent the package path within a module. | ||
| - The ``version`` will be a valid go version or pseudoversion, or empty. |
There was a problem hiding this comment.
| - The ``version`` will be a valid go version or pseudoversion, or empty. | |
| - The ``version`` may be a valid go version or pseudoversion, omitted when empty. |
There was a problem hiding this comment.
Should be done now, see the new commit (sorry for that).
|
Adding a new type for a new type is much different than adding a new type for an existing type. An old tool not recognizing a truly new type is expected, but an old tool not recognizing Go PURLs anymore because a tool producing the data says that Changing "MUST be lowercased" to "MUST NOT be lowercased" is a much less impactful change than this. From what I've seen, names with uppercase characters are uncommon, and an outdated implementation that is incorrectly lowercasing is still working correctly for all names that do not contain uppercase characters to lowercase. I would even say that on a larger scale it is not a breaking change because:
In both cases, the PURL is still parsed successfully and the meaning of the PURL is unchanged with respect to the current "MUST be lowercased" spec. The only differences would be that the canonical form changes¹ and a new consumer receiving a PURL from an old producer might be more likely to expect that the ID refers to the correct package, but since there is no good way for an outdated consumer to recover the correct ID after an outdated producer lowercases it, any consumer that relies on getting the correct ID (eg to resolve the package files) is likely already broken and not lowercasing the name can only improve the behavior in that situation. This causes the same alignment problems as introducing a ¹ Due to underspecification in the text and tests, I wouldn't trust incoming PURLs to be in the canonical form as my implementation understands it. There are numerous minor differences in which characters are escaped when (and sometimes how), so if you're accepting PURLs from an external source, even if you don't expect user-entered, non-canonical PURLs in that source, you should be canonicalizing those PURLs yourself if your application depends on them all being canonical for the same definition of canonical. |
|
Go isn't the only ecosystem that has this problem of incorrect name normalization rules in this repo. I'm also aware of:
|
If this is indeed true, then there is something really wrong with PURL: it does not allow for evolution. On the one hand, we cannot add modifications to the existing specification that could introduce breaking changes. On the other hand, we cannot introduce a new type because somehow that is a breaking change as well. So one is pretty much stuck with slight variations of the initial spec. Specs should be allowed to evolve just the way the software does. There should really be a way to add versioning on top of PURL itself. What is being proposed here might in essence be just that for the go spec. |
|
@maceonthompson Thanks for putting this together! this makes a lot sense, and we have an issue with Go alright. Let me look at the comments in details and come back with my 2 cents! |
|
@matt-phylum re:
I am not sure that's hte case, but a new type vs. updating the existing type demands some careful thinking :) |
PURL-TYPES.rst
Outdated
|
|
||
| pkg:go/google.golang.org%2Fgenproto#googleapis/api/annotations | ||
| pkg:go/github.com%2Fjmorion%2Fsqlx@v1.1.2#api | ||
| pkg:go/golang.org%2Fx%2Fvuln?goversion=1.23.2&vcs=git&vcs_modified=true#cmd/govulncheck |
There was a problem hiding this comment.
There is a likely problem with the use of subpath: there is no way to determine where the module ends and the package starts in the general case, is there?
For instance, in the path google.golang.org/genproto/googleapis/api/annotations how can I determine safely that google.golang.org/genproto is a module and that googleapis/api/annotations is a package inside this module? I need either a go proxy lookup or a full filesystem to locate a go.mod/go.sum file, right?
There was a problem hiding this comment.
There is a way, if the module's code is available to you, to determine from a package import where module path ends and the package path begins by making HTTP requests.
I think the use of the subpath here is good because it puts the burden of determining this on whatever generates the PURL, which is likely aware of Go and either has the module paths or is most likely to be able to find the module path from the full package path. Then if you want to use a tool that checks PURLs against a database of information about modules (eg vulnerabilities), the tool already has all the information it needs. Otherwise, either the tool would need to make external API calls to figure out the module path of the PURL or the database would need to have an entry for every package in the module.
There was a problem hiding this comment.
Just to add to @matt-phylum's comment. If a tool is producing a PURL for a Go artifact, then it can use go version, Debug.BuildInfo, or packages.Load to get information about the package and its corresponding module. The encoding proposed here then makes it clear what the modules and packages are.
There was a problem hiding this comment.
I don't believe this is true for the general use case of PURLs. E.g. we do static analysis of binaries and while we can get information about linked packages, there's no indication of which part of the paths correspond to modules
There was a problem hiding this comment.
Right, if you are looking at a Go symbol from the symbol table, you can get its package. You can get the module correctly by prefix-matching it with module information from debug.Buildinfo of the binary, unless there are several modules that are prefixes of the package. My inclination is that it should not affect what is proposed here. (Arguably, there should be a way to get module information for a symbol in the binary, just the way one can do it for the source analysis.)
|
BTW, an elephant in the room is whether the distinction between a namespace and name makes sense not only here, but also in the whole spec, globally. I found myself using a variable with a "namespace/name" substring more often than not.
With this the whole It could have a minimal impact on the spec. |
PURL-TYPES.rst
Outdated
|
|
||
| pkg:go/google.golang.org%2Fgenproto#googleapis/api/annotations | ||
| pkg:go/github.com%2Fjmorion%2Fsqlx@v1.1.2#api | ||
| pkg:go/golang.org%2Fx%2Fvuln?goversion=1.23.2&vcs=git&vcs_modified=true#cmd/govulncheck |
There was a problem hiding this comment.
Is the plan to include all the buildinfo structure as qualifiers?
If so, this would only apply in a built binary?
There was a problem hiding this comment.
good point.
If so, all the qualifiers MUST be documented in the type-spec.
currently it reads:
Additional Build information for binaries can be included as
qualifiers(i.e VCS info, go version info, GoArch/GoOS info etc)
I am afraid this documentation is insufficient.
PURL-TYPES.rst
Outdated
| pkg:go/google.golang.org%2Fgenproto#googleapis/api/annotations | ||
| pkg:go/github.com%2Fjmorion%2Fsqlx@v1.1.2#api | ||
| pkg:go/golang.org%2Fx%2Fvuln?goversion=1.23.2&vcs=git&vcs_modified=true#cmd/govulncheck | ||
| pkg:go/golang.org%2Fx%2Fvuln@v1.1.3?goversion=1.23.2#cmd/govulncheck |
There was a problem hiding this comment.
Are the Go module versions always to be prefixed with a v?
There was a problem hiding this comment.
A version identifies an immutable snapshot of a module, which may be either a release or a pre-release. Each version starts with the letter v, followed by a semantic version.
-- https://go.dev/ref/mod#versions
version could also be a pseudo-version -- a git-tag, a git-commit-hash, or something like this.
There was a problem hiding this comment.
A pseudoversion is a special kind of version that also starts with a v: https://go.dev/doc/modules/version-numbers#pseudo-version-number
I think for Go modules, including when using the Go module system to refer to something that predates modules, the version always starts with a v. In which case, versions that don't start with v would only be used with older tools like Dep?
There was a problem hiding this comment.
If a version exists, it should be a valid Go module version. It should start with a v.
Note that hashes should not be permitted, they are not a valid Go version (resolution of hash commits in go tooling is a convenience feature).
|
@matt-phylum you wrote
Thanks for the links! I tend to think along the same lines, and we can likely salvage the golang type.
I need to pounder this. See my other comment wrt. the namespace/name above in #338 (comment) |
It is fine that PURL spec allows for more flexibility, but there should be only one way the Go module and package information is encoded. This simplifies the work for clients. It is easy to drop qualifiers from a PURL. It is annoying to generate multiple module+package encodings to see if the incoming PURL applies to your code. In general, this proposal tries to make it simple and clear to generate and accurately check against PURLs. It might not be the most user-friendly solution, but tools that render PURLs can easily prettify the output. We believe this is worth the sacrifice. |
|
Could it also be clarified how standard library packages are to be represented? Go has special handling for these, and the 'module' is never explicitly required when using them. But the module does exist for Go uses but the exact module name would make more sense we believe. |
|
@puerco, please read and understand all previously made comments. repeating myself and others: we are here in the domain of specification.
you see, these changes to
This is a misconception - their are no "broken" purls.
You expressed you've read all comments before, maybe read again. |
How could an implementation have been written to avoid the replacement of This proposal, maybe unintentionally, does replace The only way I can think of that implementations would not be broken would be if the input data contained both For example, this usecase from the CycloneDX site will not work until tools and/or databases are updated because the component purl field will change to something meaningless to existing software: https://cyclonedx.org/use-cases/identify-known-vulnerabilities/ The OSV database already uses the corrected |
again?! Okay, lets play this again... nope. any implementation that supports I will not go into the details of the other points, as they all rely on this fundamental misconception. PS: please understand, that all the other proposed solutions, like changing |
|
You keep denying this and calling it a fundamental misconception, but you clearly do not understand what it is @puerco and I are concerned about. It doesn't matter if I know that fixing
This does not make sense to me. I never said anything about not checking whether the PURL is a Go PURL. If the input has |
I think it would be very useful if you could create some scenarios/examples that are representative of the concerns you have. We could then discuss them. I am not trying to diminish your concerns, I simply don't understand them; they are quite abstract to me at this point. It feels like we are running in circles, so I believe working out concrete scenarios might help here. |
|
Here is an example of the way The user is using an SBOM generation tool to analyze their source code and produce an SBOM. They take that SBOM and feed it into a vulnerability scanner so they can be aware of vulnerabilities in their software. The vulnerability scanner, either directly or indirectly, needs to recognize the Go modules being used. In practice this will be something like reading a CycloneDX file, extracting component PURLs, and either converting those PURLs into ecosystem-native identifiers that can be looked up in an advisory database (eg If the user updates the SBOM generation tool to a version that outputs This seems like a fairly likely case to me. SBOM generation can happen in CI close to developers while vulnerability scanning may happen in some software like Dependency-Track hosted by IT or a dedicated security team. Even if the software vendors are keeping up and have added support for |
Thanks for the example, this is really helpful.
I believe I understand the scenario, sorry if I don't. It seems to me that the concern you described here, again, can apply to any new PURL package. If the SBOM tool outputs a
They should get that
I agree that this is a likely case. When a new package type is added, I believe it is a reasonable expectation that the users will check if their scanners support it or the scanners will yell that the PURLs are not supported. Perhaps I am expecting too much. Either way, it looks like this problem, if it exists, exists for both |
Happy to. I think the point missing in all of this is to understand purls are embedded in all sorts of documents and databases across the supply chain. It is not just one tool's concern. A software supply chain tool will ingest multitudes of data from different generators and databases. Some practical examples:In general, component matching becomes a mess as you need to build - for go specifically - a compatibility layer whenever you want to match components, this is done all the time when working with SBOMs, attestations, databases. The fact that the new type does not break the other does not mean that supply chain security tools don't need to ingest both. SBOM Enrichment or Augmentation:One SBOM generator reads component data. Another produces licensing data. You need to merge the output of both. But now you can't. The inputs to both are the same: "A Go module". If one speaks CycloneDXCycloneDX has one field, and one field only, for a purl (which IMHO it's the way it should be). When generating an SBOM you need to choose one schema to use. This means that your SBOM needs to pick one type and hope that everything downstream can handle both. If a tool downstream can't translate, you just lost supply chain data. Asset Management Systems:Asset management systems cataloging components from SBOMs will need to support both. Imagine when the new log4shell hits a go module and you need to find it:
Vulnerability Advisories and VEXBoth advisories and VEX documents use the same input: "A go module". If advisories are published in Deduplication of Component DataSay you merge two SBOMs and want to de-duplicate component data. Today you can just recurse the SBOM data and match, but now you would need to (only for Go) add another compatibility normalization before deduping components. Anyway I could go on and on.. |
|
So we are again where we were 3months ago? (and you claimed you've read all comment?! ) Then read this again: #338 (comment) Tldr: downstream implementations are not our domain. Their poor decisions are not our concern., and there are solutions for that. |
I think the opposite is true. Steering a specification is a great responsibility. This is why, in general, steering members are elected from senior, experienced community members who grasp the consequences of their every move. You are in charge of fostering the adoption of the spec and ensuring decisions are made to keep a healthy community and ecosystem. In other words, you are responsible for poor implementations just as much as for those you may consider good ones. This change will break the data exchange for both. Again, note that this is not a single tool's concern. It's about wrecking an ecosystem already exchanging data. Regardless of what you think of the user's design, the new type will introduce a barrier that will blind tools using one type from the data already produced using the other and in the process, it will make naming go modules unreliable, possibly for years. We've done our part by providing feedback as adopters to fix the current type and not break the whole purl/go ecosystem. Feel free to take it or not. |
|
(i am so fed up trying to understand your points). |
Yes, except that the user wasn't getting advisories for For the software I work on, this would just be annoying. I can safely address it because it's a cloud-hosted solution and I am aware that the change might be happening and I can ensure that
This comment is confusing to me considering the #purl channel is on the OWASP CycloneDX Slack instance. Is it a poor decision that CycloneDX doesn't provide a way for indicate that a Is it a poor decision that Dependency-Track, also an OWASP project, identifies components by a single PURL and has Go-specific behavior that activates for What is a good decision where this problem doesn't occur and how is that communicated to people designing systems that use PURL so this problem does not affect users? This PR is the most active, but there have been other requests to introduce a new package type for representing packages that are already supported by an existing type and any such PR creates the same potential for costly surprises. |
|
I don't see much sense in all this back-and-forth, we are just repeating. You're very much invited to join the fortnightly PURL community meeting to discuss your points: #377 - Otherwise, I don't see a reason to give them any attention. |
|
The concerns raised by @matt-phylum resonate with me. Users not receiving vulnerability notifications is a significant potential issue.
The specification changes will impact the community. While downstream implementations are not the responsibility of the specification, serious consideration should be given to those most at risk from vulnerabilities (e.g., users who aren’t patching regularly and are waiting for alerts). @jkowalleck @zpavlinovic, do you have any thoughts on what a good mitigation strategy would be for transitioning from |
just repeating #338 (comment) lets play this whole evolution, for arbitrary SBOM generators in combination with DependencyTrack(DT), and all its related systems:
You see, none of these steps required breaking changes, none of these steps are illusionary, all of these steps are how feature development in a stack of independent software worked since ages. Most importantly, for users that dont patch anything, nothing will change for them. they still will use the old stack and everything will work like before. P.S.: I noticed in earlier comments that people seem fully aware of how a proper transition from a no-feature state to a fully rolled-out feature would function in their respective fields. Are you concerned that others might not understand this? With all due respect, you're not the only one with insight, so perhaps it's fair to trust that others are equally capable of understanding this. |
|
I'm specifically stating that thought and communication with tool developers should exist to address @matt-phylum concerns in the transition from a E.g. push the messaging that while most tools produce multiple IDs for components it would be even better multiple PURLs to identify components in SBOMs (yes I know that is not always supported in SBOM formats, we're a 100% CDX shop).
Another example is also include a "official" mapping from
Do not confuse my concern in helping improve the user experience and help address a path forward for your proposal with belittling you or end users. What I do see is a lack of addressing the concern brought up by @matt-phylum. Providing contingencies into your PR that address community concerns will help drive agreement. |
|
Regarding communication: this is open-source, you do not know all your downstream users, but all know you. I would not want to suggest/dictate how the transition or adoption is to be made. it should be a natural process driven by the community. They will figure it out. For example, when a new version of our favorite BOM standard is released, I implement them into libraries in a non-breaking fashion, and i implement them into BOM-generators and other tools using feature flags defaulting to of/false.
Sorry, this was not to you, @idunbarh , this was towards all those naysayers. those people that claimed to know how standards work and how this one will definitely break the community for they would not know how to deal with changes and adoption. |
OK, no. They don't. But now I see where you're coming from:
The problem you are not seeing is that not everything works in this simplistic way, where one person generates an SBOM and feeds it to DT (or similar). Things are much more complex than that, SBOMs will be generated and provided by suppliers and you have no control over the software they use generate them, then you often need to use a mix of tools to achieve the document with the data you want. Also, as noted before, purls are in lots of other places beyond SBOMs such as databases, vulnerability scan results and other supply chain technologies and formats such as attestations, advisories, VEX and so on which will take years to move, if they ever do it. So there is no magic feature flag that you can just flick to move everything over to the new schema. |
|
SBOMs will be generated and provided by suppliers.
True, and fully understood. Who said that things should change from one day to the other. The more complex, the more management and time it takes. Been there, done that: I've worked in domains where I've accompanied change processes that took around 5 years to be fully effective.
Oh, there is a "featue flag" for almost everything. Lets sketch one for you: This situation would not be different, if the breaking changes you suggested were implemented. Anyway, the new |
|
I doubt anyone is going to write into a contract that SBOMs use It'd be much easier in the case where you know you need one form or the other to apply a transform that rewrites the PURLs to all be the required form. |
I agree. Also, ensuring a sound technical proposal should not rely on legal assumptions. "Supplier" in the document exchange can mean any third party producing SBOMs, not necessarily as part of a contractual obligation. This is especially true for open source where there are no obligations whatsoever. |
I understand, but I also do think that the point missing right now is that the system is fundamentally broken.
This is already a huge mess with the current PURL. It allows for different modules to have the same PURL and for the same module to have multiple different PURLs.
Again, the "translation" already needs to exist due to the mess the current Go PURL is creating. Correct matching is already a mess (theoretically impossible).
I understand, but I still maintain the position that this will also happen when a PURL for a completely new, say, language is introduced.
See my previous comments on matching. |
I am not sure I understand the second part of the sentence. How would they continue not getting advisories? |
|
If their software doesn't already support |
Here is how I see the situation. There are really three options going forward.
I think option 1 is the worst one. I personally don't mind option 2 (if the definition of I would have to think more about the strategy, but here are a few thoughts.
|
Hm, it could be the case it took time for PURL specification to land for an existing software. Out of curiosity, how long did it take for Rust to get supported? I could see this still being a problem, but I also do see how this might be a bigger problem with a new type for an existing language. |
The current PURL specification for Go was created before Go 1.11 modules and thus has namespace inconsistencies and lacks semantic versioning.
Although in many cases a module path corresponds directly to the URL of the hosting repository, that is not always true. The URL formed from the module path may be an endpoint that serves a redirect to the true host. This indirection protects projects that for whatever reason must change their hosting provider: their module names will continue to work. Consequently, it is undesirable to encode any aspect of the underlying hosting system as part of the PURL.
In essence, all Go modules form a single namespace. Since it is used by the majority of Go programmers, we propose to represent this namespace by the empty string. Though not included in this commit, other namespaces could be possible and would represent package managers and/or build tools that are alternatives to the go command.
The
gotype proposed here fixes the current issues by removing the namespace, using valid Go module versions (including pseudoversions), and adds some extra functionality to encode optional information about specific builds (GOOS, GOARCH, etc).If accepted, all tools maintained by the Go project (such as govulncheck and pkg.go.dev) that surface PURLs will use this new type to provide canonical PURLs for Go modules and packages