New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/tools/go/packages: no way to easily distinguish the test version of the package from the real one #27910
Comments
There are a couple other alternatives I hadn't considered:
At any rate, if there is some specific alternative that is expected to be reliable enough for all users of |
I'm running into this right now as well -- I'm looking at transitioning the deadcode tool to work with Go modules (#24661), and encountered this when transitioning to I believe that adding a field like Without this, right now I am left to depend on parsing output of the This is based on the output that I currently get from using
|
@bcmills with regards to your 2 other suggestions in your follow-on comment:
|
cc @ianthehat because he had a hat It's not totally obvious where to draw the line of test vs non-test package. By my count, there's three categories of packages that can be considered tests:
It's difficult to distinguish between the second group and the third group (at least without doing failure prone package name comparisons). I think it would be confusing and misleading to mark all three as "tests", and I also think that the ForTest field isn't quite what people want. |
Adding to @bcmills' description above, a concrete instance of where this is a problem is a
The call to For a You can also see from the
|
PkgPath is supposed to be the path of the package in a build. In each build, there will only be one package with a given package path, and that's the path of the package given by the reflect package, etc. It's extracted from .ImportPath because that's the only way to get it from go list, and we didn't want to add a mostly redundant field. I don't think adding a field that's only populated in the go list world is a good idea. It means that tools that rely on it won't work outside that world. I think we need a clear definition of what we're classifying as "test" and "non-test" that can apply for all build systems. |
I think we retreated from documenting this claim because it is not always true: main packages do not have unique PkgPaths, for one. There may be other examples. |
To add to @adonovan's comment, the concrete example of where this breaks down is in the output above:
If that's the goal of
Alternatively would could identify the one package (in a directory in
You could either add a separate Regardless of whether we add |
I think maybe I miscommunicated? By "in a build" I meant that each binary or library with transitive dependencies can contain at most one package with a given path. And go/packages and go list can return multiple package with a given path, but they won't both appear in a given binary or test, or any (library + transitive dependency) set. Am I wrong about that? There can only be one main in each transitive dependency set, right? And |
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing.
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
@matloob and I followed up offline on this, so sharing those thoughts/findings back here (please correct any mistakes, @matloob) For some context
In the context of testing, In the Blaze world there can be multiple test variants, and these are defined by rules. But like the The Returning the conversation to the The question still remains therefore, how to distinguish the test variant from the package itself? We discussed two options, having first identified the (at most) two packages in the the slice returned by
The first of these points builds on @bcmills's suggestions in #27910 (comment). This satisfies our requirement. The second is interesting. Taking the specific example of Whilst the first of these solutions seems to work in the The second solution requires that |
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
We need to fight go/packages a bit to find the right package to use for type information (see golang/go#27910 for more details). Also, our logic for surfacing embedded fields where the embedded type had a valid go/types.Type was incorrect. It assumed that all such types must be external to the package being generated, and hence required that all fields be exported. However, it is possible for a package-local type to fully type check and therefore be valid. In this case, non-exported fields _are_ valid for surfacing. Also a small fix in cmd/gg whereby xtest packages were incorrectly being considered as not Done() because they were missing a hash (which they will never have). Added a failsafe that errors in case any work was left un-Done
[...years pass...]
Even with the go command, there may be numerous "intermediate test variants". Consider a package P, and an external test package P_test that has significant extra dependencies. (A hypothetical example would be P=net/url if P_test were to depend upon a complete web server, net/http.) When building the test executable for P, every intermediate package that is "above" P but "below" P_test must be compiled in a variant mode in which any import of P is replaced by "P [P.test]", that is, the test variant of P, which may contain extra files. (FWIW, these intermediate test variants have the potential to greatly increase the size of the overall graph; gopls goes to some lengths to ignore them in most cases.) This subtlety is a recurring source of confusion that we should clarify in both "go help list" and go/packages, however we resolve this issue. |
When
Config.Test
is set totrue
,packages.Load
may return multiplePackage
instances with the samePkgPath
.go list
encodes the information about which package is which redundantly: both in theForTest
field and also in the reportedImportPath
for test variants.go/packages
strips out theImportPath
annotations, but also fails to propagate theForTest
field. As a result, the only way to distinguish the two packages is by checking which one'sGoFiles
are a superset of the other's. That works most of the time, but I suspect it will fail in some cases (for example, whenb
importsa
anda_test
importsb
, the test version ofb
fora_test
is compile with exactly the same sources as the non-test copy ofb
).CC: @matloob @alandonovan @ianthehat
The text was updated successfully, but these errors were encountered: