Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

manager:gomod does not expose indirect dependencies in dryRun: extract #19440

Closed
jamietanna opened this issue Dec 16, 2022 · 6 comments · Fixed by #19442
Closed

manager:gomod does not expose indirect dependencies in dryRun: extract #19440

jamietanna opened this issue Dec 16, 2022 · 6 comments · Fixed by #19442
Assignees
@jamietanna
Labels
manager:gomod Go Modules status:requirements Full requirements are not yet known, so implementation should not be started type:bug Bug fix of existing functionality

Comments

@jamietanna
Copy link
Contributor

jamietanna commented Dec 16, 2022 •
edited
Loading

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

34.58.1

If you're self-hosting Renovate, select which platform you are using.

github.com

If you're self-hosting Renovate, tell us what version of the platform you run.

GitHub.com

Was this something which used to work for you, and then stopped?

I never saw this working

Describe the bug

Although recently added with unit test coverage as part of #19416, when running Renovate, we do not receive the indirect dependencies in the extraction result.

These dependencies do not appear when using dryRun: "extract", or through tools like https://gitlab.com/tanna.dev/renovate-graph.

Reproducing this can be done via:

renovate --token $GITHUB_TOKEN jamietanna/oapi-codegen --dry-run

With the following config:

config.js 
module.exports = {
  requireConfig: "ignored",
  onboarding: false,
  includeForks: true,

  dryRun: "extract",

  // adding this does not work, either
  /*
  "packageRules": [
    {
      "matchManagers": ["gomod"],
      "matchDepTypes": ["indirect"],
      "enabled": true
    }
  ],
  */
}

This leads to the following dependency extraction:

         "gomod": [
           {
             "packageFile": "go.mod",
             "deps": [
               {
                 "managerData": {"lineNumber": 3, "multiLine": true},
                 "depName": "github.com/apapsch/go-jsonmerge/v2",
                 "depType": "require",
                 "currentValue": "v2.0.0",
                 "datasource": "go",
                 "depIndex": 0
               },
               {
                 "managerData": {"lineNumber": 4, "multiLine": true},
                 "depName": "github.com/getkin/kin-openapi",
                 "depType": "require",
                 "currentValue": "v0.107.0",
                 "datasource": "go",
                 "depIndex": 1
               },
               {
                 "managerData": {"lineNumber": 5, "multiLine": true},
                 "depName": "github.com/gin-gonic/gin",
                 "depType": "require",
                 "currentValue": "v1.8.1",
                 "datasource": "go",
                 "depIndex": 2
               },
               {
                 "managerData": {"lineNumber": 6, "multiLine": true},
                 "depName": "github.com/go-chi/chi/v5",
                 "depType": "require",
                 "currentValue": "v5.0.7",
                 "datasource": "go",
                 "depIndex": 3
               },
               {
                 "managerData": {"lineNumber": 7, "multiLine": true},
                 "depName": "github.com/golangci/lint-1",
                 "depType": "require",
                 "currentValue": "v0.0.0-20181222135242-d2cdd8c08219",
                 "datasource": "go",
                 "currentDigest": "d2cdd8c08219",
                 "digestOneAndOnly": true,
                 "depIndex": 4
               },
               {
                 "managerData": {"lineNumber": 8, "multiLine": true},
                 "depName": "github.com/google/uuid",
                 "depType": "require",
                 "currentValue": "v1.3.0",
                 "datasource": "go",
                 "depIndex": 5
               },
               {
                 "managerData": {"lineNumber": 9, "multiLine": true},
                 "depName": "github.com/gorilla/mux",
                 "depType": "require",
                 "currentValue": "v1.8.0",
                 "datasource": "go",
                 "depIndex": 6
               },
               {
                 "managerData": {"lineNumber": 10, "multiLine": true},
                 "depName": "github.com/labstack/echo/v4",
                 "depType": "require",
                 "currentValue": "v4.9.1",
                 "datasource": "go",
                 "depIndex": 7
               },
               {
                 "managerData": {"lineNumber": 11, "multiLine": true},
                 "depName": "github.com/lestrrat-go/jwx",
                 "depType": "require",
                 "currentValue": "v1.2.25",
                 "datasource": "go",
                 "depIndex": 8
               },
               {
                 "managerData": {"lineNumber": 12, "multiLine": true},
                 "depName": "github.com/matryer/moq",
                 "depType": "require",
                 "currentValue": "v0.2.7",
                 "datasource": "go",
                 "depIndex": 9
               },
               {
                 "managerData": {"lineNumber": 13, "multiLine": true},
                 "depName": "github.com/stretchr/testify",
                 "depType": "require",
                 "currentValue": "v1.8.1",
                 "datasource": "go",
                 "depIndex": 10
               },
               {
                 "managerData": {"lineNumber": 14, "multiLine": true},
                 "depName": "golang.org/x/text",
                 "depType": "require",
                 "currentValue": "v0.4.0",
                 "datasource": "go",
                 "depIndex": 11
               },
               {
                 "managerData": {"lineNumber": 15, "multiLine": true},
                 "depName": "golang.org/x/tools",
                 "depType": "require",
                 "currentValue": "v0.3.0",
                 "datasource": "go",
                 "depIndex": 12
               },
               {
                 "managerData": {"lineNumber": 16, "multiLine": true},
                 "depName": "gopkg.in/yaml.v2",
                 "depType": "require",
                 "currentValue": "v2.4.0",
                 "datasource": "go",
                 "depIndex": 13
               },
               {
                 "managerData": {"lineNumber": 63},
                 "depName": "go",
                 "depType": "golang",
                 "currentValue": "1.18",
                 "datasource": "golang-version",
                 "versioning": "npm",
                 "rangeStrategy": "replace",
                 "depIndex": 14
               }
             ]
           }
         ]
       }

Relevant debug logs

TRACE logs 
TRACE: gomod.extractPackageFile() (repository=jamietanna/oapi-codegen)
       "content": "[content]"
TRACE: Matched multi-line require on line 2 (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/apapsch/go-jsonmerge/v2 v2.0.0" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/apapsch/go-jsonmerge/v2 v2.0.0" (repository=jamietanna/oapi-codegen)
       "lineNumber": 3
TRACE: require: "	github.com/getkin/kin-openapi v0.107.0" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/getkin/kin-openapi v0.107.0" (repository=jamietanna/oapi-codegen)
       "lineNumber": 4
TRACE: require: "	github.com/gin-gonic/gin v1.8.1" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/gin-gonic/gin v1.8.1" (repository=jamietanna/oapi-codegen)
       "lineNumber": 5
TRACE: require: "	github.com/go-chi/chi/v5 v5.0.7" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/go-chi/chi/v5 v5.0.7" (repository=jamietanna/oapi-codegen)
       "lineNumber": 6
TRACE: require: "	github.com/golangci/lint-1 v0.0.0-20181222135242-d2cdd8c08219" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/golangci/lint-1 v0.0.0-20181222135242-d2cdd8c08219" (repository=jamietanna/oapi-codegen)
       "lineNumber": 7
TRACE: require: "	github.com/google/uuid v1.3.0" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/google/uuid v1.3.0" (repository=jamietanna/oapi-codegen)
       "lineNumber": 8
TRACE: require: "	github.com/gorilla/mux v1.8.0" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/gorilla/mux v1.8.0" (repository=jamietanna/oapi-codegen)
       "lineNumber": 9
TRACE: require: "	github.com/labstack/echo/v4 v4.9.1" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/labstack/echo/v4 v4.9.1" (repository=jamietanna/oapi-codegen)
       "lineNumber": 10
TRACE: require: "	github.com/lestrrat-go/jwx v1.2.25" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/lestrrat-go/jwx v1.2.25" (repository=jamietanna/oapi-codegen)
       "lineNumber": 11
TRACE: require: "	github.com/matryer/moq v0.2.7" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/matryer/moq v0.2.7" (repository=jamietanna/oapi-codegen)
       "lineNumber": 12
TRACE: require: "	github.com/stretchr/testify v1.8.1" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	github.com/stretchr/testify v1.8.1" (repository=jamietanna/oapi-codegen)
       "lineNumber": 13
TRACE: require: "	golang.org/x/text v0.4.0" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	golang.org/x/text v0.4.0" (repository=jamietanna/oapi-codegen)
       "lineNumber": 14
TRACE: require: "	golang.org/x/tools v0.3.0" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	golang.org/x/tools v0.3.0" (repository=jamietanna/oapi-codegen)
       "lineNumber": 15
TRACE: require: "	gopkg.in/yaml.v2 v2.4.0" (repository=jamietanna/oapi-codegen)
TRACE: require line: "	gopkg.in/yaml.v2 v2.4.0" (repository=jamietanna/oapi-codegen)
       "lineNumber": 16
TRACE: require: ")" (repository=jamietanna/oapi-codegen)
TRACE: Matched multi-line require on line 19 (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/davecgh/go-spew v1.1.1 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/davecgh/go-spew v1.1.1 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/gin-contrib/sse v0.1.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/gin-contrib/sse v0.1.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/go-openapi/jsonpointer v0.19.5 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/go-openapi/jsonpointer v0.19.5 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/go-openapi/swag v0.21.1 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/go-openapi/swag v0.21.1 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/go-playground/locales v0.14.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/go-playground/locales v0.14.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/go-playground/universal-translator v0.18.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/go-playground/universal-translator v0.18.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/go-playground/validator/v10 v10.11.1 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/go-playground/validator/v10 v10.11.1 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/goccy/go-json v0.9.11 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/goccy/go-json v0.9.11 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/golang-jwt/jwt v3.2.2+incompatible // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/golang-jwt/jwt v3.2.2+incompatible // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/invopop/yaml v0.1.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/invopop/yaml v0.1.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/josharian/intern v1.0.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/josharian/intern v1.0.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/json-iterator/go v1.1.12 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/json-iterator/go v1.1.12 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/labstack/gommon v0.4.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/labstack/gommon v0.4.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/leodido/go-urn v1.2.1 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/leodido/go-urn v1.2.1 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/lestrrat-go/backoff/v2 v2.0.8 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/lestrrat-go/backoff/v2 v2.0.8 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/lestrrat-go/blackmagic v1.0.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/lestrrat-go/blackmagic v1.0.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/lestrrat-go/httpcc v1.0.1 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/lestrrat-go/httpcc v1.0.1 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/lestrrat-go/iter v1.0.1 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/lestrrat-go/iter v1.0.1 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/lestrrat-go/option v1.0.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/lestrrat-go/option v1.0.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/mailru/easyjson v0.7.7 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/mailru/easyjson v0.7.7 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/mattn/go-colorable v0.1.13 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/mattn/go-colorable v0.1.13 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/mattn/go-isatty v0.0.16 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/mattn/go-isatty v0.0.16 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/modern-go/reflect2 v1.0.2 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/modern-go/reflect2 v1.0.2 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/pelletier/go-toml/v2 v2.0.5 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/pelletier/go-toml/v2 v2.0.5 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/pkg/errors v0.9.1 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/pkg/errors v0.9.1 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/pmezard/go-difflib v1.0.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/pmezard/go-difflib v1.0.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/stretchr/objx v0.5.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/stretchr/objx v0.5.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/ugorji/go/codec v1.2.7 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/ugorji/go/codec v1.2.7 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/valyala/bytebufferpool v1.0.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/valyala/bytebufferpool v1.0.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	github.com/valyala/fasttemplate v1.2.2 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	github.com/valyala/fasttemplate v1.2.2 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	golang.org/x/crypto v0.1.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	golang.org/x/crypto v0.1.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	golang.org/x/mod v0.7.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	golang.org/x/mod v0.7.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	golang.org/x/net v0.2.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	golang.org/x/net v0.2.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	golang.org/x/sys v0.2.0 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	golang.org/x/sys v0.2.0 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	golang.org/x/time v0.0.0-20220411224347-583f2d630306 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	golang.org/x/time v0.0.0-20220411224347-583f2d630306 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	google.golang.org/protobuf v1.28.1 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	google.golang.org/protobuf v1.28.1 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: "	gopkg.in/yaml.v3 v3.0.1 // indirect" (repository=jamietanna/oapi-codegen)
TRACE: No multi-line match: 	gopkg.in/yaml.v3 v3.0.1 // indirect (repository=jamietanna/oapi-codegen)
TRACE: require: ")" (repository=jamietanna/oapi-codegen)

Have you created a minimal reproduction repository?

I have linked to a minimal reproduction repository in the bug description
@jamietanna jamietanna added priority-5-triage status:requirements Full requirements are not yet known, so implementation should not be started type:bug Bug fix of existing functionality labels Dec 16, 2022
@jamietanna
Copy link
Contributor Author

It looks like this is down to our text fixtures, I'll look to amend and see if I can resolve this

@jamietanna
Copy link
Contributor Author

renovate/lib/modules/manager/gomod/extract.ts

Line 132 in d19effc

if (multiMatch && !line.endsWith('// indirect')) {
needs changing 👀

@viceice
Copy link
Member

viceice commented Dec 16, 2022

next time you should do real tests before 😉

@jamietanna
Copy link
Contributor Author

That's very fair 😅 I'd thought it'd be quicker to get it released and then try it than try it locally. Next time I won't do the same!

@jamietanna jamietanna mentioned this issue Dec 16, 2022
Merged
6 tasks
jamietanna pushed a commit to jamietanna/renovate that referenced this issue Dec 16, 2022
fix(manager/gomod): resolve multi-line indirect dependencies
a757de9 
As noticed in renovatebot#19440, the handling for indirect dependencies didn't
quite work when using a multi-line `require` block.

Closes renovatebot#19440.
jamietanna pushed a commit to jamietanna/renovate that referenced this issue Dec 16, 2022
fix(manager/gomod): resolve multi-line indirect dependencies
a17dbcd 
As noticed in renovatebot#19440, the handling for indirect dependencies didn't
quite work when using a multi-line `require` block.

Closes renovatebot#19440.
jamietanna pushed a commit to jamietanna/renovate that referenced this issue Dec 16, 2022
fix(manager/gomod): resolve multi-line indirect dependencies
6ff7b71 
As noticed in renovatebot#19440, the handling for indirect dependencies didn't
quite work when using a multi-line `require` block.

Closes renovatebot#19440.
@jamietanna
Copy link
Contributor Author

Awesome, verified that #19442 works by running

npm run build
# then cd to the same config.js
node ../../renovatebot/renovate/dist/renovate.js --token $GITHUB_TOKEN jamietanna/oapi-codegen

And it now displays the indirect dependencies 👏

jamietanna pushed a commit to jamietanna/renovate that referenced this issue Dec 16, 2022
fix(manager/gomod): resolve multi-line indirect dependencies
c85c28e 
As noticed in renovatebot#19440, the handling for indirect dependencies didn't
quite work when using a multi-line `require` block.

As we're now no longer hitting the `No multi-line match` logging
statement when we're receiving an `indirect` dependency, we need to add
a test to exercise that for our coverage requirements, too.

Closes renovatebot#19440.
@renovate-release
Copy link
Collaborator

🎉 This issue has been resolved in version 34.59.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 16, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@jamietanna jamietanna

Labels
manager:gomod Go Modules status:requirements Full requirements are not yet known, so implementation should not be started type:bug Bug fix of existing functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(manager/gomod): resolve multi-line indirect dependencies jamietanna/renovate
3 participants
@viceice @jamietanna @renovate-release