Add `devcontainers` ecosystem #8445

joshspicer · 2023-11-21T19:32:15Z

This change provides initial support for a devcontainers ecosystem, leaning heavily on the devcontainers/cli for ecosystem-specific implementation.

This updater will look in the provided --directory for a devcontainer.json (or .devcontainer.json if directory === /) and update all dev container Features that are pinned to an out-of-date major version. The lockfile will also be updated if one is already present in the target directory.

I aim to add future updater granularity/modes in the future.

Requires at least dev container CLI v0.54.0:

Demos (with the `dependabot/cli`)

LOCAL_GITHUB_ACCESS_TOKEN="$LOCAL_GITHUB_ACCESS_TOKEN"  dependabot  \
     --updater-image ghcr.io/dependabot/dependabot-updater-devcontainers  \
      update \
      devcontainers \
      joshspicer/versioning-project-two-configs \
      -o /tmp/updates.yaml \
      --directory /.devcontainer

updater | +-----------------------------------------------------------------------+
updater | |                  Changes to Dependabot Pull Requests                  |
updater | +---------+-------------------------------------------------------------+
updater | | created | ghcr.io/codspace/versioning/foo:1 ( from 1.1.0 to 2.11.1 )  |
updater | | created | ghcr.io/codspace/versioning/baz:1.0 ( from 1.0.0 to 2.0.0 ) |
updater | +---------+-------------------------------------------------------------+

LOCAL_GITHUB_ACCESS_TOKEN="$LOCAL_GITHUB_ACCESS_TOKEN"  dependabot  \
     --updater-image ghcr.io/dependabot/dependabot-updater-devcontainers  \
      update \
      devcontainers \
      joshspicer/versioning-project-two-configs \
      -o /tmp/updates.yaml \
      --directory /

updater | +----------------------------------------------------------------------+
updater | |                 Changes to Dependabot Pull Requests                  |
updater | +---------+------------------------------------------------------------+
updater | | created | ghcr.io/codspace/versioning/foo:1 ( from 1.1.0 to 2.11.1 ) |
updater | +---------+------------------------------------------------------------+

Closes #7000.

updater/Gemfile.lock

deivid-rodriguez

Made a first review pass, seems pretty neat!

devcontainers/lib/dependabot/devcontainers/utils.rb

devcontainers/spec/devcontainers/devcontainers/file_parser_spec.rb

devcontainers/spec/devcontainers/devcontainers_spec.rb

devcontainers/lib/dependabot/devcontainers/metadata_finder.rb

devcontainers/lib/dependabot/devcontainers/file_updater/config_updater.rb

devcontainers/lib/dependabot/devcontainers/file_parser/feature_dependency_parser.rb

devcontainers/spec/devcontainers/devcontainers/file_parser_spec.rb

deivid-rodriguez · 2023-12-21T17:38:02Z

I rebased the PR, addressed several issues, and implemented the new way of handling updates. Now we provide updates to all fetched files, relative to the configured directory (by default "/").

For example:

$ bin/dry-run.rb devcontainers joshspicer/versioning-project-two-configs 
=> cloning into /home/dependabot/tmp/joshspicer/versioning-project-two-configs
=> parsing dependency files
Running command: devcontainer outdated --workspace-folder . --config .devcontainer.json --output-format json
Running command: devcontainer outdated --workspace-folder . --config devcontainer.json --output-format json
=> updating 3 dependencies: ghcr.io/codspace/versioning/foo:1, ghcr.io/codspace/versioning/bar:1, ghcr.io/codspace/versioning/baz:1.0

=== ghcr.io/codspace/versioning/foo:1 (1.1.0)
 => checking for updates 1/3
 => latest available version is 2.11.1
 => latest allowed version is 2.11.1
 => requirements to unlock: own
 => requirements update strategy: 
Running command: devcontainer upgrade
Running command: devcontainer upgrade
 => bump ghcr.io/codspace/versioning/foo:1 from 1.1.0 to 2.11.1

    ± .devcontainer.json
    ~~~
    --- /tmp/original20231221-316-tawni8	2023-12-21 17:35:09.209309011 +0000
    +++ /tmp/updated20231221-316-8gbvq6	2023-12-21 17:35:09.209309011 +0000
    @@ -1,7 +1,7 @@
     {
         "image": "mcr.microsoft.com/devcontainers/typescript-node:18",
         "features": {
    -        "ghcr.io/codspace/versioning/foo:1": {},
    +        "ghcr.io/codspace/versioning/foo:2": {},
             "ghcr.io/codspace/versioning/bar:1": {}
         }
     }
    ~~~
    2 insertions (+), 2 deletions (-)

    ± .devcontainer/devcontainer.json
    ~~~
    --- /tmp/original20231221-316-6o3e3w	2023-12-21 17:35:09.212309011 +0000
    +++ /tmp/updated20231221-316-j9vt98	2023-12-21 17:35:09.212309011 +0000
    @@ -1,7 +1,7 @@
     {
         "image": "mcr.microsoft.com/devcontainers/typescript-node:18",
         "features": {
    -        "ghcr.io/codspace/versioning/foo:1": {},
    +        "ghcr.io/codspace/versioning/foo:2": {},
             "ghcr.io/codspace/versioning/bar:1": {},
             "ghcr.io/codspace/versioning/baz:1.0": {}
         }
    ~~~
    2 insertions (+), 2 deletions (-)

    ± .devcontainer/devcontainer-lock.json
    ~~~
    --- /tmp/original20231221-316-xio06s	2023-12-21 17:35:09.214309011 +0000
    +++ /tmp/updated20231221-316-ugjs8o	2023-12-21 17:35:09.214309011 +0000
    @@ -10,10 +10,10 @@
           "resolved": "ghcr.io/codspace/versioning/baz@sha256:37f36051adf6da0a43764b9669b945e0f06e11973e02c57ad261b03bb1057cb7",
           "integrity": "sha256:37f36051adf6da0a43764b9669b945e0f06e11973e02c57ad261b03bb1057cb7"
         },
    -    "ghcr.io/codspace/versioning/foo:1": {
    -      "version": "1.1.0",
    -      "resolved": "ghcr.io/codspace/versioning/foo@sha256:80d2d7b58afeaf907451c6f4e24de47b09a327a24a21a2d3323b7abf76d14be5",
    -      "integrity": "sha256:80d2d7b58afeaf907451c6f4e24de47b09a327a24a21a2d3323b7abf76d14be5"
    +    "ghcr.io/codspace/versioning/foo:2": {
    +      "version": "2.11.1",
    +      "resolved": "ghcr.io/codspace/versioning/foo@sha256:e98cdc5066cff85c5076dfec32058d53e5b9bbc75b125d84adcdf295674c14ee",
    +      "integrity": "sha256:e98cdc5066cff85c5076dfec32058d53e5b9bbc75b125d84adcdf295674c14ee"
         }
       }
     }
    \ No newline at end of file
    ~~~
    5 insertions (+), 5 deletions (-)

=== ghcr.io/codspace/versioning/bar:1 (1.0.0)
 => checking for updates 2/3
 => latest available version is 1.0.0
    (no update needed as it's already up-to-date)

=== ghcr.io/codspace/versioning/baz:1.0 (1.0.0)
 => checking for updates 3/3
 => latest available version is 2.0.0
 => latest allowed version is 2.0.0
 => requirements to unlock: own
 => requirements update strategy: 
Running command: devcontainer upgrade
 => bump ghcr.io/codspace/versioning/baz:1.0 from 1.0.0 to 2.0.0

    ± .devcontainer/devcontainer.json
    ~~~
    --- /tmp/original20231221-316-44zv9n	2023-12-21 17:35:14.339309013 +0000
    +++ /tmp/updated20231221-316-e1khaj	2023-12-21 17:35:14.340309013 +0000
    @@ -3,6 +3,6 @@
         "features": {
             "ghcr.io/codspace/versioning/foo:1": {},
             "ghcr.io/codspace/versioning/bar:1": {},
    -        "ghcr.io/codspace/versioning/baz:1.0": {}
    +        "ghcr.io/codspace/versioning/baz:2": {}
         }
     }
    ~~~
    2 insertions (+), 2 deletions (-)

    ± .devcontainer/devcontainer-lock.json
    ~~~
    --- /tmp/original20231221-316-fk8v4b	2023-12-21 17:35:14.343309013 +0000
    +++ /tmp/updated20231221-316-f439qx	2023-12-21 17:35:14.343309013 +0000
    @@ -5,10 +5,10 @@
           "resolved": "ghcr.io/codspace/versioning/bar@sha256:0eb80a7a45ea6ac6d2057798608be4cacb3d3667d4818118e17acc5037d687d4",
           "integrity": "sha256:0eb80a7a45ea6ac6d2057798608be4cacb3d3667d4818118e17acc5037d687d4"
         },
    -    "ghcr.io/codspace/versioning/baz:1.0": {
    -      "version": "1.0.0",
    -      "resolved": "ghcr.io/codspace/versioning/baz@sha256:37f36051adf6da0a43764b9669b945e0f06e11973e02c57ad261b03bb1057cb7",
    -      "integrity": "sha256:37f36051adf6da0a43764b9669b945e0f06e11973e02c57ad261b03bb1057cb7"
    +    "ghcr.io/codspace/versioning/baz:2": {
    +      "version": "2.0.0",
    +      "resolved": "ghcr.io/codspace/versioning/baz@sha256:3420e9d222352d5bee4d7f2c99dd7492295f5518041650ade9e8ecd0d6ce49d8",
    +      "integrity": "sha256:3420e9d222352d5bee4d7f2c99dd7492295f5518041650ade9e8ecd0d6ce49d8"
         },
         "ghcr.io/codspace/versioning/foo:1": {
           "version": "1.1.0",
    ~~~
    5 insertions (+), 5 deletions (-)
🌍 Total requests made: '0'

I saw a few more issues that I'll be addressing next week:

Don't parse the requirement as part of the dependency name: it should be ghcr.io/codspace/versioning/foo not ghcr.io/codspace/versioning/foo:1.
Respect style of existing requirements for now: ghcr.io/codspace/versioning/baz:1.0 should be updated to ghcr.io/codspace/versioning/baz:2.0, not to ghcr.io/codspace/versioning/baz:2.

Also found an issue in the CLI that I reported at devcontainers/cli#712.

deivid-rodriguez · 2023-12-22T14:35:47Z

I fixed all the issues mentioned previously and got CI to pass.

However, I was thinking about the smart approach taken here of asking devcontainers outdated directly in the FileParser step about the latest version and "bypassing" the UpdateChecker step and I believe it's going to come short for some common Dependabot features. For example:

Users need to be able to skip certain versions from configuration. See https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#ignore.
Similarly (same doc as above), users may choose to ignore certain update types for a dependency, for example, patch and minor versions.

So I think we still need to implement a basic UpdateChecker that can access all available versions of the feature and do some filtering over them.

We'll need to work on this after holidays!

deivid-rodriguez · 2024-01-03T10:24:16Z

Alternatively, and probably better since it would only require changes to the devcontainers CLI, we could add an --ignore-range flag to the devcontainers outdated CLI command to support the first use case, and --ignore-major, --ignore-minor, and --ignore-patch flags to support the latter.

deivid-rodriguez · 2024-01-11T14:39:16Z

Found and fixed a few more issues, including one that I reported upstream (and added workaround for it here).

Also cleaned up git history to keep things a bit more tidy.

This is ready for a review!

joshspicer · 2024-01-17T20:50:31Z

✅ Did some bug bashing on the changes and I see no release blockers!

Some follow up fixes that I can explore moving forward, but I think given the new timeline this is good to go!

Features that are pinned to an outdated SHA will get updated to the major version tag, and there's a bit of a strange message generated. Pinning to a SHA is not widely advertised and i'd argue it's still helpful to be aware of new Feature releases, even if pinned.
- This could be rolled into adding some switches (like David's already suggested 🎉 ) to provide more granular control over when the updater is run
Features that are inaccessible or are pinned with invalid tags cause a crash.
- This implies an invalid devcontainer.json, or a devcontainer.json that requires some auth to query private container registries. Following up to support private CRs is known so that's OK
- Repro with ghcr.io/devcontainers/features/docker-in-docker:1.1 (ie: a version that has never been tagged!)

Error: ERR: Feature 'ghcr.io/devcontainers/features/docker-in-docker:1.1' could not be processed.  You may not have permission to access this Feature, or may not be logged in.  If the issue persists, report this to the Feature author.
    at q7 (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:285:24143)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AC (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:285:26711)
    at async Su (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:285:26937)
    at async vu (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:390:1818)
    at async meA (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:613:4322)
    at async /usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:613:3357
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:169:in `bind_call'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:169:in `validate_call_skip_block_type'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:111:in `block in create_validator_slow_skip_block_type'
        from /home/dependabot/devcontainers/lib/dependabot/devcontainers/file_updater/config_updater.rb:68:in `run_devcontainer_upgrade'
        from /home/dependabot/devcontainers/lib/dependabot/devcontainers/file_updater/config_updater.rb:54:in `update_manifest'
        from /home/dependabot/devcontainers/lib/dependabot/devcontainers/file_updater/config_updater.rb:24:in `block (2 levels) in update'
        from /home/dependabot/common/lib/dependabot/shared_helpers.rb:264:in `with_git_configured'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:272:in `bind_call'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:272:in `validate_call'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:193:in `block in create_validator_slow'
        from /home/dependabot/devcontainers/lib/dependabot/devcontainers/file_updater/config_updater.rb:23:in `block in update'
        from /home/dependabot/common/lib/dependabot/shared_helpers.rb:57:in `block in in_a_temporary_repo_directory'
        from /home/dependabot/common/lib/dependabot/shared_helpers.rb:57:in `chdir'
        from /home/dependabot/common/lib/dependabot/shared_helpers.rb:57:in `in_a_temporary_repo_directory'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:272:in `bind_call'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:272:in `validate_call'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:193:in `block in create_validator_slow'
        from /home/dependabot/devcontainers/lib/dependabot/devcontainers/file_updater/config_updater.rb:22:in `update'
        from /home/dependabot/devcontainers/lib/dependabot/devcontainers/file_updater.rb:78:in `update'
        from /home/dependabot/devcontainers/lib/dependabot/devcontainers/file_updater.rb:25:in `block in updated_dependency_files'
        from /home/dependabot/devcontainers/lib/dependabot/devcontainers/file_updater.rb:21:in `each'
        from /home/dependabot/devcontainers/lib/dependabot/devcontainers/file_updater.rb:21:in `updated_dependency_files'
        from bin/dry-run.rb:706:in `block in <main>'
        from bin/dry-run.rb:614:in `each'
        from bin/dry-run.rb:614:in `<main>'

deivid-rodriguez · 2024-01-18T16:37:29Z

@joshspicer I looked at the issues you found:

For SHA256 versions, I'm ignoring them for now, so we don't even try to update them. If someone is using those, I guess it's their preference and they have some motivation, so better for Dependabot to not be annoying to those people. By the way, I'm doing the same for deprecated features because the CLI doesn't handle them very well at the moment. I looked for a "/" in the feature name to detect non deprecated features, is that correct?
As per the second issue, after I added support for ignores, it's now working "fine": no error, and it's updating the invalid tag to the latest version. I think that's acceptable.

I also fixed some other issues I found and added support for ignores!

deivid-rodriguez · 2024-01-22T18:12:33Z

I merged the first commit in this PR as #8858, so that I can build updater images for this PR (since workflow files are picked up from the main branch).

Co-authored-by: Josh Spicer <josh@joshspicer.com>

This class at the moment just delegates to the FileUpdater, because of the heavy lifting is done by the FileParser class, since the `devcontainers outdated` command provides both current dependency and available updates information. Co-authored-by: Josh Spicer <josh@joshspicer.com>

Co-authored-by: Josh Spicer <josh@joshspicer.com>

For example, ``` bin/dry-run.rb devcontainers devcontainers/images --dir /src/go ``` would update manifests with the wrong content because it was taking the contents of the manifests in root as the updated content.

deivid-rodriguez · 2024-01-22T19:19:20Z

Let's do this!

joshspicer requested a review from a team as a code owner November 21, 2023 19:32

joshspicer force-pushed the joshspicer/devcontainers branch from 730d14f to e64e20f Compare November 21, 2023 21:23

deivid-rodriguez reviewed Nov 22, 2023

View reviewed changes

updater/Gemfile.lock Outdated Show resolved Hide resolved

joshspicer mentioned this pull request Nov 27, 2023

add devcontainers smoke test dependabot/smoke-tests#149

Closed

joshspicer force-pushed the joshspicer/devcontainers branch 2 times, most recently from f010370 to 2bcda52 Compare November 28, 2023 15:34

deivid-rodriguez reviewed Nov 28, 2023

View reviewed changes

deivid-rodriguez reviewed Dec 6, 2023

View reviewed changes

devcontainers/spec/devcontainers/devcontainers/file_parser_spec.rb Outdated Show resolved Hide resolved

joshspicer force-pushed the joshspicer/devcontainers branch from 37a83de to b1d6383 Compare December 6, 2023 18:40

github-actions bot added L: javascript and removed L: javascript labels Dec 6, 2023

deivid-rodriguez force-pushed the joshspicer/devcontainers branch from 3f77e3e to 35f9c51 Compare December 21, 2023 17:29

github-actions bot added L: terraform Terraform packages L: javascript labels Dec 21, 2023

deivid-rodriguez force-pushed the joshspicer/devcontainers branch from 35f9c51 to fb46440 Compare December 21, 2023 17:30

deivid-rodriguez force-pushed the joshspicer/devcontainers branch 3 times, most recently from ed2a6ed to c422b65 Compare December 22, 2023 12:00

deivid-rodriguez force-pushed the joshspicer/devcontainers branch 2 times, most recently from ba1c00d to 9550e68 Compare January 11, 2024 14:36

deivid-rodriguez force-pushed the joshspicer/devcontainers branch 2 times, most recently from 9698a46 to c3a6297 Compare January 11, 2024 14:53

deivid-rodriguez removed L: terraform Terraform packages L: javascript labels Jan 11, 2024

deivid-rodriguez force-pushed the joshspicer/devcontainers branch from c3a6297 to 2e37aa0 Compare January 11, 2024 14:56

deivid-rodriguez added the L: devcontainers label Jan 11, 2024

deivid-rodriguez force-pushed the joshspicer/devcontainers branch 4 times, most recently from 5c4d352 to 479b521 Compare January 17, 2024 17:28

deivid-rodriguez mentioned this pull request Jan 18, 2024

devcontainer outdated misses current version information for features pinned to minor version devcontainers/cli#724

Closed

deivid-rodriguez force-pushed the joshspicer/devcontainers branch 3 times, most recently from fd70cb9 to fe6c46a Compare January 18, 2024 16:30

abdulapopoola approved these changes Jan 19, 2024

View reviewed changes

jakecoffman approved these changes Jan 19, 2024

View reviewed changes

deivid-rodriguez force-pushed the joshspicer/devcontainers branch from 10a97b2 to 5f04f7b Compare January 22, 2024 18:11

deivid-rodriguez and others added 10 commits January 22, 2024 19:23

Implement FileFetcher for devcontainers

ae79b9e

Co-authored-by: Josh Spicer <josh@joshspicer.com>

Implemente FileParser for devcontainers

fefece7

Co-authored-by: Josh Spicer <josh@joshspicer.com>

Implement FileUpdater for devcontainers

81e2210

Co-authored-by: Josh Spicer <josh@joshspicer.com>

Skip SHA pinned versions for now

3dd36da

Support ignores

c98db9d

Fix updater misbehaving when a custom directory is configured

7f838f6

For example, ``` bin/dry-run.rb devcontainers devcontainers/images --dir /src/go ``` would update manifests with the wrong content because it was taking the contents of the manifests in root as the updated content.

We don't need to pass metadata for now

4abd402

Skip deprecated features for now

21f36e6

Cover file updater not updating past target version

ae771dc

deivid-rodriguez force-pushed the joshspicer/devcontainers branch from 5f04f7b to ae771dc Compare January 22, 2024 18:29

deivid-rodriguez merged commit 529a800 into dependabot:main Jan 22, 2024
34 checks passed

JamieMagee mentioned this pull request Jan 22, 2024

Add devcontainers as a supported Dependabot ecosystem SchemaStore/schemastore#3543

Merged

joshspicer mentioned this pull request Jan 24, 2024

Add smoke test against Dependabot integration devcontainers/cli#732

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `devcontainers` ecosystem #8445

Add `devcontainers` ecosystem #8445

joshspicer commented Nov 21, 2023 •

edited by deivid-rodriguez

Loading

deivid-rodriguez left a comment

deivid-rodriguez commented Dec 21, 2023

deivid-rodriguez commented Dec 22, 2023

deivid-rodriguez commented Jan 3, 2024

deivid-rodriguez commented Jan 11, 2024

joshspicer commented Jan 17, 2024

deivid-rodriguez commented Jan 18, 2024

deivid-rodriguez commented Jan 22, 2024

deivid-rodriguez commented Jan 22, 2024

Add devcontainers ecosystem #8445

Add devcontainers ecosystem #8445

Conversation

joshspicer commented Nov 21, 2023 • edited by deivid-rodriguez Loading

Demos (with the dependabot/cli)

deivid-rodriguez left a comment

Choose a reason for hiding this comment

deivid-rodriguez commented Dec 21, 2023

deivid-rodriguez commented Dec 22, 2023

deivid-rodriguez commented Jan 3, 2024

deivid-rodriguez commented Jan 11, 2024

joshspicer commented Jan 17, 2024

deivid-rodriguez commented Jan 18, 2024

deivid-rodriguez commented Jan 22, 2024

deivid-rodriguez commented Jan 22, 2024

Add `devcontainers` ecosystem #8445

Add `devcontainers` ecosystem #8445

joshspicer commented Nov 21, 2023 •

edited by deivid-rodriguez

Loading

Demos (with the `dependabot/cli`)