feat: support docker image history scanning

[jamestelfer]

Description

Note

This PR has been created to help illustrate a possible fix for #2881. I'd be grateful for any feedback, and if this is useful, I'll follow the guidance to ready it for merge.

Enables the docker scanner to send image history records to the scanner. These records are generally the commands that were run to build each image layer. The entries look similar to:

    {
      "created": "2024-05-24T05:26:37.56186972Z",
      "created_by": "RUN |8 PNPM_VERSION=v8.15.5 GITHUB_REGISTRY_TOKEN=ghp_ohmygodnessaleakedvalue /bin/sh -c unwise-command '${GITHUB_REGISTRY_TOKEN}' ",
      "comment": "buildkit.dockerfile.v0"
    },

In this example, the secret is leaked from a build argument, but it could equally be from a command line literal.

Fixes #2881

Questions

1. I'm currently using image://config-file/history/index as the filename for each history entry (see diff). Is this sufficient for identification? It doesn't look particularly flash, but I'm not sure what would be better.
2. The Layer in the Docker metadata is entered as the empty string. If this isn't OK, I'd welcome suggestions on fixing it.
3. I added a new metric docker_history_entries_scanned for this, assuming this is the right thing to do.

Checklist

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[CLAassistant]

All committers have signed the CLA.

[ahrav]

Thanks for the addition @jamestelfer this looks great to me.

[ahrav]

Description

Note

This PR has been created to help illustrate a possible fix for #2881. I'd be grateful for any feedback, and if this is useful, I'll follow the guidance to ready it for merge.

Enables the docker scanner to send image history records to the scanner. These records are generally the commands that were run to build each image layer. The entries look similar to:

    {
      "created": "2024-05-24T05:26:37.56186972Z",
      "created_by": "RUN |8 PNPM_VERSION=v8.15.5 GITHUB_REGISTRY_TOKEN=ghp_ohmygodnessaleakedvalue /bin/sh -c unwise-command '${GITHUB_REGISTRY_TOKEN}' ",
      "comment": "buildkit.dockerfile.v0"
    },

In this example, the secret is leaked from a build argument, but it could equally be from a command line literal.

Fixes #2881

Questions

1. I'm currently using image://config-file/history/index as the filename for each history entry (see diff). Is this sufficient for identification? It doesn't look particularly flash, but I'm not sure what would be better.
2. The Layer in the Docker metadata is entered as the empty string. If this isn't OK, I'd welcome suggestions on fixing it.
3. I added a new metric docker_history_entries_scanned for this, assuming this is the right thing to do.

Checklist

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

I am not too well versed in our Docker source, but from what I can tell this looks pretty good to me. That being said, @zricethezav @joeleonjr would you mind giving this a look as well. Thanks.

[rgmz]

The Layer in the Docker metadata is entered as the empty string. If this isn't OK, I'd welcome suggestions on fixing it.

It's possible to associate history with the layer ID by index (skipping empty layers). (On that note, the current TruffleHog implementation is incorrect/not ideal and uses the digest instead of the layer ID.)

For instance, if you look at the postgres:latest you can see:

1. #(nop) ADD file:5a is layer 254fddc33a7883e66303d20feb68795952cb91cb0af7574e129f6bfd8efad544
2. RUN /bin/sh -c set -eux; \tgroupadd -r postgres --gid=999; is layer d3b78d99bcc3eee03f029078417d7d1076170980d4073719fdb6d8c187e003df.

History

{
"history": [
        {
            "created": "2024-05-09T18:58:11Z",
            "created_by": "/bin/sh -c #(nop) ADD file:5aaace706aa00ff97d243daa2c29f5de88f124e1b97c570634f16eef90783286 in / "
        },
        {
            "created": "2024-05-09T18:58:11Z",
            "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
            "empty_layer": true
        },
        {
            "created": "2024-05-09T18:58:11Z",
            "created_by": "RUN /bin/sh -c set -eux; \tgroupadd -r postgres --gid=999; \tuseradd -r -g postgres --uid=999 --home-dir=/var/lib/postgresql --shell=/bin/bash postgres; \tmkdir -p /var/lib/postgresql; \tchown -R postgres:postgres /var/lib/postgresql # buildkit",
            "comment": "buildkit.dockerfile.v0"
        },
        ...
}

Layers

{
        "Layers": [
            "254fddc33a7883e66303d20feb68795952cb91cb0af7574e129f6bfd8efad544/layer.tar",
            "d3b78d99bcc3eee03f029078417d7d1076170980d4073719fdb6d8c187e003df/layer.tar",
            "cfb5c12ea5e8675b8fbc75bf74b539422cd99ae65e6f1b3603638a2e6152e384/layer.tar",
            "7bd588371cf03e6283e8729a4a8e28c3e02864d6865eddcdd1c9a7c8718996be/layer.tar",
            ...
       ]
}

This can be validated by either docker pull postgres && docker save postgres > layers.tar or using a GUI like Dive.

[jamestelfer]

The association with layers like this is a great point.

I can annotate an entry with its corresponding layer without issue, as long as it's ok that some records will not have an associated layer.

Not all entries will have a layer, but any may have a secret.

Given this, some records will have an empty string for the layer, and I presume that the proposed naming scheme is OK for a start?

[jamestelfer]

Thanks for all your feedback @ahrav and @rgmz! I think I've addressed it, looking forward to a re-review.

Note that I changed the entry reference format to image-metadata:history:%d:created-by which may be more self-explanatory than image://config-file/history/%d.

[ahrav]

LGTM! Thanks again for adding this change.

[ahrav]

Thanks for adding this comment.