Scope build meta-data keys by config file

[toolmantim]

This updates the meta-data key used for storing pre-built images to include the docker-compose config file name. This allows you to pre-build and run the same named service from two completely different docker-compose.yml files, without any fear of clashing.

For example:

steps:
  - name: ":docker: dev"
    plugins:
      docker-compose#x:
        build: app
        image-repository: org/app-dev

  - label: ":docker: staging"
    plugins:
      docker-compose#x:
        config: docker-compose.staging.yml
        build: app
        image-repository: org/app-staging

  - wait

  - name: ":hammer: dev"
    command: yarn test
    plugins:
      docker-compose#x:
        run: app

  - name: ":hammer: staging"
    command: yarn test-staging
    plugins:
      docker-compose#x:
        config: docker-compose.staging.yml
        run: app

The above two will now set and get the following meta-data keys:

docker-compose-plugin-built-image-tag-app
docker-compose-plugin-built-image-tag-app-docker-compose.staging.yml

If you specify multiple configs with the array syntax they're joined with a dash. For example, the following:

steps:
  - name: ":docker: dev"
    plugins:
      docker-compose#x:
        config:
          - docker-compose.one.yml
          - docker-compose.two.yml
        build: app
        image-repository: org/app-dev

Sets the following meta-data key:

docker-compose-plugin-built-image-tag-app-docker-compose.one.yml-docker-compose.two.yml

I don't think this should have any backwards compatibility problems, unless people were depending on the meta-data key, but as that's not configurable I'd consider it private.

• Tests
• Code

Fixes #111

[lox]

Gnarly problem!

[lox]

This seems pretty reasonable to me!

[lox]

I don't think this should have any backwards compatibility problems, unless people were depending on the meta-data key, but as that's not configurable I'd consider it private.

Unfortunately, whilst this is compatible with the same version, if you mix and match accidentally (which is quite easy to do), it's API incompatible with previous versions.

Running plugin github.com/buildkite-plugins/docker-compose-buildkite-plugin command hook | 0s
-- | --
  | # Executing "/tmp/buildkite-agent-bootstrap-hook-runner-856866600"
  | $ /bin/bash -c /tmp/buildkite-agent-bootstrap-hook-runner-856866600
  | $ buildkite-agent meta-data get docker-compose-plugin-built-image-tag-xxx-api-.buildkite/compose.yml
  | 2018-03-30 09:56:44 FATAL  Failed to get meta-data: POST https://agent.buildkite.com/v3/jobs/xxx/data/get: 404 No key "docker-compose-plugin-built-image-tag-ev..." found
  |   | Failed to get metdata docker-compose-plugin-built-image-tag-myapp-api-.buildkite/compose.yml (exit 1)

IMO we need to make this at least work ok with the 1.8.x series of metadata keys, and then probably bump the version to v2.0.0 to release this.

[toolmantim]

Oh, interesting! I hadn’t really considered that.

Yeah, we should bump to 2.0

Also, we should change the output message as well… that meta-data key message is obtuse, and could be much friendlier.

[toolmantim]

We haven’t cut a new version yet, I was hestitant to. Glad I didn’t!

Did this crop up for someone who was accidentally using master? (Which is perhaps another problem. 99% of the time you want it pinned, and all steps pinned to the same version).

/me ponders a GitHub bot that sends PRs to lint and update your BK pipeline plugin version deps.

[toolmantim]

Any reason we need the backward compat? Why not cut v2.0.0 and in the release notes mention it?

[jufemaiz]

Uhoh. I think v2.0.0 broke our builds :(

[lox]

More context required @jufemaiz ;)

[jufemaiz]

@lox confirmation on #112 (comment)

Release notes didn't comment on breaking change and that's the exact message (within reason) I saw by bumping to v2.0.0. v1.8.4 works fine.

We are having to do intermediary build injecting agent held SSH creds via build arguments in so that we can access private git repo so we don't do a docker-compose plugin based build but obviously need to tag the metadata for future uses by docker-compose plugin.

buildkite-agent meta-data set docker-compose-plugin-built-image-tag-app $AWS_ACCOUNT.dkr.ecr.$MASTER_AWS_REGION.amazonaws.com/$REPOSITORY:$DEPLOY_VERSION

Using v2.0.0 plugin threw the following:

# Executing "/tmp/buildkite-agent-bootstrap-hook-runner-#########"
--
  | $ /bin/bash -c /tmp/buildkite-agent-bootstrap-hook-runner-#########
  | $ buildkite-agent meta-data get docker-compose-plugin-built-image-tag-app-docker-compose.test.yml
  | 2018-04-19 00:11:02 FATAL  Failed to get meta-data: POST https://agent.buildkite.com/v3/jobs/#{UUID}/data/get: 404 No key "docker-compose-plugin-built-image-tag-ap..." found
  |  

Which then attempted to do a standard docker-compose build (but no build arguments so no dice).

Would be good to know what the key is there to attempt to further debug.

[lox]

Sorry about that @jufemaiz! We're trying to avoid having the metadata key schemes being a public API, but we should have been clearer about that.

Might multi-stage docker builds be a good solution for you? https://medium.com/@kevinsimper/how-we-do-docker-multi-stage-builds-and-secure-sharing-of-private-repositories-956eda84f3b8

Alternately, we could prioritize build args to the docker-compose plugin.

[jufemaiz]

No worries @lox. Would you like me to open as a separate issue?

[lox]

That would be great if you could @jufemaiz!