(pipelines): additionalInputs are not available in ShellStep or CodeBuildStep

[vasyl-nvt]

Describe the bug

I am using pipelines.CodePipeline with an existing AWS CodePipeline (aws-codepipeline.Pipeline).

Source artefacts are provided to the synth/ShellStep, or synth/CodeBuildStep via additionalSources.
It looks like the input cannot be mounted.

Expected Behavior

"additionalInputs" sources should be mounted correctly and accessible.

Current Behavior

The task fails with an error.

.....

[Container] 2023/06/29 06:34:56 CODEBUILD_SRC_DIR=/codebuild/output/src1689/src/s3/00
[Container] 2023/06/29 06:34:56 CODEBUILD_SRC_DIR_Artifact_BuildApplicationComponents_BuildFrontendApp=/codebuild/output/src1689/src/s3/01
[Container] 2023/06/29 06:34:56 YAML location is /codebuild/readonly/buildspec.yml
[Container] 2023/06/29 06:34:56 Selecting shell bash as specified in buildspec.
[Container] 2023/06/29 06:34:56 Setting HTTP client timeout to higher timeout for S3 source
[Container] 2023/06/29 06:34:56 Processing environment variables
[Container] 2023/06/29 06:34:56 Selecting 'nodejs' runtime version '16' based on manual selections...
[Container] 2023/06/29 06:34:56 Running command echo "Installing Node.js version 16 ..."
Installing Node.js version 16 ...
 
[Container] 2023/06/29 06:34:56 Running command n $NODE_16_VERSION
copying : node/16.20.0
installed : v16.20.0 (with npm 8.19.4)
 
[Container] 2023/06/29 06:35:22 Moving to directory /codebuild/output/src1689/src/s3/00
[Container] 2023/06/29 06:35:22 Configuring ssm agent with target id: codebuild:68909ae2-d973-4487-bd56-445e56689099
[Container] 2023/06/29 06:35:22 Successfully updated ssm agent configuration
[Container] 2023/06/29 06:35:22 Registering with agent
[Container] 2023/06/29 06:35:22 Phases found in YAML: 2
[Container] 2023/06/29 06:35:22 BUILD: 3 commands
[Container] 2023/06/29 06:35:22 INSTALL: 2 commands
[Container] 2023/06/29 06:35:22 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2023/06/29 06:35:22 Phase context status code: Message:
[Container] 2023/06/29 06:35:22 Entering phase INSTALL
[Container] 2023/06/29 06:35:22 Running command [ ! -d "../frontend" ] || { echo 'additionalInputs: "../frontend" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.'; exit 1; } && ln -s -- "$CODEBUILD_SRC_DIR_undefined" "../frontend"
ln: failed to create symbolic link '../frontend' -> '': No such file or directory
 
[Container] 2023/06/29 06:35:22 Command did not exit successfully [ ! -d "../frontend" ] || { echo 'additionalInputs: "../frontend" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.'; exit 1; } && ln -s -- "$CODEBUILD_SRC_DIR_undefined" "../frontend" exit status 1
[Container] 2023/06/29 06:35:22 Phase complete: INSTALL State: FAILED
[Container] 2023/06/29 06:35:22 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: [ ! -d "../frontend" ] || { echo 'additionalInputs: "../frontend" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.'; exit 1; } && ln -s -- "$CODEBUILD_SRC_DIR_undefined" "../frontend". Reason: exit status 1

Reproduction Steps

The code example uses CodeBuildStep. But the same behaviour happens when using ShellStep.

cdkAppSourceArtifact and frontendDeployArtifact contain correct content during execution.

const frontendDeployArtifact = new Artifact()
const cdkAppSourceArtifact = new Artifact()
....
  const basePipeline = new Pipeline(this, 'BasePipeline', {
      pipelineName: this.stackName,
    })
...
  // basePipeline stages that produce input artefacts.
...

 const pipeline = new cdk.pipelines.CodePipeline(this, 'CodePipeline', {
      codePipeline: basePipeline,
      selfMutation: false,
      synth: new CodeBuildStep('Synth', {
        partialBuildSpec: BuildSpec.fromObject({
          version: '0.2',
          env: {
            shell: 'bash'
          },
          phases: {
            install: {
              'runtime-versions': {
                nodejs: '16'
              }
            }
          }
        }),
        input: CodePipelineFileSet.fromArtifact(cdkAppSourceArtifact),
        additionalInputs: {
          '../frontend': CodePipelineFileSet.fromArtifact(frontendDeployArtifact)
        },
        commands: [
          'echo "No commands yet. additionalInputs not available"',
          'ls -la ../'
        ],
        primaryOutputDirectory: 'cdk/cdk.out'
      })

Possible Solution

It seems additionalInputs are incorrectly referenced in the script.
See highlighted CODEBUILD_SRC_DIR variables in the provided log.

The input is defined in CODEBUILD_SRC_DIR_Artifact_BuildApplicationComponents_BuildFrontendApp but referenced via $CODEBUILD_SRC_DIR_undefined*

Additional Information/Context

No response

CDK CLI Version

2.85

Framework Version

2.85

Node.js Version

18

OS

macOS Ventura

Language

Typescript

Language Version

Typescript (4.9.5)

Other information

No response

[pahud]

Unfortunately I can't reproduce this from your provided reproduction code. Are you able to provide a working sample that I can simply copy/past to my IDE to verify it? Thank you.

[vasyl-nvt]

Hi @pahud ,

Moved on with another solution but tried to reproduce this one today and it seems to work correctly now. Might be some local issue in the region that day.
No more $CODEBUILD_SRC_DIR_undefined in the log output and the additionalSources are mapped correctly.

Closing the issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[dennisvang]

@pahud Perhaps you could answer a related question:

Could you tell us what is the proper way to get the absolute path of an "additional input" from within a shell script running in a CodeBuildStep with additionalInputs?

For example, if we have the following (from above example):

...
additionalInputs: {
  '../frontend': CodePipelineFileSet.fromArtifact(frontendDeployArtifact)
},
...

then how do we get the absolute path for '../frontend' in a script?

Based on the OP, I suppose there will be an environment variable like CODEBUILD_SRC_DIR_Artifact_BuildApplicationComponents_BuildFrontendApp, is that correct?

If so, how is the name of this variable defined, and where is this documented?

I guess it would be CODEBUILD_SRC_DIR_<something>?

Thanks for your help.