sam build cache does not seem to work with CodeBuild

[mateogianolio]

Description:

Cache works locally on my computer (MacOS 11.3), but never in CodeBuild (aws/codebuild/amazonlinux2-x86_64-standard:3.0).

I have verified that the CodeBuild cache path is exactly the same for two subsequent builds, but sam build still says the cache is invalid.

I think it might have something to do with paths or symlinks, but not sure.

Sorry if this is the wrong place to report this.

Steps to reproduce:

CodeBuild set to custom local cache (also tried with s3 cache, still does not work).

buildspec.yml (modified for brevity)

version: 0.2

cache:
  paths:
    - '.aws-sam/**/*' # Cache `.aws-sam`
phases:
  build:
    commands:
      - ./build.sh <stack_name>

build.sh (modified for brevity)

stack_name=$1
region=eu-west-1

sam build \
  --region "$region" \
  --template-file "$stack_name/template.yaml" \
  --build-dir ".aws-sam/$stack_name/build" \
  --cache-dir ".aws-sam/$stack_name/cache" \
  --use-container \
  --cached \
  --parallel;

Observed result:

CodeBuild mounts the cache path correctly (<id> is the cache id that I verified stays the same for subsequent builds):

[Container] Expanded cache path .aws-sam/**/*
[Container] MkdirAll: /codebuild/local-cache/custom/<id>/.aws-sam
[Container] Symlinking: /codebuild/output/srcXXXXXXXXX/src/github.com/<repo>/.aws-sam => /codebuild/local-cache/custom/<id>/.aws-sam

Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(nodejs14.x, /codebuild/output/srcXXXXXXXXX/src/github.com/<repo>/<stack_name>/<codeuri>, Zip, , <uuid>, {}, {}, []), Function: Function(name='<redacted>', functionname='<redacted>', runtime='nodejs14.x', memory=None, timeout=5, handler='index.handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/codebuild/output/srcXXXXXXXXX/src/github.com/<repo>/<stack_name>/<codeuri>', environment=None, rolearn='<redacted>', layers=[], events=None, metadata=None, inlinecode=None, codesign_config_arn=None, stack_path=''))

...

Cache is invalid, running build and copying resources to function build definition of <uuid>

Expected result:

Same function build definition found, adding function (Previous: BuildDefinition(nodejs14.x, /codebuild/output/srcXXXXXXXXX/src/github.com/<repo>/<stack_name>/<codeuri>, Zip, <id>, <uuid>, {}, {}, []), Current: BuildDefinition(nodejs14.x, /codebuild/output/srcXXXXXXXXX/src/github.com/<repo>/<stack_name>/<codeuri>, Zip, , <uuid>, {}, {}, []), Function: Function(name='<redacted>', functionname='<redacted>', runtime='nodejs14.x', memory=None, timeout=5, handler='index.handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/codebuild/output/srcXXXXXXXXX/src/github.com/<repo>/<stack_name>/<codeuri>', environment=None, rolearn='<redacted>', layers=[], events=None, metadata=None, inlinecode=None, codesign_config_arn=None, stack_path=''))

...

Valid cache found, copying previously built resources from function build definition of <uuid>

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: aws/codebuild/amazonlinux2-x86_64-standard:3.0
2. sam --version: 1.22.0
3. AWS region: does not matter

[aahung]

Possible cause: SAM CLI cannot recognize .aws-sam if it is a symbolic link

[mateogianolio]

Update: On MacOS it works fine with symbolic links:

$ mkdir tmp
$ mv .aws-sam tmp
$ ln -s tmp/.aws-sam .aws-sam
$ sam build # same build command as above
...
Same function build definition found, adding function...

[mateogianolio]

After looking around I'm inclined to think this is a CodeBuild issue. People are having all kinds of trouble getting CodeBuild caching to work:

• https://forums.aws.amazon.com/thread.jspa?threadID=312569&tstart=0
• Local cache of node_modules doesn't work with yarn workspaces aws-samples/aws-codebuild-samples#8
• https://stackoverflow.com/questions/66231501/why-aws-codebuild-local-custom-cache-is-not-caching-file
• https://stackoverflow.com/questions/58793704/aws-codebuild-local-cache-failing-to-actually-cache

[aahung]

the code directory (absolute path) is also part of what we use to check cache. Can you check whether /codebuild/output/srcXXXXXXXXX/src/github.com/<repo>/<stack_name>/<codeuri> is consistent in different code build runs?

[Edit] I think build cache should not use absolute path as part of ID, working on a fix

[mateogianolio]

You have a very good point, it looks like the srcXXXXXXXXX part of the path always differs between builds even if the cache path stays the same!

[mufumade]

I am facing the same issue and I found a stack overflow post discussing a similar issue.
Is the workaround mentioned in the stackoverflow post also suitable for .aws-sam? If not is there any workaround for now ?

[mateogianolio]

Did you find a workaround @mufumade? Seems like a fix is not coming any time soon.

[mufumade]

@mateogianolio

1. you have the .aws-sam folder or what ever it is called. In there you have a folder that Is used to store all build artifacts. But .aws-sam is not where your builds happen. This happens in a temp folder under /usr/something. So you need to find a place where you build your stuff that not gets blown away every time you build something and copy the outputs to .aws-sam.

In the following vid he shows exactly that problem.

The relevant part starts at minute 53.
Deep dive into aws sam

He builds a custom swift lambda runtime but he nicely shows how to handle the folder deletion of sam build. So you can apply that to whatever you are building.

2. If your project can handle incremental builds AND can cope with relative paths and changing file structure( by that I mean changing file timestamps, chancing inodes etc) than you probably can get this to work.

Inodes get changed because of the changing container you get from codebuild every time you build something.

I even tried to mount a efs to codebuild ( which worked but it is way to expensive because you need a nat gateway) and did all the building on that efs to counter the changing inodes and relative paths but no luck ( at least for me with swift build )

[mateogianolio]

Thank you for the detailed explanation! I will try that and see if it works for my use case.

[TheFlexican]

Hi,

I think I know what's happening here. Only adding the cache folder like below is not enough to validate the cache.

cache:
paths:
- '.aws-sam/**/*'

You should also include the build.toml file, which holds the hashes and what not to validate the cache,

cache:
paths:
- .aws-sam/cache/**/*
- .aws-sam/build.toml

Try it out and let me know.

You shouldn't have to do any weird stuff to make sam caching work with codebuild cache.
This is my build command:

sam build --template sam.yaml --base-dir ./ --cached --parallel

[mateogianolio]

Interesting.. will try it out when I have time! Hopefully next week.

[danieldaugherty]

@TheFlexican Are you talking about changing the sam source code or is there something in my project I can add that configuration to in order to see these improvements?

[TheFlexican]

@TheFlexican Are you talking about changing the sam source code or is there something in my project I can add that configuration to in order to see these improvements?

You need to put it in your buildspec.yml. Here is an example.

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.9
    commands:
      - echo nothing to install
  build:
    commands:
      - sam build --template sam.yaml --cached --parallel
      - sam package ...
  post_build:
    commands:
      - echo Build completed on `date`
artifacts:
  type: zip
  base-directory: ./
  files:
    - path/**/*
cache:
  paths:
    - .aws-sam/cache/**/*
    - .aws-sam/build.toml

[mateogianolio]

All right, sorry for the long delay.

According to the buildspec specification (https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html), the path I provided (.aws-sam/**/*) "represents all files recursively starting from a subdirectory named [.aws-sam]" so specifying the path to build.toml should not make any difference.

The problem is instead that without adding --base-dir to the sam build command, the build paths become absolute in build.toml and the cache in CodeBuild fails.

Adding --base-dir "$stack_name" to my sam build command finally solved the problem!!

Thanks for the help 🎉