sam build --cached always claims cache is invalid (python3.8 environment & runtime)

[ahurlburt]

Description:

sam build --cached always claims cache is invalid for me, I've never witnessed it indicate it is valid and sam build takes an extremely long time for me (i have ~20 serverless functions defined in the template)

Steps to reproduce:

sam build --cached --parallel --use-container --skip-pull-image
Starting Build use cache
Starting Build inside a container

gives:

Cache is invalid, running build and copying resources to function build definition of d5f003bb...

Observed result:

Output indicates cache is invalid. I've never seen the cache being used. I have several serverless functions defined in my template.yml and all are using python3.8 runtime.

sam build takes an extremely long time for me, it seems it pulls all dependencies each time and rebuild everything.

Am I misunderstanding how the cache is intended to work?

Expected result:

Cache is valid and will be used

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Mac
Running the cli in a pipenv environment

[mingkun2020]

Cache will always invalid for the first time running sam build-cached in a sam project since there is nothing to be cached before. If you make some small change in 1 function and keep other ~19 functions unchanged(in your example) and run sam build-cached again, you will see other 19 function will directly copy from the cache and only 1 function will be build again, it will be much more faster then the first build.

[ahurlburt]

@mingkun2020 I'm running sam build --cached multiple times but it continues to say Cache is invalid

"you will see other 19 function will directly copy from the cache and only 1 function will be build again, it will be much more faster then the first build"

This is not what I seem to be experiencing... is there a way I can provide more information which may be helpful? I just see the "cache is invalid" every time I run...

[mingkun2020]

@ahurlburt I see. Could you provide which sam cli version are you using? I couldn't reproduce this issue with the latest sam cli version. Could you try it with the latest sam cli version(If you are not using the latest one)? Also could you try to run sam init to generate a hello world project then test it with sam build cache to see if cache works with the generated project?

[sriram-mv]

@ahurlburt could you go into a bit more detail on how you use sam cli? are you changing any source code between builds?

[ahurlburt]

@sriram-mv @mingkun2020 thanks for the replies. I had a chance to retest this using latest sam and a clean hello world project. I was able to reproduce the issue, it's possible my understanding of how the cache is supposed to work is invalid. Here's what I'm seeing.

The command I'm using:
sam build --cached --parallel --use-container --skip-pull-image

The first time I run this it says cache invalid, which is expected since it's first run.

if I don't change anything and run the command again it says cache is valid, as expected.

Now, I added another file (app_new.py) again under hello_world project and added a new function to the sam template.

So my template now looks like this:

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.8
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get
  SecondFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app_new.handler
      Runtime: python3.8
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello2
            Method: get

If I then run the build command again it says cache invalid. I'm not sure what to expect here. I would think the cache would be valid for the HelloWorldFunction since nothing changed there. Since I updated the template I run the command again with no changes and it says "cache valid".

Now I ran a final test where I added a print statement to the app_new.handler code for the second function.

I would expect that the cache would be valid for the HelloWorldFunction since nothing changed however, when running the build again I get 'Cache Invalid' and it seems to rebuild everything. Is my understand how the cache is supposed to work incorrect or am I seeing unexpected behaviour?

An additional note why this is important to me: In my actual project I have 20 or so actual functions so I change something in one or more frequently and running sam build takes a very long time. So I was hoping the cache would allow only those functions which hadn't changed to be rebuilt. I also noticed sam has some options for building only a single function, however when I did this I was getting very strange behaviour (at least when running sam local start-api so I have to rebuild everything each time and as a result local dev is much slower than I believe it should be.

Thanks again.

[ahurlburt]

@mingkun2020 @sriram-mv sorry for bump but thought this may have got lost. I answered your questions above, is my understanding of how the cache is supposed to work flawed?

[ahurlburt]

Can anyone help with this? It's really hindering dev speed to have these super long builds.

I can confirm that if I do not change any code and just touch say the template file the "valid cache" message appears and it uses the cache however, I would think this is not the only valid use of the cache?

[ahurlburt]

bump. Still seeing this EVERY BUILD.

[mlem8]

I am also seeing this same behavior

update: I realize this might be due to the fact that I am using a custom Makefile with my build, as the release notes for https://github.com/aws/aws-sam-cli/releases/tag/v1.9.0 mention caching is not supported yet with a custom Makefile

[qingchm]

@ahurlburt Sorry for the late reply but what you are experiencing is unfortunately an expected behaviour because the code_uri that you have for the two functions are the same. For SAM-CLI we are evaluating the cache changes based on the code directory content of the function. So since you added a new file into the code directory, the code directory is now changed so the cache will no longer be valid. In order to use --cached flag smoothly please not share the same code_uri for different functions! We would really expect each function and layer to have their unique uri!

[ahurlburt]

@qingchm thank you for the response.

I just wanted to clarify what you are saying about the code_uri.. are you saying it's expected that each lambda has a different code_uri? I have different handlers for each lambda and in cases different files but they are under the same code_uri, does this not follow best practice and if not are there any reference/docs describing best practice for lambda structure that mentions the unique code_uri, etc?

I was also thinking that the cache flag would cause it to skip dependency building (since those don't change often) and it seems that using the --beta-features flag actually does support that. Though as I read more I think a better approach would be to use a lambda layer to store the deps.

More info on that in case anyone else reads this and is interested: #805

[qingchm]

@ahurlburt Right, when two lambda's share the same code uri folder, even if they have different handlers the change of one function would result in the other's cache status to be regarded as "changed". I'm glad that you are trying out our new approach to automatically create dependency layers! This will make cache build way clearer as code and dependencies are now treated as separate components. Right now the languages that support auto dependency layer creation is limited but python is one of them! When you build with --beta-features you will see the log message like "Auto creating dependency layer for each function resource into a nested stack", and you can check the deployed stacks to see how this changes your stack structure! Hope you like this feature and let me know if this solves your problems!

[thoward-godaddy]

Tried out the --beta-features for a python lambda and it worked perfectly. Thank you!

[ahurlburt]

Thanks @qingchm I'm closing this. I ended up creating my own lambda layer for dependency and it solved my build problems. @thoward-godaddy I would recommend creating your own layer as well as this will make deploys faster and just in general provide for a much better dev experience.