Bug: sam sync with python causes endless cpu saturation

[cshenrik]

Description:

When running sam sync --watch --region us-east-1 on a python project, the sam sync process utilizes 100% cpu until I kill it (after at least 30 minutes)

Steps to reproduce:

1. Create a new python project using sam init.
2. Use python3.11
3. Run sam deploy --guided
4. Run sam sync --watch --region us-east-1

After it has deployed the application and the CPU utilization has gone down, it suddenly goes to 100% and stays there.

Observed result:

The last part of the output from sam sync:

2024-05-13 21:53:37,488 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,489 | Sam customer defined id is more priority than other IDs. Customer defined id for resource CreateSignedUrlFunction is CreateSignedUrlFunction                                                  
2024-05-13 21:53:37,490 | There is no customer defined id or cdk path defined for resource ServerlessHttpApi, so we will use the resource logical id as the resource id                                                 
2024-05-13 21:53:37,491 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,492 | Sam customer defined id is more priority than other IDs. Customer defined id for resource CreateSignedUrlFunction is CreateSignedUrlFunction                                                  
2024-05-13 21:53:37,493 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ServerlessHttpApi is ServerlessHttpApi                                                              
2024-05-13 21:53:37,493 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,495 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,496 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,497 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,498 | Sam customer defined id is more priority than other IDs. Customer defined id for resource CreateSignedUrlFunction is CreateSignedUrlFunction                                                  
2024-05-13 21:53:37,499 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,501 | Sam customer defined id is more priority than other IDs. Customer defined id for resource CreateSignedUrlFunction is CreateSignedUrlFunction                                                  
2024-05-13 21:53:37,503 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,503 | Sam customer defined id is more priority than other IDs. Customer defined id for resource CreateSignedUrlFunction is CreateSignedUrlFunction                                                  
2024-05-13 21:53:37,505 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,506 | Sam customer defined id is more priority than other IDs. Customer defined id for resource CreateSignedUrlFunction is CreateSignedUrlFunction                                                  
2024-05-13 21:53:37,507 | 3 resources found in the stack                                                                                                                                                                
2024-05-13 21:53:37,508 | Found Serverless function with name='CreateSignedUrlFunction' and CodeUri='create_signed_url'                                                                                                 
2024-05-13 21:53:37,510 | --base-dir is not presented, adjusting uri create_signed_url relative to <<<<REDACTED>>>>                                                
2024-05-13 21:53:37,513 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,514 | Sam customer defined id is more priority than other IDs. Customer defined id for resource CreateSignedUrlFunction is CreateSignedUrlFunction                                                  
2024-05-13 21:53:37,515 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ServerlessHttpApi is ServerlessHttpApi                                                              
2024-05-13 21:53:37,516 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,517 | Sam customer defined id is more priority than other IDs. Customer defined id for resource CreateSignedUrlFunction is CreateSignedUrlFunction                                                  
2024-05-13 21:53:37,518 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ServerlessHttpApi is ServerlessHttpApi                                                              
2024-05-13 21:53:37,519 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,519 | Sam customer defined id is more priority than other IDs. Customer defined id for resource CreateSignedUrlFunction is CreateSignedUrlFunction                                                  
2024-05-13 21:53:37,520 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ServerlessHttpApi is ServerlessHttpApi                                                              
2024-05-13 21:53:37,521 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ImageBucket is ImageBucket                                                                          
2024-05-13 21:53:37,522 | Sam customer defined id is more priority than other IDs. Customer defined id for resource CreateSignedUrlFunction is CreateSignedUrlFunction                                                  
2024-05-13 21:53:37,523 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ServerlessHttpApi is ServerlessHttpApi                                                              
2024-05-13 21:53:37,524 | CodeTrigger not created as CodeUri or DefinitionUri is missing for ServerlessHttpApi.                                                                                                         
2024-05-13 21:53:37,525 | Skipped infra sync as the local template is in sync with the cloud template.```


### Expected result:
`Sam sync` should use almost no cpu when no changes are being made.


### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

<!-- Either provide the following info (for AWS SAM CLI v1.68.0 or before) or paste the output of `sam --info` (AWS SAM CLI v1.69.0 or after). -->

1. OS: Ubuntu 22.04.4 LTS
6. `sam --version`: SAM CLI, version 1.116.0
7. AWS region: us-east-1

Paste the output of sam --info here

{
  "version": "1.116.0",
  "system": {
    "python": "3.11.8",
    "os": "Linux-6.5.0-28-generic-x86_64-with-glibc2.35"
  },
  "additional_dependencies": {
    "docker_engine": "26.1.0",
    "aws_cdk": "2.97.0 (build d7cf3be)",
    "terraform": "1.5.7"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[cshenrik]

Found out that it is triggered by touch template.yaml. So whenever I open template.yaml in my vscode, sam sync will instantly consume all CPU. This renders sam sync --watch rather difficult to use.

[mndeveci]

Hi @cshenrik thanks for reporting this issue.

Are you using any plugin in VSCode which might be changing the file constantly? We are listening to file change events which is published by OS, and we usually drop the ones which doesn't change the file (open event for instance).

I've quickly tested the scenario locally, but SAM CLI is not showing a high CPU usage like 100%.

[cshenrik]

Hi @mndeveci,
I can reproduce the behaviour without VSCode running. Just using touch template.yaml while sam sync --watch --region us-east-1 is running.

[mndeveci]

Running touch template.yaml in another terminal also didn't trigger this issue on my side. I saw that you are using a linux OS (Ubuntu) I can try to reproduce it on that machine and get back to you.

[mndeveci]

@cshenrik I've tried on a linux machine as well, however it doesn't spike the CPU usage. Are there any more details that can help us to re-produce the issue on our side?

[cshenrik]

@mndeveci, I've tested it on older versions of sam-cli, and I found that it has been introduced in v1.97.0. I cannot reproduce it on earlier versions.

Let me know what I can do to help.

[mndeveci]

Thanks for pinning the issue. I can see that we bumped the watchdog dependency to a major version which is used for listening file changes. I've quickly checked their repository but I couldn't find any relevant issue there.

We can add more debug logs to see which event is getting triggered with your setup, and use that information (to see which even is getting triggered and causing high cpu usage) to open an issue on their repo. Would you be able to run a development version of SAM CLI and try that with additional logs?

I forgot to ask but how do you install SAM CLI, are you using official installers or an alternative method?

[cshenrik]

Sure, I can run a development version.

And yes, I use the official installers.

[mndeveci]

I've added some logs and also an early return if we receive file open event. Can you pull this branch and test it on your side? https://github.com/mndeveci/aws-sam-cli/tree/add_logs_for_file_watchers

You need to create a python virtual environment with python3.11, activate it and run make init. After that, in that virtual environment, you should have samdev available. And you can try samdev sync ...{additional parameters}.

What I've seen so far, we are getting constant file opened event if you do touch template.yaml or vim template.yaml. And once we receive an event, we do load the file and do some comparison to decide whether we need to trigger an infra sync (deployment) or not. So loading that file and comparing it might be causing your high CPU usage.

Let me know if this branch fixes issue on your side, we can then move forward and raise a PR for the fix.

[cshenrik]

That fixed it!

[github-actions]

Patch is released in v1.118.0. Closing