Bug: Failed to CreateCacheEntry

[icanhazstring]

Greetings.

Since today Feb 7th around 1AM CET we got a problem with the actions/cache.
We can't use actions/cache/save anymore and get the following error

Warning: Failed to save: Failed to CreateCacheEntry: Received non-retryable error: Failed request: (404) Not Found: invalid request

This is a simple cache step as described in the docs:

      - name: Cache vendor
        uses: actions/cache/save@v4
        with:
          key: ${{ env.VENDOR_CACHE_KEY }}
          path: |
            vendor/
            vendor-bin/

Current workaround:
Fix the version to v4.1.2. This should solve the issue for now.

[tanya-ok]

Did you find any workaround?

[domman95]

We've noticed the same issue. actions/cache/restore@v3 returns a warning

Warning: Failed to restore: Failed to GetCacheEntryDownloadURL: Received non-retryable error: Failed request: (404) Not Found: invalid request

our cache step looks pretty the same as above

[icanhazstring]

Yes. Workaround is to fix the version to v4.1.2 then it works again.

[zsliu98]

I have the same issue. From #1510, I would guess the problem is related to the new cache service (v2) APIs. Cause v4.1.2 will become deprecated in one month, it is only a temporary solution. Could somebody fix it as soon as possible? (TBH I don't know where to report the issue other than here)

---

Update: Yes, it works again. I am using v4.

[domman95]

I think it works again - we use v3.

[deitch]

I am curious as to why the action didn't fail when the cache upload failed? I get that the upload might have failed, either due to a latent bug in the action or changes in the API (or some combination). Should the action itself not have failed when the upload didn't succeed?

[icanhazstring]

I am curious as to why the action didn't fail when the cache upload failed? I get that the upload might have failed, either due to a latent bug in the action or changes in the API (or some combination). Should the action itself not have failed when the upload didn't succeed?

The upload only gives a warning.
Retrieving gives an error if you use the fail on miss setting only.

And I also created a support ticket where they mentioned they rolled back the change.
As they updated there backend there seems to be an issue that devs reverted already. So it should work with the latest version again.

[deitch]

Yeah following this issue, I restarted the various jobs I manage and see them working again.

Would it not make sense to have the action fail (error) on upload error, instead of warning?

[Link-]

On February 7th we began the rollout of the new cache service backend. There was a bug in the implementation that caused a wider range of repositories to start calling the new backend even though they weren't intended to do so just yet. The service responded with 404s.

That problem was resolved on the same day. Please do not revert back to previous versions.

There are only 4 compatible versions with the new service backend. Please only use one of the below versions:

Supported versions and tags
actions/cache@v4
actions/cache@v3
actions/cache@v4.2.0
actions/cache@v3.4.0

Every other version is deprecated: #1510

[Tiphereth-A]

In my repository, it reports

Failed to save: Failed to CreateCacheEntry: Received non-retryable error: Failed request: (409) Conflict: cache entry with the same key, version, and scope already exists

so maybe there is another issue of new service backend.

[Link-]

In my repository, it reports

Failed to save: Failed to CreateCacheEntry: Received non-retryable error: Failed request: (409) Conflict: cache entry with the same key, version, and scope already exists

so maybe there is another issue of new service backend.

@Tiphereth-A - This is not an error, cache entries are immutable. If you're trying to create a cache entry with the same key, you will get this warning. In a future release we will be removing this annotation to reduce confusion.

[alexanderb2911]

I'm currently also facing this issue. The save action is not able to save a new cache although none are existing at that moment. It always tells me that the cache is already existing although it's not.

[yonitou]

Same issue here.
We are using the v4.0.0 @actions/cache package and the error Failed to save: Failed to CreateCacheEntry: Received non-retryable error: Failed request: (409) Conflict: cache entry with the same key, version, and scope already exists always pops even if the cache is completely new and clear.
We upgraded to v4.0.0 2 days ago and the error started to happen 2 hours ago

[Link-]

@yonitou @alexanderb2911 - we need more information to troubleshoot. If you have public repos, share the workflow runs that failed here.

Ohterwise, please create support tickets and make sure to include links to the workflows that are failing with the 409 Conflict errors.

Also, please share the link to this issue and ask the support engineer to escalate to the Actions team please.

[cderv]

@Link- We are also facing the issue in this public workflow

• https://github.com/quarto-dev/quarto-cli/actions/runs/13396218760/job/37415844748?pr=12108

Both when trying to update to using actions/cache/restore and actions/cache/save but also while using
julia-actions/cache which seems to use a pinned version of actions/cache: https://github.com/julia-actions/cache/blob/2b1bf4d8a138668ac719ea7ca149b53ed8d8401e/action.yml#L124-L131

Our current usage of caching involves:

• Using job matrix for several job in parallel
• Each of them having a cache step so they check for cache-hit, and needs to save it (we used save-always in the past).

However, it seems that even using cache-hit is problematic

• For a new cache, all workflow with have cache-hit = false
• The first job that end will save the cache without problem
• All the other will try to save the cache as cache-hit = false, but key already exists (because of first job that end) and warning is thrown

I would be happy to have misunderstood though...

Anyhow, I'll be interested to really understand the new constraint to correctly adapt key and restore/save in a situation of multiple job created through Matrix. All jobs should share a cache and there should not be one for each matrix element.

Thanks a lot.

[Link-]

@cderv - I looked at your setup, the cache is being saved, it's being saved once and any attempt to save it after that you will get this warning.

The warning annotation is most likely what's causing confusion because the legacy service would have not set a warning annotation for a cache entry key collision. This is not a failure, your workflow jobs are finishing successfully.

Warning: Failed to save: Failed to CreateCacheEntry: Received non-retryable error: Failed request: (409) Conflict: cache entry with the same key, version, and scope already exists

The (409) Conflict is normal and expected behaviour. Cache entries are also immutable at the moment, there are no configurable parameters to change that behaviour.

I'll work with the team to reverse the warning annotation addition to prevent further confusion after I verify the behaviour with the former versions.

Am I misunderstanding the situation?

[cderv]

Am I misunderstanding the situation?

I think you got it right.

The warning annotation is most likely what's causing confusion because the legacy service would have not set a warning annotation for a cache entry key collision. This is not a failure, your workflow jobs are finishing successfully.

Yes there is no failure. One of the job is doing the saving, and the other jobs try to save later on but there is this warning thrown from each one of them as they indeed can't. This clutter the log with a lot of warning, and this indeed created confusion has it seemed to work ok.

The (409) Conflict is normal and expected behaviour.

That is good to know. This confirm my initial understanding. I will go on with my initial change.

I'll work with the team to reverse the warning annotation addition to prevent further confusion after I verify the behaviour with the former versions.

That would really help to not have this warning thrown by default, especially if everything is going as expected. maybe annotate in workflow debug mode only ? or do not create an annotation that go up in summary ?

This created the confusion indeed, especially as I did not see anyway to avoid each following job to detect cache have been saved already, and not try to save the cache.

Thank you.

[Link-]

To conclude this report. The previous implementation of actions/cache also posts an annotation when it fails to upload a cache entry if the key exists already. The new version is just a bit more verbose.

I'll adjust the annotation text to match the former version and add some guidance.

[cderv]

It did not remember this and this was less prominent. It means it woud still be a good thing to find a way to avoid this warning, but this is another topic. Thanks for having looked into this !

[dime10]

@Link- We have a case of the 409 error showing up even though there is no existing entry with that key. In fact, we cleared the entire cache and the error still shows up. This is the most recent run: https://github.com/PennyLaneAI/catalyst/actions/runs/13418034619/job/37483671023
Specifically, the Cache LLVM Build step, with the Linux-llvm-3a8316216807d64a586b971f51695e23883331f7-default-build-gcc key. There are no cache entries with that key currently present in the cache, nor have any been created since the entire cache was wiped.

(Note while the Cache LLVM Source step shows the same warning, that one is legitimate since a parallel job created that entry while this job was running.)

[Link-]

@dime10 - let me review, thanks for sharing the run

[Link-]

We identified the problem, thank you all for chiming in! The fix will be released in a day or two.

In short, when a cache entry is created, if for whatever reason it's not finalized that slot is locked and will not be reused in future attempts to create a cache entry with the same key. Since the cache entry was never finalized, it will also not be served.

[yonitou]

It seems to be exactly the issue :) after changing the name of the cache key I used before (even if my cache was completely empty), the 409 errors disappeared

[erlichmen]

My case is a little different. I expect to get this warning because I have several parallel jobs, and the first one to complete will save the cache while the rest will receive the warning. I think this is a valid pattern.

In my opinion, this warning should be removed or at least moved to the debug level. I don't care that there is already a cache entry, and moreover, there is nothing I can do about it.

[MikeMcC399]

@Link-

We identified the problem, thank you all for chiming in! The fix will be released in a day or two.

Is this a backend change? Will you let us know when the fix has been released?

[Link-]

@Link-

We identified the problem, thank you all for chiming in! The fix will be released in a day or two.

Is this a backend change? Will you let us know when the fix has been released?

This is a backend change and I'll update this issue once we deploy the fix.

[jozefizso]

We have this issue on Windows Server and macOS runners on many different workflows.

[tking16]

Similar issue here (MacOS)

[yonitou]

The issue just started to appear again .. It's quite urgent to fix it @Link-

[yonitou]

Sorry to post again but we are completely stuck since yesterday and I need to insist on this matter.
@Link- , I have couple of questions :

• When this will be fixed ? We need something quite precise because it's just breaking all our pipelines ..
• In our package.json, we are currently using "@actions/cache": "4.0.0" from https://github.com/actions/toolkit/tree/main
  Can it be related ? As I said, the problem was fixed for one week and came back yesterday around 13h00 UTC

[MikeMcC399]

@yonitou

You are commenting on an issue which is only about a warning which does not prevent workflows from running. You may have a different issue.

The latest version of @actions/cache is 4.0.2 - see https://github.com/actions/toolkit/blob/main/packages/cache/RELEASES.md. Perhaps you should be updating to this version?

[yonitou]

Hi @MikeMcC399, thanks for your quick answer.
If I understood well the issue, it's related to a bug with the Failed to CreateCacheEntry log. Our error is exactly the same :

The key mentioned in the error is not even in our cache. It's an error from the actions/cache backend resulting in a false positive and not saving the cache key.

Our workflows need the cache to make up some critic actions depending on the diff between 2 workflows.

If we update the package to 4.0.2, will it fix it ?

[MikeMcC399]

@yonitou

If we update the package to 4.0.2, will it fix it ?

According to #1541 (comment) a backend change is needed to fix the warning. The warning is not fixed by updating @actions/cache.

In your situation I would test updating to the latest version to see if it improves your workflow success.

[yonitou]

Thanks @MikeMcC399
Just updated to 4.0.2 and the warning is still there, the cache key is not saved.
So, until the backend change is made, I have no solution ? Do you have an ETA ?

[MikeMcC399]

@yonitou

Do you have an ETA ?

I am a community member like you, so I can't provide any ETA. I also doubt whether the backend fix will solve your problem. The fix is only to suppress the warning. That's all. You may need to open a new issue for your problem.

[cderv]

the cache key is not saved.

Did you check your caches ? You can use gh cache list

The problem we discussed until now in this discussion is about the Warning annotations when a workflow tries to save a cache with a key that is already saved (by a concurrent workflow for example).

See #1541 (comment) and #1541 (comment)

And specific one related to new cache #1541 (comment) - maybe you are hitting that one.

In short, when a cache entry is created, if for whatever reason it's not finalized that slot is locked and will not be reused in future attempts to create a cache entry with the same key. Since the cache entry was never finalized, it will also not be served.

They say they will update as soon as backend is fixed (#1541 (comment))

[yonitou]

Thanks @MikeMcC399 for your help
@cderv, I check my cache multiple times. The cache key is not there. The slot is just locked as you mentioned. This is exactly what I'm talking about

[cderv]

The slot is just locked as you mentioned. This is exactly what I'm talking about

Ok. Sorry then for the unnecessary repeat. I wasn't sure. Then we are waiting for the official update.

[Link-]

@yonitou et al, the backend change is coming this week. We had to iterate on a couple of ideas and put the change in motion. In short, cache keys that are locked due to a failed upload will be cleared every 24 hours from the time they're created (after the backend fix finishes rolling out).

In the meantime, I recommend that you maintain some level of flexibility with your cache keys. As a best practice, cache availability should not be blocking to your workflows even if we aim to provide the same level of stability and availability as workflow artifacts and packages.