Help! I'm randomly getting '410 Gone: editDeleted' errors. What's going on?

[SUPERCILEX]

Well aren't you a little speed demon! 😈 Unfortunately, you're going to have to sloooowwwww down. 🛑

Ok, so what's happening? In brief, there is such a thing as builds that run too fast. Disappointing, I know.

Workaround

Add this to your app/build.gradle.kts file:

// TODO remove
tasks.matching { it.name.startsWith("publish") }.configureEach {
    outputs.file(File(rootProject.buildDir, "gpp-hack"))
}

This basically tells gradle that tasks whose name starts with publish cannot execute in parallel. Otherwise, they would step over each other trying to write to the same output directory, project/build/gpp-hack.

I want details!

Here's the full error message:

{
  "code" : 410,
  "errors" : [ {
    "domain" : "androidpublisher",
    "message" : "This Edit has been deleted.",
    "reason" : "editDeleted"
  } ],
  "message" : "This Edit has been deleted."
}

This error gets thrown because of a bug in the Android Publisher API: if two requests for an edit ID occur within a few seconds of each other, the same ID will be returned. This is a problem because GPP will then try to commit that edit twice or more—once in the first task, another time in the other task, and so on. Commiting an edit is an atomic operation, meaning the edit gets deleted and can no longer be used. Hence, the other tasks unknowingly sharing the same edit ID will end up using an invalid edit ID.

Potential solutions

1. [IDEAL] Fix the bug in the Android Publisher server code. I'll start poking around next week, but my guess is that it's not going to be that simple.
2. [OK] Use a shared pool of edit ID. This sucks for two reasons:
   1. Boundaries between tasks are lost. One might fail because of an incorrect configuration in another.
   2. We can't use an in-memory pool since Gradle will parallelize across processes, etc. This means we'll have to use a file-based pool which is gross.
3. [BAD] Block any parallelization across any GPP tasks by using a slightly smarter version of the workaround above.

[ponsuyambu]

After adding the workaround also, the build still fails randomly with the same error :(

[SUPERCILEX]

@ponsuyambu what version of GPP are you using?

[technoir42]

@SUPERCILEX we are getting this error on 2.5.0 too. We upload 1 flavor with 3 ABI splits + universal APK, hence 4 errors:

> 410 Gone
--
{
"code" : 410,
"errors" : [ {
"domain" : "androidpublisher",
"message" : "This Edit has been deleted.",
"reason" : "editDeleted"
} ],
"message" : "This Edit has been deleted."
}
> A failure occurred while executing com.github.triplet.gradle.play.tasks.PublishApk$ApkUploader
> 410 Gone
{
"code" : 410,
"errors" : [ {
"domain" : "androidpublisher",
"message" : "This Edit has been deleted.",
"reason" : "editDeleted"
} ],
"message" : "This Edit has been deleted."
}
> A failure occurred while executing com.github.triplet.gradle.play.tasks.PublishApk$ApkUploader
> 410 Gone
{
"code" : 410,
"errors" : [ {
"domain" : "androidpublisher",
"message" : "This Edit has been deleted.",
"reason" : "editDeleted"
} ],
"message" : "This Edit has been deleted."
}
> A failure occurred while executing com.github.triplet.gradle.play.tasks.PublishApk$ApkUploader
> 410 Gone
{
"code" : 410,
"errors" : [ {
"domain" : "androidpublisher",
"message" : "This Edit has been deleted.",
"reason" : "editDeleted"
} ],
"message" : "This Edit has been deleted."
}

[SUPERCILEX]

@technoir42 I'm confused, what tasks are you running? If it's just one flavor, it should only be a single task which makes a conflict impossible.

[technoir42]

We are running just one task publishProductionReleaseApk but we have ABI splits enabled so the plugin uploads 4 APKs in parallel.

[SUPERCILEX]

@technoir42 Hmmm, does this error happen every time you try and run the task? My guess is that you just have some bad state left over, but I don't understand why GPP wouldn't clean up after itself. Try running ./gradlew --stop && rm -rf build && ./gradlew clean && ./gradlew publishProductionReleaseApk in your project root. If that doesn't work, please open a new issue, fill out the template, and share the logs of running ./gradlew publishProductionReleaseApk --info -S.

[technoir42]

I don't think some old state could be the reason because we are uploading from CI while doing a clean build with disabled build cache. We just integrated this plugin and the very first upload failed with an error.

[SUPERCILEX]

Gotya, then something's got to be wrong with the setup (I just checked and am able to upload splits). I can't really help you without knowing the state of the project. Are all the build folders gone? What are the logs from running with --info -S?

[ccjernigan]

I've been frequently seeing this error message when uploading a single app bundle from my CI server, with 2.4.2 and 2.5 versions of the plug-in. (Also using Gradle 5.6.4 and AGP 3.5.2).

I do use a build cache, but I doubt that's the issue because re-triggering the build with no code changes will eventually succeed (sometimes it takes more than one try).

The task I'm using is:
./gradlew :app:publishBundle :app:packageReleaseUniversalApk

What other info might be helpful to track this down?

[ponsuyambu]

I got this error suddenly even when I run the lint task. I think running parallel jobs on the CI server causes problem. I never encountered this error when there is only one job is running on the code base.

[SUPERCILEX]

@ccjernigan @ponsuyambu Dang, I didn't know it was that bad. Just to clarify, the AndroidPublisher API has limitations GPP can't really get around. These are the cases GPP should handle:

• Modifying different apps (keyed by package name) concurrently
• Modifying the same app in different modules concurrently as long as they're in the same Gradle build

These cases will not work:

• Modifying the same app in separate Gradle builds concurrently (e.g. different jobs in CI)

If that's not the case for you, please share the output of a failed build run with --info -S. If possible, it would be helpful to see the state of the projectRoot/build/gpp/ directory.

[tasomaniac]

I ended up here after seeing this issue in our build.

Ours was because Jenkins CI was running 2 full builds in parallel and coincidentally GPP tasks were running at the same time. They were overriding each other.

[Chuck-Townsend]

I thought that this change: #632 would fix the problem but I'm using v2.4.2 which includes that change and the problem still exists. Like tasomaniac, I can see the problem when running 2 Jenkins CI publishing builds on two different agents at the same time, publishing different apps to different tracks using the same google play credientials.