Make manifest upload less error prone

[ipanova]

Version
main

Describe the bug
Image push can fail with 429 too many requests.
Client can properly backoff during 429 blob upload 1s,2s,4s,8s,16s (tot 31s) and wait for the task to finish, client does not back off during manifest push. Api is locked only for 3 seconds and if within that time tasks did not complete, client will re-try ( multiple times, podman/skopeo re-tries 3 times by default )by making another request which will lead to another task creation. Only first task will be operational, other subsequent tasks will be no-op.
The overall podman push command will have a non 0 exit code, however the image manifest will be properly uploaded to the registry. It means that task will complete after client will fail

I am not sure what would be best to do.
a) we can lock api for 3+ seconds but this will fail anyway if workers are under load and task will be picked up after 3+ seconds, locking api for long time is not good approach anyway https://github.com/pulp/pulp_container/blob/main/pulp_container/app/utils.py#L130
b) we could check in the existing repo version whether such manifest and tag already are there and not dispatch the task ( which will anyway be no-op). This is also a half solution because first try will anyway fail( task will be dispatched/finish after 3 secs, however on the second try no task will be dispatched and 201 will be returned to the client https://github.com/pulp/pulp_container/blob/main/pulp_container/app/registry_api.py#L1005
c) dispatch task for the manifest upload but do not check it result/wait for it and return 201 https://github.com/pulp/pulp_container/blob/main/pulp_container/app/registry_api.py#L1021 . This is not best either because task might not be picked up for X time and client will not be able to pull from that repo and in adidition will not know why because of misleading 201
d) other

To Reproduce

$ skopeo copy --retry-times 5    docker://registry.redhat.io/ansible-automation-platform-23/ee-minimal-rhel8 docker://pulp3-source-fedora36.puffy.example.com/ee-minimal-rhel8:latest --remove-signatures --dest-creds admin:password
Copying blob 370fc4ac3f1f skipped: already exists  
Copying blob 7ecdec58f172 skipped: already exists  
Copying blob 272c29e2f36c skipped: already exists  
Copying config 5a85baceb7 done  
Writing manifest to image destination
WARN[0005] Failed, retrying in 1s ... (1/5). Error: copying system image from manifest list: writing manifest: uploading manifest latest to pulp3-source-fedora36.puffy.example.com/ee-minimal-rhel8: unsupported: Request was throttled. 
Copying blob 370fc4ac3f1f skipped: already exists  
Copying blob 7ecdec58f172 skipped: already exists  
Copying blob 272c29e2f36c skipped: already exists  
Copying config 5a85baceb7 done  
Writing manifest to image destination
WARN[0010] Failed, retrying in 2s ... (2/5). Error: copying system image from manifest list: writing manifest: uploading manifest latest to pulp3-source-fedora36.puffy.example.com/ee-minimal-rhel8: unsupported: Request was throttled. 
Copying blob 7ecdec58f172 skipped: already exists  
Copying blob 370fc4ac3f1f skipped: already exists  
Copying blob 272c29e2f36c skipped: already exists  
Copying config 5a85baceb7 done  
Writing manifest to image destination
WARN[0016] Failed, retrying in 4s ... (3/5). Error: copying system image from manifest list: writing manifest: uploading manifest latest to pulp3-source-fedora36.puffy.example.com/ee-minimal-rhel8: unsupported: Request was throttled. 
Copying blob 370fc4ac3f1f skipped: already exists  
Copying blob 7ecdec58f172 skipped: already exists  
Copying blob 272c29e2f36c skipped: already exists  
Copying config 5a85baceb7 done  
Writing manifest to image destination
WARN[0024] Failed, retrying in 8s ... (4/5). Error: copying system image from manifest list: writing manifest: uploading manifest latest to pulp3-source-fedora36.puffy.example.com/ee-minimal-rhel8: unsupported: Request was throttled. 
Copying blob 272c29e2f36c skipped: already exists  
Copying blob 7ecdec58f172 skipped: already exists  
Copying blob 370fc4ac3f1f skipped: already exists  
Copying config 5a85baceb7 done  
Writing manifest to image destination
WARN[0036] Failed, retrying in 16s ... (5/5). Error: copying system image from manifest list: writing manifest: uploading manifest latest to pulp3-source-fedora36.puffy.example.com/ee-minimal-rhel8: unsupported: Request was throttled. 
Copying blob 370fc4ac3f1f skipped: already exists  
Copying blob 272c29e2f36c skipped: already exists  
Copying blob 7ecdec58f172 skipped: already exists  
Copying config 5a85baceb7 done  
Writing manifest to image destination
FATA[0056] copying system image from manifest list: writing manifest: uploading manifest latest to pulp3-source-fedora36.puffy.example.com/ee-minimal-rhel8: unsupported: Request was throttled. 

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
#1211 This behavior exacerbates the overall situation which leads to additional tasks created.

[mdellweg]

Mainly for completeness' sake more solution ideas:

• Associate "pending" blobs to a repository without creating a repository version (no need for a task). Only add blobs with the manifests to a new repository version.
  • Benefit: Less tasks, less repo versions, blob upload much more responsive.
  • Issue: Manifest creation will still trigger a task and block the api for up to 3 seconds.
• Implement immediate tasks. The task will be executed immediately by the worker, given the resource constraints are given, otherwise it will be canceled right away and 429 returned.
  • Benefit: No waiting for tasks in the api.
  • Contra: A locked repository will not be operable by the push api. Podman (and other clients) pushes blobs concurrently, giving a high chance for a locked repository.

For the specific issue described here, both options may be combined.

[ipanova]

After brainstorming with @mdellweg the suggested approach from his comment will eliminate the tasking load during blob mount #1211 and blob upload.
The challenge is to figure out how to represent 'new pending blobs' because they will not be part of any repo version, make sure they are not wiped out by orphan cleanup during the upload(use touch() for existing content).
During manifest PUT call, while already parsing the manifest, we will ask DB for the expected pending and existing blobs by its sha and in the task add those and manifest at once using co-called 'immediate' tasks. It is also worth adding a check whether such manifest with the tag already exist in the previous repo version to avoid execution of the task which will be no-op( suggestion b from the initial comment).