Upload child commit to pulp takes more than 40 minutes #289
Comments
|
Can you provide a script that I can use for testing this performance issue locally, please? |
|
I think you can refer to this https://github.com/osbuild/osbuild-composer/blob/main/test/cases/ostree.sh This script is to test osbuild-composer, but it has steps to build parent commit and child commit, I think this is what you need. It builds two commits, the first one is parent commit, and then it use the parent hash to build a child commit. You can find the steps to build parent and child commit and test the upload locally. I also have a new version of this scrip to test pulp in osbuild-composer, you can also refer to it. osbuild/osbuild-composer@a80135b |
|
This might be related to #169. @yih-redhat, how many content units does this repository have before and after the subsequent upload? |
|
hmm, I am not very familiar with pulp, can you please give me a command that I can use to check? How to run this command? I tried to add curl before this ,but it doesn't work. |
|
The |
|
Hi, there is no output for these commands: [root@yih-rr93 ~]# http localhost:9090/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/1/ | jq -r '.content_summary.present' |
|
Are you sure you are querying the correct repository? |
|
Here is my env, can you please help to check if my command is correct? [root@yih-rr93 ~]# pulp ostree repository list [root@yih-rr93 ~]# http :9090/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/1/ | jq -r '.content_summary.present' d8f34de49f162c956c9261f69dadfba036d1490928b5719b0b539f34d05d07cf |
|
So, together with @yih-redhat, we managed to get the information about the repository in question. The repository has got 29507 ostree objects in total, adding 198 after the second import. I conclude that this issue is closely related to #169: |
|
What was the baseline for uploading commits in the old workflow (no pulp)? Is there any threshold considered to be satisfactory? |
|
Duplicate of #169 |
|
I profiled the original code and here are the results:
Things to consider:
Reproducer
|
|
Then, I profiled the code with the "optimized" solution for querying existing artifacts:
Things to consider:
ReproducerSame as in the previous comment and modified async def run(self):
"""
The coroutine for this stage.
Returns:
The coroutine for this stage.
"""
async for batch in self.batches():
artifacts_digests = []
# For each unsaved artifact, check its digests in the order of COMMON_DIGEST_FIELDS
# and the first digest which is found is added to the list of digests of that type.
# We assume that in general only one digest is provided and that it will be
# sufficient to identify the Artifact.
for d_content in batch:
d_artifact = d_content.d_artifacts[0]
if d_artifact.artifact._state.adding:
digest_value = d_artifact.artifact.sha256
artifacts_digests.append(digest_value)
# For each type of digest, fetch all the existing Artifacts where digest "in"
# the list we built earlier. Walk over all the artifacts again compare the
# digest of the new artifact to those of the existing ones - if one matches,
# swap it out with the existing one.
query_params = {
"sha256__in": artifacts_digests,
"pulp_domain": self.domain,
}
existing_artifacts_qs = Artifact.objects.filter(**query_params)
await sync_to_async(existing_artifacts_qs.touch)()
d = {}
async for result in sync_to_async_iterable(existing_artifacts_qs):
d[result.sha256] = result
for d_content in batch:
d_artifact = d_content.d_artifacts[0]
artifact_digest = d_artifact.artifact.sha256
m = d.get(artifact_digest)
if m:
d_artifact.artifact = m
for d_content in batch:
await self.put(d_content) |
|
@yih-redhat, if we execute the following commands and there is no error raised, it proves that the content on the repository is valid and not corrupted, right? |
|
Verified with @yih-redhat that this PR #309 is working. |
In this change, the stage is comparing only sha256 digests between existing and newly added artifacts. Furthermore, the time complexity for the old comparison was O(N^2). With the attached change, we work with O(2N). closes pulp#289
In this change, the stage is comparing only sha256 digests between existing and newly added artifacts. Furthermore, the time complexity for the old comparison was O(N^2). With the attached change, we work with O(2N). closes #289
Version
docker.io/pulp/pulp:latest
Describe the bug
Upload parent commit takes around 5 minutes, but upload child commit takes more than 40 minutes, that's too long as there is only minor update content in child commit.
To Reproduce
[root@yih-rr93 ~]# pulp ostree repository import-all --name test --file 63ce5a81-96e7-47b2-abf1-02f6334b069a-commit.tar --repository_name repo
Uploading file 63ce5a81-96e7-47b2-abf1-02f6334b069a-commit.tar
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Upload complete.
Creating artifact.
Started background task /pulp/api/v3/tasks/018a8d74-ae70-73b4-bd16-bb1a32568fc6/
....Done.
Started background task /pulp/api/v3/tasks/018a8d74-c30f-72df-ba11-06a7494216ea/
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done.
[root@yih-rr93 ~]# pulp ostree repository list
{
"pulp_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/",
"pulp_created": "2023-09-13T07:28:28.587741Z",
"versions_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/",
"pulp_labels": {},
"latest_version_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/1/",
"name": "test",
"description": null,
"retain_repo_versions": null,
"remote": null,
"compute_delta": true
}
[root@yih-rr93 ~]# pulp ostree repository import-all --name test --file d5fe22f7-bc2c-4772-960f-66977c39eba9-commit.tar --repository_name repo
Uploading file d5fe22f7-bc2c-4772-960f-66977c39eba9-commit.tar
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Upload complete.
Creating artifact.
Started background task /pulp/api/v3/tasks/018a8d93-1ece-7f69-ba3e-0ab4bb97aa28/
....Done.
Started background task /pulp/api/v3/tasks/018a8d93-3376-70a4-8192-888e6f674b4c/
...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done.
[root@yih-rr93 ~]# pulp ostree repository list
{
"pulp_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/",
"pulp_created": "2023-09-13T07:28:28.587741Z",
"versions_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/",
"pulp_labels": {},
"latest_version_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/2/",
"name": "test",
"description": null,
"retain_repo_versions": null,
"remote": null,
"compute_delta": true
}
Expected behavior
Upload child commit should not take too long, comparing to parent commit, there is no much new content change in child.
Additional context
Add any other context about the problem here. Please provide links to any previous discussions via Discourse or Bugzilla.
The text was updated successfully, but these errors were encountered: