ostree pull cannot pull child commit content from pulp

[yih-redhat]

Version
docker.io/pulp/pulp:latest

Describe the bug
Upload parent commit into pulp and ostree pull can pull the commit content from pulp, then upload child commit into pulp, but ostree pull cannot find the new commit content.

To Reproduce

1. Build a parent commit and upload to pulp.
   [root@yih-rr93 ~]# pulp ostree repository list
   {
   "pulp_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/",
   "pulp_created": "2023-09-13T07:28:28.587741Z",
   "versions_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/",
   "pulp_labels": {},
   "latest_version_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/1/",
   "name": "test",
   "description": null,
   "retain_repo_versions": null,
   "remote": null,
   "compute_delta": true
   }

2. Pull commit content from pulp to local repo.
   [root@yih-rr93 ~]# mkdir -p /var/www/html/repo
   [root@yih-rr93 ~]# ostree --repo=/var/www/html/repo init --mode=archive
   [root@yih-rr93 ~]# ostree --repo=/var/www/html/repo remote add --no-gpg-verify edge-pulp http://localhost:9090/pulp/content/test/
   [root@yih-rr93 ~]# ostree --repo=/var/www/html/repo pull --mirror edge-pulp rhel/9/x86_64/edge
   3344 metadata, 25967 content objects fetched; 839709 KiB transferred in 484 seconds; 1.4 GB content written

3. Check hash of commit in pulp and local repo, they are same.
   [root@yih-rr93 ~]# curl http://localhost/repo/refs/heads/rhel/9/x86_64/edge
   37b860943b1b39f1cbade7e0f71016c8e64b34c96510d0b73981a567ac4514d2
   [root@yih-rr93 ~]# curl -sf http://192.168.100.1:9090/pulp/content/test/refs/heads/rhel/9/x86_64/edge
   37b860943b1b39f1cbade7e0f71016c8e64b34c96510d0b73981a567ac4514d2

4. Build a child commit and upload to pulp.
   [root@yih-rr93 ~]# pulp ostree repository list
   {
   "pulp_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/",
   "pulp_created": "2023-09-13T07:28:28.587741Z",
   "versions_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/",
   "pulp_labels": {},
   "latest_version_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/2/",
   "name": "test",
   "description": null,
   "retain_repo_versions": null,
   "remote": null,
   "compute_delta": true
   }

5. Check hash of commit in pulp, the hash of commit in pulp has changed to child commit hash.
   [root@yih-rr93 ~]# curl -sf http://192.168.100.1:9090/pulp/content/test/refs/heads/rhel/9/x86_64/edge
   d8f34de49f162c956c9261f69dadfba036d1490928b5719b0b539f34d05d07cf

6. Pull child commit content from pulp to local production repo, but failed. The command seems cannot found the child commit, nothing new was pulled from pulp.
   [root@yih-rr93 ~]# ostree --repo=/var/www/html/repo pull --mirror edge-pulp rhel/9/x86_64/edge
   1 metadata, 0 content objects fetched; 14 B transferred in 0 seconds; 0 bytes content written

7. Check hash of commit in pulp and local repo, it is still the parent commit hash.
   [root@yih-rr93 ~]# curl http://localhost/repo/refs/heads/rhel/9/x86_64/edge
   37b860943b1b39f1cbade7e0f71016c8e64b34c96510d0b73981a567ac4514d2

Expected behavior
Should be able to pull the latest commit content (which is child commit) from pulp

Additional context
Add any other context about the problem here. Please provide links to any previous discussions via Discourse or Bugzilla.

[lubosmj]

Can you ensure that the commit was not uploaded to Pulp and is not accessible at http://localhost/repo/objects/37/b860943b1b39f1cbade7e0f71016c8e64b34c96510d0b73981a567ac4514d2? Is there a chance that the commit is already present in the local repository?

[yih-redhat]

Can you ensure that the commit was not uploaded to Pulp and is not accessible at http://localhost/repo/objects/37/b860943b1b39f1cbade7e0f71016c8e64b34c96510d0b73981a567ac4514d2?

I don't understand 'the commit was not uploaded to Pulp and is not accessible', but I tried in my env and I found the parent commit in local repo:
[root@yih-rr93 37]# pwd
/var/www/html/repo/objects/37
[root@yih-rr93 37]# ls | grep b8609
b860943b1b39f1cbade7e0f71016c8e64b34c96510d0b73981a567ac4514d2.commit

And I tried to find child commit, but didn't find it.
[root@yih-rr93 d8]# pwd
/var/www/html/repo/objects/d8
[root@yih-rr93 d8]# ls |grep f34de

Is there a chance that the commit is already present in the local repository?

This is the first time we use pulp to serve the commit content. We usually use http server or container to serve parent commit, and then pull it to local repo, then we serve child commit again and pull it to local repo.
And in this case, I am sure the child commit is not in local repo, because:

1. when run 'ostree pull' to pull child commit, it shows "0 bytes content written"
2. the hash of local repo is still parent hash.
   [root@yih-rr93 ~]# curl localhost/repo/refs/heads/rhel/9/x86_64/edge
   37b860943b1b39f1cbade7e0f71016c8e64b34c96510d0b73981a567ac4514d2

[lubosmj]

By "the commit was not uploaded to Pulp and is not accessible" I meant to verify whether the commit was uploaded to Pulp or not.

http GET http://localhost/pulp/api/v3/content/ostree/commits/?repository_version={REPO_VERSION_HREF_AFTER_UPLOAD}

Furthermore, if it was uploaded, can we verify whether it is accessible from the distribution?

http GET http://pulpserver/repo/objects/{commit_sha256}

And I tried to find child commit, but didn't find it.

Now, I am in doubt. 😆 Are you saying that the commit does not even exist locally?

[yih-redhat]

Hi, following are my steps: (to make sure we are on the same page)

Reproduce steps:

1. Setup a local pulp server (I used one container, using port 9090, IP can be localhost or 192.168.100.1)
2. Customer build a parent commit image, upload it to pulp server(it's version 1).
3. Setup a local ostree repo as PRODUCTION repo, enable httpd service:
   mkdir -p /var/www/html/repo
   ostree --repo=/var/www/html/repo init --mode=archive
   ostree --repo="$PROD_REPO" remote add --no-gpg-verify edge-pulp http://localhost:9090/pulp/content/test/
4. In PRODUCTION repo, pull the parent commit content from pulp server:
   ostree --repo="$PROD_REPO" pull --mirror edge-pulp "$OSTREE_REF"
5. Customer use PRODUCTION repo to provision vm.
6. Customer build a child commit based on the parent commit, upload it to pulp (now it's version 2).
7. In PRODUCTION repo, pull the child commit content from pulp with same cmd:
   ostree --repo="$PROD_REPO" pull --mirror edge-pulp "$OSTREE_REF"
8. Customer use PRODUCTION repo to upgrade vm.

Actual result:

1. At step 4, I can pull commit content from pulp, it's about 1.7G.
   Check the commit hash of pulp and local PRODUCTION repo, they are same, which means I have pulled commit from pulp to local repo.
   [root@yih-rr93 ~]# curl -sf http://localhost:9090/pulp/content/test/refs/heads/rhel/9/x86_64/edge
   37b860943b1b39f1cbade7e0f71016c8e64b34c96510d0b73981a567ac4514d2
   [root@yih-rr93 ~]# curl http://localhost/repo/refs/heads/rhel/9/x86_64/edge
   37b860943b1b39f1cbade7e0f71016c8e64b34c96510d0b73981a567ac4514d2

2. At step 7, I pull nothing from pulp, it shows "0 bytes content written"
   Check the commit hash of pulp and local PRODUCTION repo, the pulp hash has updated to child commit, but local repo remains as parent, which means I didn't pull anything from pulp.
   [root@yih-rr93 ~]# curl -sf http://localhost:9090/pulp/content/test/refs/heads/rhel/9/x86_64/edge
   d8f34de49f162c956c9261f69dadfba036d1490928b5719b0b539f34d05d07cf
   [root@yih-rr93 ~]# curl http://localhost/repo/refs/heads/rhel/9/x86_64/edge
   37b860943b1b39f1cbade7e0f71016c8e64b34c96510d0b73981a567ac4514d2

Expected result:
I can pull child commit (which is version 2) from pulp.

[yih-redhat]

By "the commit was not uploaded to Pulp and is not accessible" I meant to verify whether the commit was uploaded to Pulp or not.

http GET http://localhost/pulp/api/v3/content/ostree/commits/?repository_version={REPO_VERSION_HREF_AFTER_UPLOAD}

Furthermore, if it was uploaded, can we verify whether it is accessible from the distribution?

http GET http://pulpserver/repo/objects/{commit_sha256}

And I tried to find child commit, but didn't find it.

Now, I am in doubt. 😆 Are you saying that the commit does not even exist locally?

Following is my pulp repository info:

[root@yih-rr93 ~]# pulp ostree repository list
{
"pulp_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/",
"pulp_created": "2023-09-13T07:28:28.587741Z",
"versions_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/",
"pulp_labels": {},
"latest_version_href": "/pulp/api/v3/repositories/ostree/ostree/018a8d71-03ab-7e7c-b0f4-f81804d0915c/versions/2/",
"name": "test",
"description": null,
"retain_repo_versions": null,
"remote": null,
"compute_delta": true
}

Can you please let me know what value should be used for {REPO_VERSION_HREF_AFTER_UPLOAD}?
curl -sf -X GET http://localhost:9090/pulp/api/v3/content/ostree/commits/?repository_version={REPO_VERSION_HREF_AFTER_UPLOAD}

[lubosmj]

I could reproduce the issue. The problem could lie in using the import-all facility for uploading new commits to the existing repository.

#!/usr/bin/env bash

set -euo pipefail

workdir=$(mktemp -d)
cd "${workdir}"

# prepare pulp
pulp ostree repository create --name test
pulp ostree distribution create --name test --repository test --base-path test

# first commit
mkdir "${workdir}/first"
cd "${workdir}/first"
ostree --repo="${workdir}/first/repo" init --mode=archive
mkdir "${workdir}/first/files"
echo "one" > files/file.txt
commit=$(ostree commit --repo "${workdir}/first/repo" --branch ostree-main "${workdir}/first/files/")

cd "${workdir}/first"
tar czvf repo.tar "repo/"

# first upload
pulp ostree repository import-all --name test --file repo.tar --repository_name repo

cd "${workdir}"
mkdir "${workdir}/rremote"
ostree --repo=rremote init
ostree --repo=rremote remote add --no-gpg-verify iot http://localhost:5001/pulp/content/test/
ostree --repo=rremote remote refs iot
ostree --repo=rremote remote summary iot
ostree --repo=rremote pull -v iot ostree-main
echo "first commit tree"
tree "${workdir}/rremote"

# second commit
mkdir "${workdir}/second"
cd "${workdir}/second"
mkdir files
echo "two" > files/file2.txt
ostree --repo="${workdir}/second/repo" init --mode=archive
ostree commit --repo repo --branch ostree-main files/ --parent="${commit}"
tar czvf repo.tar repo/

# second upload
echo "Uploading and importing second repo"
pulp ostree repository import-commits --name test --file repo.tar --repository_name repo --ref ostree-main # HERE: when you replace import-commits by import-all, the workflow seems to fail and the new commit is not visible to the ostree client.

# local remote repo
cd "${workdir}"
ostree --repo=rremote remote refs iot
ostree --repo=rremote remote summary iot
ostree --repo=rremote pull -v iot ostree-main
echo "second commit tree"
tree "${workdir}/rremote"

[lubosmj]

The problem is that the summary file is not correctly updated. Commits and other objects are uploaded to Pulp with no issues.

[lmjachky@localhost test]$ ostree --repo=rremote remote add --no-gpg-verify iot http://localhost:5001/pulp/content/test/
[lmjachky@localhost test]$ ostree --repo=rremote remote refs iot
iot:ostree-main
[lmjachky@localhost test]$ ostree --repo=rremote remote summary iot
* ostree-main
    Latest Commit (126 bytes):
      f1e73ca3b0aaa60ba664983236d9f0029e698c108714fd29c8f3a2c3f358caed
    Timestamp (ostree.commit.timestamp): 2023-09-17T20:12:42+02

Last-Modified (ostree.summary.last-modified): 2023-09-17T20:12:46+02
Repository Mode (ostree.summary.mode): archive-z2
Has Tombstone Commits (ostree.summary.tombstone-commits): No
ostree.summary.indexed-deltas: true
[lmjachky@localhost test]$ http :5001/pulp/content/test/refs/heads/ostree-main
HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: keep-alive
Content-Length: 65
Content-Type: application/octet-stream
Date: Sun, 17 Sep 2023 18:16:20 GMT
Etag: "1785c234c06f0d72-41"
Last-Modified: Sun, 17 Sep 2023 18:12:47 GMT
Server: nginx/1.22.1

4b88c924220bff2c68b4f7f978e5782a3b838b724c2a784723404dee14d63199

[lubosmj]

Can you verify if the attached patch resolved the issue, please: #291?

[yih-redhat]

Can you verify if the attached patch resolved the issue, please: #291?

Sure, working on it now.