feat: index referrers when pushing manifests by wangxiaoxuan273 · Pull Request #35 · oras-project/distribution

wangxiaoxuan273 · 2022-11-11T07:52:52Z

Part 6 of #21

This pr implements indexing referrers during manifest push. This is needed to enable the referrers API. It changes the registry/storage package. Based on the existing implementation on the rc2 branch.

Signed-off-by: wangxiaoxuan273 wangxiaoxuan119@gmail.com

Signed-off-by: wangxiaoxuan273 <wangxiaoxuan119@gmail.com>

wangxiaoxuan273 · 2022-11-11T07:56:11Z

registry/storage/ociartifactmanifesthandler.go

+
+// TODO: Should be removed and defined instead in paths package
+// Requires paths package to be exported
+func referrersLinkPath(name string) string {


Need advice on how to handle these. Move them to the path package?

@wangxiaoxuan273 Yes, please move them to the path package and update the related documentation.

codecov-commenter · 2022-11-11T07:58:57Z

Codecov Report

Merging #35 (9400f77) into main (6f0e3ea) will decrease coverage by 0.00%.
The diff coverage is 58.82%.

@@            Coverage Diff             @@
##             main      #35      +/-   ##
==========================================
- Coverage   57.18%   57.18%   -0.01%     
==========================================
  Files         107      107              
  Lines       10990    11021      +31     
==========================================
+ Hits         6285     6302      +17     
- Misses       4001     4011      +10     
- Partials      704      708       +4

Impacted Files	Coverage Δ
registry/storage/paths.go	`70.43% <50.00%> (-1.67%)`	⬇️
registry/storage/ociartifactmanifesthandler.go	`57.83% <56.25%> (-0.38%)`	⬇️
registry/storage/registry.go	`88.99% <100.00%> (+0.05%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

shizhMSFT · 2022-11-14T12:15:31Z

registry/storage/ociartifactmanifesthandler.go

+	rootPath := path.Join(referrersLinkPath(ms.repository.Named().Name()), subjectRevision.Algorithm().String(), subjectRevision.Hex())
+	referenceLinkPath := path.Join(rootPath, revision.Algorithm().String(), revision.Hex(), "link")


What would be proper path considering referrers is a first-class item like manifests, blobs?

How would it be impacted if we delete a repository?

Also think about how artifact type can be supported.

Hint: Hashing will be useful for user input projection.

Currently, only PUT path is considered. How about the DELETE path in terms of indexing referrers?

After offline discussion, I will address these issues in later prs. Issues #36, #37, #38 are created to keep track of them.

Signed-off-by: wangxiaoxuan273 <wangxiaoxuan119@gmail.com>

wangxiaoxuan273 · 2022-11-16T04:56:13Z

registry/storage/paths.go

 //
 // The path layout in the storage backend is roughly as follows:
 //
-//		<root>/v2


All these formatting changes are auto-generated by the go formatter when saving the file.

Signed-off-by: wangxiaoxuan273 <wangxiaoxuan119@gmail.com>

shizhMSFT · 2022-11-16T14:50:22Z

registry/storage/paths.go

+//		    Referrers:
 //
-//	blobsPathSpec:                  <root>/v2/blobs/
-// 	blobPathSpec:                   <root>/v2/blobs/<algorithm>/<first two hex bytes of digest>/<hex digest>
-// 	blobDataPathSpec:               <root>/v2/blobs/<algorithm>/<first two hex bytes of digest>/<hex digest>/data
-// 	blobMediaTypePathSpec:               <root>/v2/blobs/<algorithm>/<first two hex bytes of digest>/<hex digest>/data
+//	        referrersLinkPathSpec:          <root>/v2/repositories/<name>/_referrers/<algorithm>/<hex digest>/<subject algorithm>/<subject hex digest>/link


indentation issue.

Fixed the indentation.

shizhMSFT · 2022-11-16T14:50:40Z

registry/storage/paths.go

-// 						hashstates/<algorithm>/<offset>
-//			-> blob/<algorithm>
-//				<split directory content addressable storage>
+//	<root>/v2


referrer related paths should also be here.

Please check if line 43-45 are correct.

shizhMSFT · 2022-11-16T14:52:18Z

registry/storage/paths.go

+		if err != nil {
+			return "", err
+		}
+		return path.Join(append(append(append(append(repoPrefix, v.name, "_referrers", "subjects"), revisionComponents...), subjectComponents...), "link")...), nil


4 x append do not seem right. Could you rewrite it to multiple lines?

Rewrote it.

shizhMSFT · 2022-11-16T14:52:51Z

registry/storage/paths.go

 // We cover the path formats implemented by this path mapper below.
 //
-//	Manifests:
+//			Manifests:


Should it just one tab?

Suggested change

// Manifests:

// Manifests:

Changed with one tab.

shizhMSFT · 2022-11-16T14:53:42Z

registry/storage/ociartifactmanifesthandler.go

+	if err := ms.storageDriver.PutContent(ctx, referrersLinkPath, []byte(revision.String())); err != nil {
+		return err
+	}
+	return nil


nit:

Suggested change

if err := ms.storageDriver.PutContent(ctx, referrersLinkPath, []byte(revision.String())); err != nil {

return err

}

return nil

return ms.storageDriver.PutContent(ctx, referrersLinkPath, []byte(revision.String()))

changed accordingly.

Signed-off-by: wangxiaoxuan273 <wangxiaoxuan119@gmail.com>

shizhMSFT

LGTM

feat: index referrers when pushing manifests

2a394e9

Signed-off-by: wangxiaoxuan273 <wangxiaoxuan119@gmail.com>

wangxiaoxuan273 commented Nov 11, 2022

View reviewed changes

wangxiaoxuan273 mentioned this pull request Nov 11, 2022

Add OCI Artifact Support #21

Open

shizhMSFT requested changes Nov 14, 2022

View reviewed changes

moved path generation to path.go

5fdba45

Signed-off-by: wangxiaoxuan273 <wangxiaoxuan119@gmail.com>

wangxiaoxuan273 commented Nov 16, 2022

View reviewed changes

wangxiaoxuan273 requested a review from shizhMSFT November 16, 2022 04:57

fixed error in path generation

4838e99

Signed-off-by: wangxiaoxuan273 <wangxiaoxuan119@gmail.com>

shizhMSFT requested changes Nov 16, 2022

View reviewed changes

wangxiaoxuan273 added 2 commits November 17, 2022 04:43

resolved all comments except referrer storage doc

9400f77

Signed-off-by: wangxiaoxuan273 <wangxiaoxuan119@gmail.com>

updated referrer path doc

b71a541

Signed-off-by: wangxiaoxuan273 <wangxiaoxuan119@gmail.com>

wangxiaoxuan273 requested a review from shizhMSFT November 17, 2022 05:00

shizhMSFT approved these changes Nov 17, 2022

View reviewed changes

shizhMSFT merged commit 7c54a46 into oras-project:main Nov 17, 2022

wangxiaoxuan273 deleted the index-referrers branch November 17, 2022 11:22

		rootPath := path.Join(referrersLinkPath(ms.repository.Named().Name()), subjectRevision.Algorithm().String(), subjectRevision.Hex())
		referenceLinkPath := path.Join(rootPath, revision.Algorithm().String(), revision.Hex(), "link")

Conversation

wangxiaoxuan273 commented Nov 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangxiaoxuan273 Nov 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Nov 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangxiaoxuan273 Nov 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shizhMSFT left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wangxiaoxuan273 commented Nov 11, 2022 •

edited

Loading

wangxiaoxuan273 Nov 11, 2022 •

edited

Loading

codecov-commenter commented Nov 11, 2022 •

edited

Loading

wangxiaoxuan273 Nov 15, 2022 •

edited

Loading