ORAS is actually S3 Compatible*

[WesselAtWork]

* If you consider the level of shenanigans "Compatible"

Everything Is A File

One of the great Unix philosophies is the concept of making everything (modems, keyboards, printers, etc.) a file that is accessible to the user as a file-descriptor. This usually has the benefit that you can plug things into programs without writing special integration code.

• Generally programs write and read files.
• Making everything a file means that you can generally interface with every program.

Oras is kinda-almost EIAF compatible

oras has a --oci-layout

Instead of talking to an upstream service over http/https, oras can create the artefact structure and data as files and directories.

Most, if not every operation you can perform on an upstream registry you can perform on a local file-based repo.

s3 is kinda-almost EIAF compatible

Did you know you can mount s3 buckets in your filesystem!

I didn't either but there is this cool project called s3fs that allows you to do just that!

ohno

so what if you then mount your bucket on a folder and then do "--oci-layout" on that folder?

echo "Hello World!" > hello.txt
echo "Goodbye World!" > goodbye.txt
mkdir s3
s3fs my-bucket s3 -o profile=MY_AWS_PROFILE
oras push --oci-layout s3/hello:latest hello.txt:application/vnd.test.text
oras push --oci-layout s3/goodbye:latest goodbye.txt:application/vnd.test.text

rm hello.txt
rm goodbye.txt
oras pull --oci-layout s3/hello:latest
oras pull --oci-layout s3/goodbye:latest
cat hello.txt
cat goodbye.txt

There we go!
You are no running your oci """registry""" on one of the cheapest storage platforms available!

Performance

Local

I've found that running locally on my computer is a little slow in comparison to running with s3 directly.
If this is a requirement to run outside of AWS you might want to look at s3ql, goofys or another s3-mounter alternative that is more performant.

time oras pull --oci-layout s3/junk:5MiB
> Downloading 0a0941a1970d 5Mi.junk
> Downloaded  0a0941a1970d 5Mi.junk
> Pulled [oci-layout] s3/junk:5MiB
> Digest: sha256:a9f9a2cfc5d8efc6d66993e358a9e09fd4b71abe1473a34628092dec2837dc4d

> real    0m25.499s
> user    0m0.024s
> sys     0m0.024s

time aws s3 cp s3://my-bucket/junk/blobs/sha256/0a0941a1970d...  .

> real    0m1.978s
> user    0m0.696s
> sys     0m0.111s

AWS

Running inside of aws is blisteringly fast!

The mount option to use the EC2 instance profile is -o iam_role

time oras pull --oci-layout s3/junk:5MiB
> Downloading 0a0941a1970d 5Mi.junk
> Downloaded  0a0941a1970d 5Mi.junk
> Pulled [oci-layout] s3/junk:5MiB
> Digest: sha256:a9f9a2cfc5d8efc6d66993e358a9e09fd4b71abe1473a34628092dec2837dc4d

> real    0m1.065s
> user    0m0.021s
> sys     0m0.028s

time aws s3 cp s3://my-bucket/junk/blobs/sha256/0a0941a1970d...  .

> real    0m1.133s
> user    0m0.811s
> sys     0m0.139s

What I observed was that the s3 command was consistently slower than oras.
Apparently: the overhead is s3 mount + oras parse < cold s3 connection setup

Unsure of the performance on bigger files but I assume it does the multi-part download efficiently.

Note: AL2023 removed the EPEL repo so you need to run something else, I ran this on Debian for this test.

Caveats

Sadly there are somethings you can't do. The most important of which YOU CAN'T DELETE

• NO DELETE!
  oras manifest delete and oras blob delete DO NOT WORK ON OCI-LAYOUTS!
  You can easily delete the ENTIRE repo (e.i. s3/hello) but that's probably not what you want to do.
• oras repo ls does not work because it does not have a --oci-layout option.

It's kinda funny to me that the only thing holding this back is not actually the technology but instead the fact you can't delete anything efficiently.

[TerryHowe]

Thanks! Interesting write up.

[qweeah]

It's cool to start a lightweight registry without extra computing resources. @WesselAtWork Thanks for the innovative writing.

[WesselAtWork]

I did some more size/speed tests.

Single

64MiB

time oras pull --oci-layout s3/junk:64MiB
> Downloading 8c6eff957de6 64MiB.junk
> Downloaded  8c6eff957de6 64MiB.junk
> Pulled [oci-layout] s3/junk:64MiB

> real    0m1.604s
> user    0m0.287s
> sys     0m0.146s

time aws s3 cp s3://my-bucket/junk/blobs/sha256/8c6eff957de6... .

> real    0m1.719s
> user    0m1.635s
> sys     0m0.545s

256MiB

time oras pull --oci-layout s3/junk:256MiB
> Downloading e509473da713 256MiB.junk
> Downloaded  e509473da713 256MiB.junk
> Pulled [oci-layout] s3/junk:256MiB

> real    0m3.502s
> user    0m1.054s
> sys     0m0.465s

time aws s3 cp s3://my-bucket/junk/blobs/sha256/e509473da713... .

> real    0m3.308s
> user    0m2.663s
> sys     0m1.675s

512MiB

time oras pull --oci-layout s3/junk:512MiB
> Downloading a190e8422520 512MiB.junk
> Downloaded  a190e8422520 512MiB.junk
> Pulled [oci-layout] s3/junk:512MiB

> real    0m5.775s
> user    0m1.927s
> sys     0m0.749s

time aws s3 cp s3://my-bucket/junk/blobs/sha256/a190e8422520... .

> real    0m4.750s
> user    0m3.667s
> sys     0m2.751s

1GiB

time oras pull --oci-layout s3/junk:1GiB
> Downloading 8599aa97e60f 1GiB.junk
> Downloaded  8599aa97e60f 1GiB.junk
> Pulled [oci-layout] s3/junk:1GiB

> real    0m11.907s
> user    0m4.111s
> sys     0m1.850s

time aws s3 cp s3://my-bucket/junk/blobs/sha256/8599aa97e60f... .

> real    0m8.653s
> user    0m5.426s
> sys     0m5.995s

2GiB

time oras pull --oci-layout s3/junk:2GiB
Downloading 9d209eb0b797 2GiB.junk
Downloaded  9d209eb0b797 2GiB.junk
Pulled [oci-layout] s3/junk:2GiB

> real    0m25.710s
> user    0m8.059s
> sys     0m3.739s

time aws s3 cp s3://my-bucket/junk/blobs/sha256/9d209eb0b797... .

> real    0m14.433s
> user    0m9.261s
> sys     0m10.331s

---

Split

• The junk repo was deleted between theses tests
• Each 64MiB junk file contains unique data

Split 1GiB

time oras pull --oci-layout s3/junk:s1GiB
> Downloading 5f75fe70dcb5 64MiB.junk.1
> Downloading 46638039938e 64MiB.junk.2
> Downloading 8d54481e8115 64MiB.junk.3
> Downloaded  46638039938e 64MiB.junk.2
> Downloading c66ab1cdd33d 64MiB.junk.4
> Downloaded  8d54481e8115 64MiB.junk.3
> Downloading 406bb28b28ec 64MiB.junk.5
> Downloaded  5f75fe70dcb5 64MiB.junk.1
> Downloading de3a3d92ac93 64MiB.junk.6
> Downloaded  de3a3d92ac93 64MiB.junk.6
> Downloading 39d141565786 64MiB.junk.7
> Downloaded  c66ab1cdd33d 64MiB.junk.4
> Downloading 386dfc1ad6d4 64MiB.junk.8
> Downloaded  406bb28b28ec 64MiB.junk.5
> Downloading 2849d4fc776d 64MiB.junk.9
> Downloaded  39d141565786 64MiB.junk.7
> Downloading c2e57b5c7a5a 64MiB.junk.10
> Downloaded  386dfc1ad6d4 64MiB.junk.8
> Downloading ab05e487f785 64MiB.junk.11
> Downloaded  2849d4fc776d 64MiB.junk.9
> Downloading 9620d5b854b4 64MiB.junk.12
> Downloaded  c2e57b5c7a5a 64MiB.junk.10
> Downloading c1d439758d7f 64MiB.junk.13
> Downloaded  ab05e487f785 64MiB.junk.11
> Downloading 3fc96ea9f70e 64MiB.junk.14
> Downloaded  9620d5b854b4 64MiB.junk.12
> Downloading 511d09cfd327 64MiB.junk.15
> Downloaded  c1d439758d7f 64MiB.junk.13
> Downloading 18cdaead1432 64MiB.junk.16
> Downloaded  3fc96ea9f70e 64MiB.junk.14
> Downloaded  511d09cfd327 64MiB.junk.15
> Downloaded  18cdaead1432 64MiB.junk.16
> Pulled [oci-layout] s3/junk:s1GiB

> real    0m10.980s
> user    0m5.770s
> sys     0m2.872s

time aws s3 cp --recursive s3://my-bucket/junk/blobs/sha256/ .

> real    0m8.782s
> user    0m5.663s
> sys     0m5.965s

Split 2GiB

time oras pull --oci-layout s3/junk:s2GiB
> Downloading 7afe02713078 64MiB.junk.3
> Downloading e54662bef97f 64MiB.junk.2
> Downloading 9e8c746718f9 64MiB.junk.1
> Downloaded  e54662bef97f 64MiB.junk.2
> Downloading a3ff75c2e827 64MiB.junk.4
> Downloaded  9e8c746718f9 64MiB.junk.1
> Downloading 6c5b53da4f6a 64MiB.junk.5
> Downloaded  7afe02713078 64MiB.junk.3
> Downloading 9780318adb5d 64MiB.junk.6
> Downloaded  a3ff75c2e827 64MiB.junk.4
> Downloading 5499f2bcce06 64MiB.junk.7
> Downloaded  6c5b53da4f6a 64MiB.junk.5
> Downloading 57be1010cca5 64MiB.junk.8
> Downloaded  9780318adb5d 64MiB.junk.6
> Downloading eefc7e833dcb 64MiB.junk.9
> Downloaded  57be1010cca5 64MiB.junk.8
> Downloading 47b8062cfb13 64MiB.junk.10
> Downloaded  5499f2bcce06 64MiB.junk.7
> Downloading 1189dba990ba 64MiB.junk.11
> Downloaded  eefc7e833dcb 64MiB.junk.9
> Downloading 75ef758097f2 64MiB.junk.12
> Downloaded  1189dba990ba 64MiB.junk.11
> Downloading 462e001ee7bc 64MiB.junk.13
> Downloaded  47b8062cfb13 64MiB.junk.10
> Downloading b13672b0c6ef 64MiB.junk.14
> Downloaded  75ef758097f2 64MiB.junk.12
> Downloading 7f27d3cbb75a 64MiB.junk.15
> Downloaded  462e001ee7bc 64MiB.junk.13
> Downloading 9b1d7c6b725b 64MiB.junk.16
> Downloaded  b13672b0c6ef 64MiB.junk.14
> Downloading 429365319d72 64MiB.junk.17
> Downloaded  7f27d3cbb75a 64MiB.junk.15
> Downloading 6023a0cc8af2 64MiB.junk.18
> Downloaded  9b1d7c6b725b 64MiB.junk.16
> Downloading 8c2b624dd9a9 64MiB.junk.19
> Downloaded  429365319d72 64MiB.junk.17
> Downloading 88ea95da635a 64MiB.junk.20
> Downloaded  6023a0cc8af2 64MiB.junk.18
> Downloading 43c83945403f 64MiB.junk.21
> Downloaded  8c2b624dd9a9 64MiB.junk.19
> Downloading 8908a0ad4fb3 64MiB.junk.22
> Downloaded  88ea95da635a 64MiB.junk.20
> Downloading 3c33536bdd2f 64MiB.junk.23
> Downloaded  43c83945403f 64MiB.junk.21
> Downloading f2bee40da716 64MiB.junk.24
> Downloaded  3c33536bdd2f 64MiB.junk.23
> Downloading 8ab7581f34ce 64MiB.junk.25
> Downloaded  8908a0ad4fb3 64MiB.junk.22
> Downloading 8f79ad0e54fc 64MiB.junk.26
> Downloaded  f2bee40da716 64MiB.junk.24
> Downloading f8972f2a1637 64MiB.junk.27
> Downloaded  8ab7581f34ce 64MiB.junk.25
> Downloading 83f01adc8cf8 64MiB.junk.28
> Downloaded  8f79ad0e54fc 64MiB.junk.26
> Downloading ae4f1cb72036 64MiB.junk.29
> Downloaded  f8972f2a1637 64MiB.junk.27
> Downloading 1f172a198f15 64MiB.junk.30
> Downloaded  83f01adc8cf8 64MiB.junk.28
> Downloading 2117c32a877a 64MiB.junk.31
> Downloaded  ae4f1cb72036 64MiB.junk.29
> Downloading 2174a73cf2e3 64MiB.junk.32
> Downloaded  1f172a198f15 64MiB.junk.30
> Downloaded  2117c32a877a 64MiB.junk.31
> Downloaded  2174a73cf2e3 64MiB.junk.32
> Pulled [oci-layout] s3/junk:s2GiB

> real    0m25.304s
> user    0m12.055s
> sys     0m5.792s

time aws s3 cp --recursive s3://my-bucket/junk/blobs/sha256/ .

> real    0m14.679s
> user    0m9.290s
> sys     0m10.241s

---

Looks like after a certain size the speed flips to s3 being faster (somewhere around 64MiB)
2GiB becomes a little painful, but hopefully you don't have to store big blobs.

Splitting the blobs result in no speed up.
Interesting.
I thought oras downloaded multiple artefacts in parallel?
Is there specific reason it does it in serial?
It was processing the pulls in parallel (default is 3).
Tried up to --concurrency 32 but anything above --concurrency 4 resulted in slower pulls.

I am unsure if it is s3fs or oras that is the bottleneck on the single blob performance.
I am thinking it might be the s3fs, should test another s3 mount to compare.

If you are planning on using this:
Keep your artefacts bellow 1GiB > and prefer artefacts below 512MiB > IF you want to use this and performance is a key factor.

[WesselAtWork]

Tried with Goofys

Split 2GiB

time oras pull --oci-layout goofys3/junk:s2GiB
> Downloading 8883ad4b7e58 64MiB.junk.1
> Downloading 13eba181c77d 64MiB.junk.3
> Downloading 13a32d451ddd 64MiB.junk.2
> Downloaded  8883ad4b7e58 64MiB.junk.1
> Downloading 236886e517f6 64MiB.junk.4
> Downloaded  13a32d451ddd 64MiB.junk.2
> Downloading c3d7041c5849 64MiB.junk.5
> Downloaded  13eba181c77d 64MiB.junk.3
> Downloading a76e0c74e91a 64MiB.junk.6
> Downloaded  236886e517f6 64MiB.junk.4
> Downloading 84a97aa4872e 64MiB.junk.7
> Downloaded  c3d7041c5849 64MiB.junk.5
> Downloading 1833ef456134 64MiB.junk.8
> Downloaded  a76e0c74e91a 64MiB.junk.6
> Downloading 0ee53649ff78 64MiB.junk.9
> Downloaded  84a97aa4872e 64MiB.junk.7
> Downloading 21b01839a9c4 64MiB.junk.10
> Downloaded  1833ef456134 64MiB.junk.8
> Downloading e204914e9040 64MiB.junk.11
> Downloaded  0ee53649ff78 64MiB.junk.9
> Downloading f344dd9b9445 64MiB.junk.12
> Downloaded  21b01839a9c4 64MiB.junk.10
> Downloading 38a445573022 64MiB.junk.13
> Downloaded  e204914e9040 64MiB.junk.11
> Downloading 8186c7141b1e 64MiB.junk.14
> Downloaded  f344dd9b9445 64MiB.junk.12
> Downloading 559980c9c85a 64MiB.junk.15
> Downloaded  38a445573022 64MiB.junk.13
> Downloading 2748b03ff407 64MiB.junk.16
> Downloaded  8186c7141b1e 64MiB.junk.14
> Downloading 96ba42d5c027 64MiB.junk.17
> Downloaded  559980c9c85a 64MiB.junk.15
> Downloading 3ea3d3b3951b 64MiB.junk.18
> Downloaded  2748b03ff407 64MiB.junk.16
> Downloading ef75276eca28 64MiB.junk.19
> Downloaded  96ba42d5c027 64MiB.junk.17
> Downloading d6dc5ae1b9ab 64MiB.junk.20
> Downloaded  3ea3d3b3951b 64MiB.junk.18
> Downloading 203544f303f5 64MiB.junk.21
> Downloaded  ef75276eca28 64MiB.junk.19
> Downloading 10cc4ac1fb40 64MiB.junk.22
> Downloaded  d6dc5ae1b9ab 64MiB.junk.20
> Downloading 937345ce86c6 64MiB.junk.23
> Downloaded  203544f303f5 64MiB.junk.21
> Downloading 9e12d45e2e1a 64MiB.junk.24
> Downloaded  10cc4ac1fb40 64MiB.junk.22
> Downloading dbe15e6265ac 64MiB.junk.25
> Downloaded  937345ce86c6 64MiB.junk.23
> Downloading 598995f9f6c1 64MiB.junk.26
> Downloaded  9e12d45e2e1a 64MiB.junk.24
> Downloading 05fc594b8f1a 64MiB.junk.27
> Downloaded  dbe15e6265ac 64MiB.junk.25
> Downloading c0881dd2e361 64MiB.junk.28
> Downloaded  598995f9f6c1 64MiB.junk.26
> Downloading 6bc35e40e93b 64MiB.junk.29
> Downloaded  05fc594b8f1a 64MiB.junk.27
> Downloading 143e4fe7111e 64MiB.junk.30
> Downloaded  c0881dd2e361 64MiB.junk.28
> Downloading 3d1905872876 64MiB.junk.31
> Downloaded  6bc35e40e93b 64MiB.junk.29
> Downloading 9aa30d67b4b1 64MiB.junk.32
> Downloaded  143e4fe7111e 64MiB.junk.30
> Downloaded  3d1905872876 64MiB.junk.31
> Downloaded  9aa30d67b4b1 64MiB.junk.32
> Pulled [oci-layout] goofys3/junk:s2GiB

> real    0m16.112s
> user    0m9.274s
> sys     0m3.927s

2GIB

time oras pull --oci-layout goofys3/junk:2GiB
> Downloading 5990cc48fea2 2GiB.junk
> Downloaded  5990cc48fea2 2GiB.junk
> Pulled [oci-layout] goofys3/junk:2GiB

> real    0m19.355s
> user    0m7.473s
> sys     0m3.217s

---

Better. :)
If you have 1GiB < artefacts, or just value speed > convenience, consider using goofys instead of s3fs.

[shizhMSFT]

Nice write up. It is indeed a valid scenario by our design.

We have more interesting scenarios on the OCI layout you might want to have a try.

1. The OCI layout folder produced by oras can be consumed by zot directly so that it can be a real registry.
2. You can have multiple folders to simulate multiple registries with multiple repositories. Meanwhile, use shell commands to query.

Here's an example

$ oras cp docker.io/library/hello-world:linux --to-oci-layout docker.io/library/hello-world:linux
Copied [registry] docker.io/library/hello-world:linux => [oci-layout] docker.io/library/hello-world:linux
Digest: sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
$ oras cp docker.io/library/nginx:latest --to-oci-layout docker.io/library/nginx:latest
Copied [registry] docker.io/library/nginx:latest => [oci-layout] docker.io/library/nginx:latest
Digest: sha256:104c7c5c54f2685f0f46f3be607ce60da7085da3eaa5ad22d3d9f01594295e9c
$ oras cp ghcr.io/oras-project/oras:v1.0.1 --to-oci-layout ghcr.io/oras-project/oras:v1.0.1
Copied [registry] ghcr.io/oras-project/oras:v1.0.1 => [oci-layout] ghcr.io/oras-project/oras:v1.0.1
Digest: sha256:6762f2c764e216591a6b9d25b1cec70bb8183e5841d0f249f310fccb19ed2d94
$ # list registries
$ ls
docker.io  ghcr.io
$ # list repositories
$ find docker.io -name "oci-layout" -exec dirname {} \;
docker.io/library/nginx
docker.io/library/hello-world

Of course, OCI layout support in oras has some limitations.

1. No delete support for now but it is in our roadmap: oci.Store and oci.Storage should implement content.Deleter oras-go#454
2. No concurrent writing support since we don't support file locks. However, we do support concurrent reading.

[FeynmanZhou]

Nice content. @WesselAtWork would you mind submitting this article from a Discussion to the ORAS blog post?

[WesselAtWork]

I don't mind at all!
Just clarifying, do I submit a pr on the oras-www for a new blog post or is there a different way I submit this discussion for a blog post?

[FeynmanZhou]

Yes, that's correct. Thank you!
You may need to polish it a little bit to a blog post. We can also provide detailed comments and feedback on your PR.