backend: support s3 backend

[loheagn]

Signed-off-by: Nan Li loheagn@icloud.com

[loheagn]

update dragonflyoss/nydus#831

[codecov-commenter]

Codecov Report

Base: 33.88% // Head: 31.48% // Decreases project coverage by -2.39% ⚠️

Coverage data is based on head (9038ab8) compared to base (3d0e02d).
Patch coverage: 12.63% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #277      +/-   ##
==========================================
- Coverage   33.88%   31.48%   -2.40%     
==========================================
  Files          30       34       +4     
  Lines        3223     3506     +283     
==========================================
+ Hits         1092     1104      +12     
- Misses       2019     2286     +267     
- Partials      112      116       +4     

Impacted Files	Coverage Δ
pkg/backend/backend.go	0.00% <0.00%> (ø)
pkg/backend/oss.go	0.00% <0.00%> (ø)
pkg/backend/s3.go	13.04% <13.04%> (ø)
pkg/backend/localfs.go	0.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[loheagn]

Hi, the CI seems to be blocked by something has nothing to do with this PR. Can anyone please help to rerun the failed job? Thanks :)

[changweige]

How should we test it?

[loheagn]

How should we test it?

I can add a new smoke test to this patch. The test will setup a minio server and execute image conversion using the minio server as s3 backend.

[changweige]

How should we test it?

I can add a new smoke test to this patch. The test will setup a minio server and execute image conversion using the minio server as s3 backend.

It will be great to have such a test. Thanks

[imeoer]

LGTM! Thanks @loheagn

[loheagn]

@changweige @imeoer Hi, I added a new smoke test to this patch to test the s3 backend. You can review the code again :)

[imeoer]

@loheagn It seems a data race be detected in https://github.com/containerd/nydus-snapshotter/actions/runs/3695539855/jobs/6258056733. :)

[loheagn]

@loheagn It seems a data race be detected in https://github.com/containerd/nydus-snapshotter/actions/runs/3695539855/jobs/6258056733. :)

@imeoer Thanks for pointing the error! Tha data race was fixed.

[changweige]

Very nice test, thanks