test(e2e): add E2E testing utilities and sample specs #639

qweeah · 2022-10-11T05:43:23Z

This PR includes below changes in the E2E utility package:

Enables basic auth in test infra
Supports excuting oras or other binaries with fluent-styled runtime configuring and result matching
Introduces match packages to validate execution results
Added sample test specs for manifest fetch command and common OCI image scenario.

Resolves #554, resolves #555, resolves #556, resolves #565

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT · 2022-11-08T07:06:09Z

.github/workflows/build.yml

          rm -rf $mnt_root/docker
          for layer in $(ls $mnt_root/*.tar.gz); do
            tar -xvzf $layer -C $mnt_root
          done
          trap 'docker kill oras-e2e || true' ERR
-          docker run -d -p 5000:5000 --rm --name oras-e2e \
-            --env STORAGE_DELETE_ENABLED=true \
+          docker run --pull always -d -p 5000:5000 --rm --name oras-e2e \


Do we need to --pull always?

The default behavior is missing, which can be wrong if github provided an uncleaned VM. always is more defensive.

shizhMSFT · 2022-11-08T07:07:52Z

.github/workflows/build.yml

            --mount type=bind,source=$mnt_root/docker,target=/opt/data/registry-root-dir/docker \
+            --mount type=bind,source=$mnt_root/passwd_bcrypt,target=/etc/docker/registry/passwd \


Can we generate passwd file on the fly? I have concerns that security checkers will scan the repo and find we are storing credentials.

It's doable but will complicate the setup. Can we set this aside and consider the design in another issue #680 with configurable username/password/token?

test/e2e/README.md

test/e2e/internal/utils/exec.go

test/e2e/internal/utils/match/keywords.go

test/e2e/internal/utils/match/matchable.go

test/e2e/internal/utils/match/status.go

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

test/e2e/internal/utils/exec.go

test/e2e/internal/utils/match/status.go

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT

LGTM

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT

LGTM

This PR includes below changes in the E2E utility package: 1. Enables basic auth in test infra 2. Supports excuting oras or other binaries with fluent-styled runtime configuring and result matching 3. Introduces `match` packages to validate execution results 4. Added sample test specs for `manifest fetch` command and common OCI image scenario. Resolves oras-project#554, resolves oras-project#555, resolves oras-project#556, resolves oras-project#565 Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah added 30 commits September 14, 2022 07:29

test(e2e): add ginkgo and start distribution

7844882

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

add a spec

f76783b

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

rename package

978219a

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

add _test suffix

72dd79d

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

correct package name

dccb967

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

clean module file

dc2c8d6

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

ignore certain dep

8876a01

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

Merge remote-tracking branch 'origin_src/main' into e2e-infra

1b86ea3

Merge remote-tracking branch 'origin_src/main' into e2e-infra

e9aa7ad

rename util package

b3c5206

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

Merge remote-tracking branch 'origin_src/main' into e2e-infra

8fe7427

remove test delay

66661ab

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

temp: add utility code in

88d2623

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

refactor with exec option

463cc60

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

fix write error

4cd813f

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

fix scenario tests

bbf3fc0

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

add mount files for test initialization

3f32cd1

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

revert mount file

c2dd82a

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

add mount tarball

14c0e64

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

add document for mounting data

d6007c0

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

omit decompress logs

e3a0305

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

Merge branch 'e2e-infra' into e2e-utility

41e9e26

fix git ignore

8ab1b59

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

Merge branch 'e2e-infra' into e2e-utility

87d0d53

use registry auth config for logout testing

635215b

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

fix: use cred config file when logout

0a0ef4a

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

Merge branch 'main' of https://github.com/oras-project/oras into main

83f02a0

Merge branch 'fix-logout' into e2e-utility

0c2bd61

Merge remote-tracking branch 'origin_src/main' into e2e-infra

479ca54

temp: refactoring state machine

e6b1ba6

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah added 2 commits October 24, 2022 19:15

add manifest fetch to file and desc to stdout

f439e1b

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

more strict logout order

a0d7559

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

FeynmanZhou requested a review from shizhMSFT November 1, 2022 08:09

qweeah added 11 commits November 3, 2022 15:08

update test file structure

6f650cc

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

fix init copying

1a4cb3a

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

Merge remote-tracking branch 'origin_src/main' into e2e-utility

a2dc628

create dir only

6c30fd3

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

run auth before suite

2255a02

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

check readme in

47ee168

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

Merge remote-tracking branch 'origin_src/main' into e2e-utility

eba0431

add auth before testing

fcc3828

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

logout before login

a2f9ff5

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

Merge remote-tracking branch 'origin_src/main' into e2e-utility

e02cc86

add doc & remove unused file

98f5e03

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT reviewed Nov 8, 2022

View reviewed changes

resolve comments

9ba0a17

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

sajayantony mentioned this pull request Nov 8, 2022

Add OCI Conformance validation with ORAS #257

Closed

qweeah added 5 commits November 9, 2022 10:55

try string type

456a3ba

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

rename matcher

eea1fb3

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

add newline

ed9af5b

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

code clean

c0a09bb

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

use unit32 instead

1b0fa11

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT reviewed Nov 9, 2022

View reviewed changes

test/e2e/internal/utils/exec.go Outdated Show resolved Hide resolved

test/e2e/internal/utils/match/status.go Outdated Show resolved Hide resolved

qweeah added 2 commits November 9, 2022 16:16

use uint ptr

0d24bac

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

rename status

29c24b3

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT approved these changes Nov 9, 2022

View reviewed changes

add limitation in readme

19be1f8

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT approved these changes Nov 9, 2022

View reviewed changes

shizhMSFT merged commit 2d65e5c into oras-project:main Nov 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(e2e): add E2E testing utilities and sample specs #639

test(e2e): add E2E testing utilities and sample specs #639

qweeah commented Oct 11, 2022 •

edited

shizhMSFT Nov 8, 2022

qweeah Nov 8, 2022

shizhMSFT Nov 8, 2022

qweeah Nov 8, 2022 •

edited

shizhMSFT left a comment

shizhMSFT left a comment

		--mount type=bind,source=$mnt_root/docker,target=/opt/data/registry-root-dir/docker \
		--mount type=bind,source=$mnt_root/passwd_bcrypt,target=/etc/docker/registry/passwd \

test(e2e): add E2E testing utilities and sample specs #639

test(e2e): add E2E testing utilities and sample specs #639

Conversation

qweeah commented Oct 11, 2022 • edited

shizhMSFT Nov 8, 2022

Choose a reason for hiding this comment

qweeah Nov 8, 2022

Choose a reason for hiding this comment

shizhMSFT Nov 8, 2022

Choose a reason for hiding this comment

qweeah Nov 8, 2022 • edited

Choose a reason for hiding this comment

shizhMSFT left a comment

Choose a reason for hiding this comment

shizhMSFT left a comment

Choose a reason for hiding this comment

qweeah commented Oct 11, 2022 •

edited

qweeah Nov 8, 2022 •

edited