Build match worker image #2313

0xverin · 2023-12-06T15:14:04Z

Context

resolves p-330

Test steps:

Build the worker match parachain version's image and push it to Docker Hub.
Docker pull parachain and worker image and ensure it runs successfully.
Use the release-ts-api-package gha to generate the client API types.

linear · 2023-12-06T15:14:08Z

P-330 Build the matching worker image with the release job

0xverin · 2023-12-07T07:25:47Z

After testing, the p0.9.17-9170-w0.0.1-100 release was published, and the worker v0.0.1-100 tag was also pushed, see gha. However, it appears that this worker is still compatible with the parachain latest tag. Testing has shown:

worker v0.0.1-100 + parachain v0.9.17 results in failure.
worker v0.0.1-100 + parachain latest results in success.

Did I miss anything?

BillyWooo · 2023-12-07T16:49:34Z

After testing, the p0.9.17-9170-w0.0.1-100 release was published, and the worker v0.0.1-100 tag was also pushed, see gha. However, it appears that this worker is still compatible with the parachain latest tag. Testing has shown:

worker v0.0.1-100 + parachain v0.9.17 results in failure.

worker v0.0.1-100 + parachain latest results in success.

Did I miss anything?

Did our session together solve your problem?

0xverin · 2023-12-11T06:21:22Z

Did our session together solve your problem?

Definitely helpful; it's just that there were some deviations during testing.

Kailai-Wang · 2023-12-11T16:24:50Z

Is this PR ready to be reviewed?

I see there're unfinished TODOs but at the same time WIP was removed from the title and this is not a draft PR anymore

0xverin

Overall, it looks good. Thank you for resolving the connect issue.

0xverin · 2024-01-19T10:58:02Z

.github/workflows/release-ts-api-package.yml

-          default: 'latest'
+    inputs:
+      release-tag:
+        description: "Client-api release tag (e.g. p1.2.0-9701-w0.0.1-101)"


We can't use the latest version if we change it here, right?

it won't affect the use of lates version and we still are able to set release tag as latest in order to generate types from latest docker images

0xverin · 2024-01-19T10:58:14Z

tee-worker/docker/lit-ts-api-package-build.yml

@@ -13,11 +13,13 @@ services:
            litentry-node:
                condition: service_healthy
            litentry-worker-1:
-                condition: service_healthy
+                # using +service_started+ over +service_healthy+ since worker runs successfully but can not connect to parachain


What does "3" mean?

Kailai-Wang

Sorry I'm a bit late to it - just found it was merged already

No worries, we can always make it up with subsequent changes.

Kailai-Wang · 2024-01-22T06:39:54Z

.github/workflows/create-release-draft.yml

@@ -114,8 +115,8 @@ jobs:
            ${{ matrix.chain }}-parachain-srtool-digest.json
            ${{ matrix.chain }}-parachain-runtime.compact.compressed.wasm

-  ## build docker image of parachain binary ##
-  build-docker:
+  # build docker image of parachain binary ##


Any reason to change it to single #? It used the ## ... ## style

Kailai-Wang · 2024-01-22T07:22:31Z

.github/workflows/create-release-draft.yml

@@ -127,7 +128,7 @@ jobs:

      - name: Set env
        run: |
-          DOCKER_TAG=$(echo ${{ env.RELEASE_TAG }} | cut -d'-' -f1 | sed 's/p/v/')
+          DOCKER_TAG=$(echo ${{ env.RELEASE_TAG }} | sed 's/p/v/;s/\(.*\)-w.*/\1/')


Any specific reason for this, or just a casual change?

If we change the job name from build-docker to build-parachain-docker, we should change the var to PARACHAIN_DOCKER_TAG too, for workers it should be WORKER_DOCKER_TAG - please let's keep the naming consistent.

Any specific reason for this, or just a casual change?
Yes this changes allows us to consider a variety of options for specific versions

latest

p0.9.18-w0.0.2

p0.9.18-9181-w0.0.2-101

Kailai-Wang · 2024-01-22T07:27:34Z

.github/workflows/create-release-draft.yml

+      - name: Free up disk space
+        if: startsWith(runner.name, 'GitHub Actions')
+        uses: jlumbroso/free-disk-space@main
+        with:
+          tool-cache: true
+          swap-storage: false
+          large-packages: false
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+        with:
+          # use the docker driver to access the local image
+          # we don't need external caches or multi platforms here
+          # see https://docs.docker.com/build/drivers/
+          driver: docker
+
+      - name: Cache worker-cache
+        uses: actions/cache@v3


For releases I suggest not using any cache at all

Kailai-Wang · 2024-01-22T09:22:45Z

tee-worker/docker/lit-ts-api-package-build.yml

        networks:
            - litentry-test-network
-        entrypoint:
-            "/usr/local/worker-cli/lit_ts_api_package_build.sh -p 9912 -u ws://litentry-node
+        entrypoint: "/usr/local/worker-cli/lit_ts_api_package_build.sh -p 9912 -u ws://litentry-node


Shall we keep the old format?

Kailai-Wang · 2024-01-22T09:23:51Z

tee-worker/docker/lit-ts-api-package-build.yml

@@ -13,11 +13,13 @@ services:
            litentry-node:
                condition: service_healthy
            litentry-worker-1:
-                condition: service_healthy
+                # using +service_started+ over +service_healthy+ since worker runs successfully but can not connect to parachain


What does "3" mean?

Kailai-Wang · 2024-01-22T09:27:39Z

tee-worker/docker/lit-ts-api-package-build.yml

+                # using +service_started+ over +service_healthy+ since worker runs successfully but can not connect to parachain
+                # as requires additional pre-setup for parachain image which built in production mode
+                # for generating types there is no need for fully workable interaction between worker and parachain
+                condition: service_started


Does it mean we can get the data even tho enclave is not registered on the parachain? (I assume so

yes, that's correct. I have one concern though, what if not registred enclave generate types which are different in case it was registred 🤔

Traf333 · 2024-01-22T09:54:12Z

Sorry I'm a bit late to it - just found it was merged already

No worries, we can always make it up with subsequent changes.

Opening follow up PR for that. ~~I've also observed one critical part which was commented out for testing purpose~~ nope all seems fine, just cached diff

0xverin added 11 commits December 6, 2023 16:20

add worker && cli build

7d260a6

separate docker builds

eb2b57f

add Docker Buildx

88e84b4

add caches

a1373de

free up disk space

bb277e0

remove caches

eba2b33

build local builder

cf32267

re-base worker-cache

c5be054

push docker image

cb33d19

change condition

636efd8

uncomment

a82bc50

0xverin mentioned this pull request Dec 7, 2023

Fix: Add worker tag input #2307

Merged

0xverin changed the title ~~[WIP]:Build match worker image~~ Build match worker image Dec 7, 2023

0xverin requested review from BillyWooo, Kailai-Wang and grumpygreenguy December 7, 2023 07:26

improve worker tag

9080e64

0xverin force-pushed the p-330-build-match-worker-image branch from a627359 to 9080e64 Compare December 11, 2023 11:22

0xverin added 3 commits December 11, 2023 19:23

remove hardcode

8ec2d88

sort && rename

773df6a

fix tag

a8945b0

0xverin added 4 commits December 20, 2023 16:11

echo worker tag

293b62b

test tag

0d10952

run echo

a541f37

echo images

8eddcae

Traf333 added 11 commits January 12, 2024 12:51

adjust inputs description

0651173

correct version name for draft release

a32a5e9

rolback to release version

2582d9e

include patch version to the image

cf8775f

Merge branch 'dev' into p-330-build-match-worker-image

b5e0a84

fix included patch number

cae2983

include exception for shorter version

dea5314

simplify ts release package + refactoring

9861dba

fix script typo

9c05391

check without timeout

6046b55

debug artefacts

8924412

Traf333 force-pushed the p-330-build-match-worker-image branch from 2d26ebb to 8924412 Compare January 17, 2024 15:59

Traf333 added 5 commits January 18, 2024 13:22

check with started worker

a0054ce

add comment

244221e

cleanup

ab29516

debugging

ac3df19

add meaningful comment

b86b527

0xverin commented Jan 19, 2024

View reviewed changes

uncommit publishing packages

15dddb3

BillyWooo approved these changes Jan 21, 2024

View reviewed changes

BillyWooo and others added 3 commits January 21, 2024 22:21

Merge branch 'dev' into p-330-build-match-worker-image

4d8f544

Merge branch 'dev' into p-330-build-match-worker-image

0b7f59c

Merge branch 'dev' into p-330-build-match-worker-image

6658e36

Traf333 enabled auto-merge (squash) January 22, 2024 09:17

Traf333 merged commit 3869810 into dev Jan 22, 2024
31 checks passed

Traf333 deleted the p-330-build-match-worker-image branch January 22, 2024 09:22

Kailai-Wang reviewed Jan 22, 2024

View reviewed changes

This was referenced Jan 22, 2024

Follow up changes for release draft action #2413

Merged

Worker Docker release build without using cache #2423

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build match worker image #2313

Build match worker image #2313

0xverin commented Dec 6, 2023 •

edited by Traf333

Loading

linear bot commented Dec 6, 2023

0xverin commented Dec 7, 2023 •

edited

Loading

BillyWooo commented Dec 7, 2023

0xverin commented Dec 11, 2023 •

edited

Loading

Kailai-Wang commented Dec 11, 2023 •

edited

Loading

0xverin left a comment

0xverin Jan 19, 2024 •

edited

Loading

Traf333 Jan 19, 2024

0xverin Jan 19, 2024

Kailai-Wang Jan 22, 2024

Kailai-Wang left a comment •

edited

Loading

Kailai-Wang Jan 22, 2024

Kailai-Wang Jan 22, 2024

Kailai-Wang Jan 22, 2024

Traf333 Jan 22, 2024

Kailai-Wang Jan 22, 2024

Kailai-Wang Jan 22, 2024

Kailai-Wang Jan 22, 2024

Kailai-Wang Jan 22, 2024

Traf333 Jan 22, 2024

Traf333 commented Jan 22, 2024 •

edited

Loading

Build match worker image #2313

Build match worker image #2313

Conversation

0xverin commented Dec 6, 2023 • edited by Traf333 Loading

Context

linear bot commented Dec 6, 2023

0xverin commented Dec 7, 2023 • edited Loading

BillyWooo commented Dec 7, 2023

0xverin commented Dec 11, 2023 • edited Loading

Kailai-Wang commented Dec 11, 2023 • edited Loading

0xverin left a comment

Choose a reason for hiding this comment

0xverin Jan 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kailai-Wang left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Traf333 commented Jan 22, 2024 • edited Loading

0xverin commented Dec 6, 2023 •

edited by Traf333

Loading

0xverin commented Dec 7, 2023 •

edited

Loading

0xverin commented Dec 11, 2023 •

edited

Loading

Kailai-Wang commented Dec 11, 2023 •

edited

Loading

0xverin Jan 19, 2024 •

edited

Loading

Kailai-Wang left a comment •

edited

Loading

Traf333 commented Jan 22, 2024 •

edited

Loading