Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make tee-prod build less resource intensive #2133

Merged
merged 4 commits into from
Sep 15, 2023
Merged

Make tee-prod build less resource intensive #2133

merged 4 commits into from
Sep 15, 2023

Conversation

Kailai-Wang
Copy link
Collaborator

Context

We have problems building the tee-prod docker image of parachain:
https://github.com/litentry/litentry-parachain/actions/runs/6173744153/job/16767091897

The error log is:

lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

It's most likely CPU/RAM starvation.

This PR tries to use thin LTO to reduce the resource usage.
It looks fine in the manual test: https://github.com/litentry/litentry-parachain/actions/runs/6185784593

@Kailai-Wang Kailai-Wang self-assigned this Sep 14, 2023
@Kailai-Wang Kailai-Wang requested a review from a team September 14, 2023 15:39
BillyWooo
BillyWooo approved these changes Sep 15, 2023
View reviewed changes
Copy link
Collaborator

@BillyWooo BillyWooo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. New things to learn.

@BillyWooo BillyWooo merged commit 278f948 into dev Sep 15, 2023
56 of 58 checks passed
@BillyWooo BillyWooo deleted the fix-prod-build branch September 15, 2023 08:31
@kziemianek
Copy link
Member

@Kailai-Wang is this short or long term solution ?
I think we should try to find a way to keep well optimized build.
Would increasing runner resources help?

wangminqi added a commit that referenced this pull request Sep 20, 2023
@wangminqi @Kailai-Wang
@outofxxx @cryptoade1 @felixfaisal @BillyWooo @kziemianek @dependabot
0.9.42 minqi parachain (#2147)
2e02777 
* 1847 evm support (#2016)

* feat: initial commit

* feat: Runtime EVM Support (test TODO)

* feat: add evm test scripts

* debug: add evm parameter for test

* debug: quick format fix

* debug:

* debug: taplo

* debbug: taplo

* debug: try_runtime add

* debug: try default feature

* debug: test mode compile add runtime benchmarks

* debug: modify in non-dev

* debug: update yarn.lock file

* debug: remove all docker yarn changes

* debug: recover yarn.lock

* debug: using yarn instead of yarn install

* debug: corepack yarn install to generate yarn.lock

* debug: add extra test

* debug: fix test bug

* debug: ts debug

* debug: fix bug

* debug: ts

* debug: ts

* debug: modify workflow for better log

* debug: change node impl and ts

* debug: change gas price impl

* debug: using string addres directly

* debug: adjust paramter for test.ts

* debug: add extra ts for monitor the error

* debug: add filter concerns

* debug: add back monitor

* debug: more monitor logic

* debug: add more monitor

* debug: switch proper order

* debug: add double check method

* debug: change slice range

* debug: revert

* debug: using pubkey instead of addres

* debug: clean code

* debug: add time sleep due to evm mechnism

* debug: no wget log

* debug: Improve test and script

* debug: ubuntu version for correct GLIBCXX_ version

* debug: increase time out

* debug: #2086 change entity name

* debug: a series of minor change

* debug: remove overlapping code from dev

* debug: clippy

* debug: comment regarding Cargo

* update cargo.lock

* increase the max header size

---------

Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>
Co-authored-by: Kailai Wang <Kailai.Wang@hotmail.com>

* Update Oneblock assertion type&description (#2123)

* update type&description

* using [] in outstanding description

* update supported networks

* fmt

* update Assertion Breakdown

* using outstanding_student

* Make tee-prod build less resource intensive (#2133)

* use thin lto

* force to build tee-prod too

* avoid tests

* revert ci

* Update README.md (#2023)

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

---------

Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>
Co-authored-by: WMQ <46511820+wangminqi@users.noreply.github.com>
Co-authored-by: Faisal Ahmed <42486737+felixfaisal@users.noreply.github.com>
Co-authored-by: BillyWooo <thedreamofbilly@gmail.com>

* fix vc di test (#2128)

* fix vc di test

* fmt

---------

Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>

* Bump serde_json from 1.0.105 to 1.0.107 (#2137)

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.105 to 1.0.107.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](serde-rs/json@v1.0.105...v1.0.107)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kasper Ziemianek <kasper.ziemianek@gmail.com>

* feat: 0.9.42 without TEE, clippy pass

* debug: fix template

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>
Co-authored-by: Kailai Wang <Kailai.Wang@hotmail.com>
Co-authored-by: Zhouhui Tian <125243011+zhouhuitian@users.noreply.github.com>
Co-authored-by: Adegoke Yusuff <88367184+cryptoade1@users.noreply.github.com>
Co-authored-by: Faisal Ahmed <42486737+felixfaisal@users.noreply.github.com>
Co-authored-by: BillyWooo <thedreamofbilly@gmail.com>
Co-authored-by: Kasper Ziemianek <kasper.ziemianek@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Kailai-Wang added a commit that referenced this pull request Nov 3, 2023
@BillyWooo @wangminqi
@Kailai-Wang @outofxxx @cryptoade1 @felixfaisal @kziemianek @dependabot @github-actions
update to upstream and polkadot v0.9.42 (#2173)
e5cc1b3 
* only update worker; not build

* bump version from 0.9.39 -> 0.9.42

* continue trying

* direct compare worker code

* one step further

* 0.9.42 minqi parachain (#2147)

* 1847 evm support (#2016)

* feat: initial commit

* feat: Runtime EVM Support (test TODO)

* feat: add evm test scripts

* debug: add evm parameter for test

* debug: quick format fix

* debug:

* debug: taplo

* debbug: taplo

* debug: try_runtime add

* debug: try default feature

* debug: test mode compile add runtime benchmarks

* debug: modify in non-dev

* debug: update yarn.lock file

* debug: remove all docker yarn changes

* debug: recover yarn.lock

* debug: using yarn instead of yarn install

* debug: corepack yarn install to generate yarn.lock

* debug: add extra test

* debug: fix test bug

* debug: ts debug

* debug: fix bug

* debug: ts

* debug: ts

* debug: modify workflow for better log

* debug: change node impl and ts

* debug: change gas price impl

* debug: using string addres directly

* debug: adjust paramter for test.ts

* debug: add extra ts for monitor the error

* debug: add filter concerns

* debug: add back monitor

* debug: more monitor logic

* debug: add more monitor

* debug: switch proper order

* debug: add double check method

* debug: change slice range

* debug: revert

* debug: using pubkey instead of addres

* debug: clean code

* debug: add time sleep due to evm mechnism

* debug: no wget log

* debug: Improve test and script

* debug: ubuntu version for correct GLIBCXX_ version

* debug: increase time out

* debug: #2086 change entity name

* debug: a series of minor change

* debug: remove overlapping code from dev

* debug: clippy

* debug: comment regarding Cargo

* update cargo.lock

* increase the max header size

---------

Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>
Co-authored-by: Kailai Wang <Kailai.Wang@hotmail.com>

* Update Oneblock assertion type&description (#2123)

* update type&description

* using [] in outstanding description

* update supported networks

* fmt

* update Assertion Breakdown

* using outstanding_student

* Make tee-prod build less resource intensive (#2133)

* use thin lto

* force to build tee-prod too

* avoid tests

* revert ci

* Update README.md (#2023)

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

---------

Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>
Co-authored-by: WMQ <46511820+wangminqi@users.noreply.github.com>
Co-authored-by: Faisal Ahmed <42486737+felixfaisal@users.noreply.github.com>
Co-authored-by: BillyWooo <thedreamofbilly@gmail.com>

* fix vc di test (#2128)

* fix vc di test

* fmt

---------

Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>

* Bump serde_json from 1.0.105 to 1.0.107 (#2137)

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.105 to 1.0.107.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](serde-rs/json@v1.0.105...v1.0.107)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kasper Ziemianek <kasper.ziemianek@gmail.com>

* feat: 0.9.42 without TEE, clippy pass

* debug: fix template

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>
Co-authored-by: Kailai Wang <Kailai.Wang@hotmail.com>
Co-authored-by: Zhouhui Tian <125243011+zhouhuitian@users.noreply.github.com>
Co-authored-by: Adegoke Yusuff <88367184+cryptoade1@users.noreply.github.com>
Co-authored-by: Faisal Ahmed <42486737+felixfaisal@users.noreply.github.com>
Co-authored-by: BillyWooo <thedreamofbilly@gmail.com>
Co-authored-by: Kasper Ziemianek <kasper.ziemianek@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* debug: TEE parachain

* debug: Cargo lock

* move forward

* worker and cli pass, enclave to go

* update worker to master (b9c80164)

* update substrate-api-client to polkadot-v0.9.42-tag-v0.14.0

* align with upstream worker

* make fmt

* something more for comparing

* another two for enclave

* direct compare with upstream worker again

* cargo fmt

* something more

* missing type define

* build worker and client

* first success build

* clean up some warnings

* bring back comment out code

* fix header compile error

* all build pass for the moment.
Test, then decide to remove the comment out code or not in worker_on_chain_ocall.rs

* debug: fix new weight test set for xcm

* debug: fix xcm test parameter

* debug: xcm weight parameter

* debug: reserve balance require free > ed

* debug: staking change due to pallet_balance change

* worker build

* parachain: build, clippy; worker: build, clippy, cargo test; enclave: build; PASS

* revert changes

* comment out checking rules, because of upstream code

* align with dev branch

* further fix of clippy and test

* comment out for the moment

* fix panic due to ED

* try older version 'checkout'

* comment out

* use 0 ED in test

* revert to old name

* revert comment

* minimize the code change

* debug: no insecure_zero_ed in test

* debug: bencmark test parameter

* debug: fix benchmark 0.9.42 ed failure again

* revert changes

* fix stack overflow

* minor update

* enable "send_extrinsic_failed" checking

* debug: benchmark production and test diff

* fix worker cargo test

* enable teaclave clippy

* revert param names

* debug: try clean disk when benchmark

* debug: ajust benchmark

* debug: benchmark

* debug:fmt

* debug: benchmark

* [benchmarking bot] Auto commit generated weights files (#2198)

Co-authored-by: wangminqi <wangminqi@users.noreply.github.com>

* fix error debugged for days...

* make fmt

* fix clippy

* cargo clean in the middle to save space

* no enough space again...

* add --release flag

* revert changes for the moment for testing

* fix di_vc test

* minor update

* fix mismatch of call names

* add missing param

* make fmt for all (rust and ts)

* one missing parameter

* fix CI tests

* fix shield_funds parameter wrong order

* fix param order

* remove the merge part from upstream

* fix resuming worker test

* try again

* fix clippy

* revert `nonce` cmd back to PublicGetter

* remove unused file.

* keep it only for cargo fmt

* remove mod name

* should not change extrinsic name within parachain pallet. (license issue)

* remove shell check.

* rename integritee->litentry

* remove 'evm' and 'teeracle' feature checking

* missing comment revert

* revert changes from upstream. Because pallet extrinsic stay unchanged.

* revert back print info

* add missing variable

* fmt

* missing one rename

* change attribute name

* rename extrinsic (missed one)

* revert nonce field back

* update api

* revert missing nonce field

* update client api

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: BillyWooo <yang@litentry.com>
Co-authored-by: WMQ <46511820+wangminqi@users.noreply.github.com>
Co-authored-by: Kai <7630809+Kailai-Wang@users.noreply.github.com>
Co-authored-by: Kailai Wang <Kailai.Wang@hotmail.com>
Co-authored-by: Zhouhui Tian <125243011+zhouhuitian@users.noreply.github.com>
Co-authored-by: Adegoke Yusuff <88367184+cryptoade1@users.noreply.github.com>
Co-authored-by: Faisal Ahmed <42486737+felixfaisal@users.noreply.github.com>
Co-authored-by: Kasper Ziemianek <kasper.ziemianek@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: WMQ <wangminqi@aliyun.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: wangminqi <wangminqi@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BillyWooo BillyWooo BillyWooo approved these changes

Assignees

@Kailai-Wang Kailai-Wang

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@Kailai-Wang @kziemianek @BillyWooo