Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add AssetManager #603

Merged
merged 26 commits into from
Oct 5, 2022
Merged

feat: add AssetManager #603

merged 26 commits into from
Oct 5, 2022

Conversation

sea-snake
Copy link
Contributor

@sea-snake sea-snake commented Jul 28, 2022

Description

Currently there's the AssetManager implementation available here.

This PR improves on this implementation and moves it to @dfinity/assets.

The AssetManager has list, store, delete, get, batch and clear methods to manage assets in an asset storage canister.

Fixes #602

How Has This Been Tested?

  • AssetManager has e2e tests (Node.js)
  • Readable has unit tests

Checklist:

  • My changes follow the guidelines in CONTRIBUTING.md.
  • The title of this PR complies with Conventional Commits.
  • I have edited the CHANGELOG accordingly.
  • I have made corresponding changes to the documentation.

@dfinity-droid-prod
Copy link

Dear @sea-snake,

In order to potentially merge your code in this open-source repository and therefore proceed with your contribution, we need to have your approval on DFINITY's CLA1.

If you decide to agree with it, please visit this issue and read the instructions there.

— The DFINITY Foundation

Footnotes

  1. Contributor License Agreement

@sea-snake
Copy link
Contributor Author

sea-snake commented Jul 28, 2022

PR is work in progress.

Main points that are still TODO:

  • Implement a way to track events and/or progress
  • Additional unit and e2e tests
  • Documentation
  • Example application

@sea-snake sea-snake marked this pull request as ready for review September 26, 2022 16:21
@sea-snake sea-snake requested a review from a team as a code owner September 26, 2022 16:21
@sea-snake
Copy link
Contributor Author

Example application PR: dfinity/examples#355

packages/assets/jest.config.ts Outdated Show resolved Hide resolved
packages/assets/src/canisters/assets_idl.ts Outdated Show resolved Hide resolved
packages/assets/src/index.ts Outdated Show resolved Hide resolved
packages/assets/src/index.ts Outdated Show resolved Hide resolved
packages/assets/tsconfig.json Outdated Show resolved Hide resolved
@krpeacock
Copy link
Contributor

This is close to ready, and the main logic looks solid! To manually run the lint checks, you can run npm run lint --workspace @dfinity/assets

@sea-snake
Copy link
Contributor Author

This is close to ready, and the main logic looks solid! To manually run the lint checks, you can run npm run lint --workspace @dfinity/assets

Thanks for the helpful feedback! I've resolved these points in the latest commit.

@domwoe
Copy link
Member

domwoe commented Oct 4, 2022

@sea-snake Seems there is still an issue with the timeout for the E2E tests.

@sea-snake
Copy link
Contributor Author

sea-snake commented Oct 4, 2022

@sea-snake Seems there is still an issue with the timeout for the E2E tests.

Ended up reducing the asset size in the tests (e.g. from 1MB to 1KB) to decrease the test duration. On a local environment, the processing speed is more in line with expectation (2s) for each chunk but it seems things take a bit longer on the automated test env.

@domwoe
Copy link
Member

domwoe commented Oct 5, 2022

Thanks @sea-snake!
@krpeacock Seems to be ready to be merged now.

@krpeacock
Copy link
Contributor

@sea-snake is it concerning that the tests were timing out with a minute to run for a 3mb upload?

@sea-snake
Copy link
Contributor Author

@krpeacock
I've benchmarked all steps involved. Some observations I've made:

  • Sha256 in JS is slow, sadly the current native web implementation doesn't allow incremental hashing so it's not an alternative (You don't want to load a big file in memory if avoidable).
  • Every update call takes approx. 2s. For a 3MB asset test there are 5 calls (create_batch, create_chunk, create_chunk, commit_batch and delete_asset). If all these calls were made sequentially, this would be 10s. The create_chunk calls are made in parallel in theory but the dfx environment (local, production, system it's running on) might make a difference there.
  • The psuedo random bytes generator in the test isn't the fastest code out there but didn't seem to make a big influence on the overall test duration.
  • Maybe the cbor encoding of the update call messages might also affect the speed depending on the system it's running on.

@sea-snake
Copy link
Contributor Author

@krpeacock
One difference between node and web is that fetch in node is still using http1.1 instead of http2 if I remember correctly. Might be interesting to see if there's a difference in performance when tests are run in cypress.

@krpeacock
Copy link
Contributor

I'll go ahead and merge so we can get this work in front of people, and I'll create an issue to follow up on automating some benchmarks

@sea-snake
Copy link
Contributor Author

Doing some more debugging now to see if there are some other issues slowing things down.

@krpeacock krpeacock merged commit a3b3224 into dfinity:main Oct 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migrate AssetManager to @dfinity/assets
3 participants