Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: chunk API module #60

Merged
merged 2 commits into from
Jan 6, 2021
Merged

feat: chunk API module #60

merged 2 commits into from
Jan 6, 2021

Conversation

agazso
Copy link
Member

@agazso agazso commented Jan 5, 2021

This PR implements the endpoints related to chunks (chunk upload, download and pinning). It only adds at the module level because at the Bee class level I plan to add higher level functionality (e.g. Chunk interface instead of Uint8Arrays).

The upload API is quite strange that it already requires the hash to be calculated in order to be able to upload. Since this functionality was not yet added, I hardcoded some pre-calculated values in the upload and download test.

The chunk pinning API has a few extra functions compared to the other (bytes, fiels, collection) APIs and testing it is a bit difficult with only having hardcoded values because this way the tests affect each other. I plan to change that in the next iteration when we are able to create (pseudo-)random chunks, in this PR the goal of the tests are to provide coverage for the endpoint wrapper functions.

@agazso agazso mentioned this pull request Jan 5, 2021
7 tasks
@agazso agazso marked this pull request as ready for review January 5, 2021 19:57
Copy link
Contributor

@vojtechsimetka vojtechsimetka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment on lines +9 to +21
/**
* Upload chunk to a Bee node
*
* The chunk data consists of 8 byte span and up to 4096 bytes of payload data.
* The span stores the length of the payload in uint64 little endian encoding.
* Upload expects the chuck data to be set accordingly.
*
* @param url Bee URL
* @param hash Chunk reference
* @param data Chunk data to be uploaded
* @param options Aditional options like tag, encryption, pinning
*/
export async function upload(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it makes sense to add validation in this function as well? I am talking about checking the span, length of the payload and hash. Or would you add this to some utilities?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this API too low-level for practical usage and probably subject to change anyhow. Because of this I also wouldn't expose this API on the Bee interface. Instead I plan to implement utilities for creating correct chunks and calculate the BMT hash. We will need that to implement single owner chunks, but this is only necessary because there is no semantic API yet for handling those. Then I would expose only the SOC related functions on the Bee interface.

Copy link
Member

@nugaon nugaon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only problem that I noticed, on local test with cluster nodes the chunk (and also file, byte) download attempt on not existing reference throws timeout error.
we should raise the default jest timeout because of that.

*
* The chunk data consists of 8 byte span and up to 4096 bytes of payload data.
* The span stores the length of the payload in uint64 little endian encoding.
* Upload expects the chuck data to be set accordingly.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typeo

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean exactly? :)

@agazso agazso merged commit 110b092 into master Jan 6, 2021
@agazso agazso deleted the feat/chunk branch January 6, 2021 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants