[Bounty] Add CDN for the file service #2160

yisiliu · 2020-12-31T07:48:45Z

gitcoinbot · 2020-12-31T07:54:10Z

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 500.0 DAI (500.0 USD @ $1.0/DAI) attached to it.

If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
Want to chip in? Add your own contribution here.
Questions? Checkout Gitcoin Help or the Gitcoin Chat
$1,000,195.82 more funded OSS Work available on the Gitcoin Issue Explorer

sherlock-shi-x · 2020-12-31T08:04:06Z

🙋‍♂️🙋‍♂️🙋‍♂️

gitcoinbot · 2020-12-31T08:04:08Z

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 265 years, 10 months from now.
Please review their action plans below:

1) haihongs has been approved to start work.

Test arweave api endpoint
Communicate with Mask
Add CDN service for Mask
Test in different area
Whooooa

Learn more on the Gitcoin Issue Details page.

yisiliu · 2020-12-31T08:30:54Z

@haihongS please sync any update with us here, thanks

sherlock-shi-x · 2020-12-31T08:34:16Z

Ok, let's start it.

The usual flow:

https://arweave.net/j8BNG1iTI0kebUQM9UlR9abW56fRmBZLnV3gsUOfo44

redirects to ->

https://r7ae2g2ysmrushtniqgpkskr6wtnnz5h2gmbms45lxqlcq47uoha.arweave.net/j8BNG1iTI0kebUQM9UlR9abW56fRmBZLnV3gsUOfo44

CDN cache

We can offer an acceleration link:
https://coldcdn.com/api/cdn/bronil/j8BNG1iTI0kebUQM9UlR9abW56fRmBZLnV3gsUOfo44

When we check the content in HTML, found the real file link in Arweave which needs to be changed in Mask.

yisiliu · 2020-12-31T08:44:09Z

We'd like some speed tests from different areas. What's your thoughts? @septs

sherlock-shi-x · 2020-12-31T08:54:33Z

Can we get much more information about the user location in using Mask & Arweave

yisiliu · 2020-12-31T08:56:30Z

We do have some stats in Mask but you should probably test it all over the world to test out the robustness of your service

sherlock-shi-x · 2020-12-31T09:09:47Z

We do have some stats in Mask but you should probably test it all over the world to test out the robustness of your service

Yeah, so can we start with an alternative link which gives users an option to reach files through cache service, just like live streaming has multiple lines.

e.g. users based in China Mainland and North America can try the alternative link, and we will check the status of service and do some tunings.

After several periods, we will know the performance and robustness of service, then comes the real CDN.

@yisiliu

yisiliu · 2020-12-31T09:19:12Z

Yeah, so can we start with an alternative link which gives users an option to reach files through cache service, just like live streaming has multiple lines.

I guess the ideal solution would require you to modify our code so that it'll test out all the options and choose the best available.

sherlock-shi-x · 2020-12-31T10:00:41Z

sure, let me check the code

gitcoinbot · 2021-01-03T17:10:56Z

@haihongS Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

reminder (3 days)
escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

sherlock-shi-x · 2021-01-05T01:39:12Z

yeah, wip right now

sherlock-shi-x · 2021-01-05T07:00:22Z

export const signing = 'https://service.maskbook.com/arweave-remote-signing'

Does Mask maintain the signing system in private and keep SK?

yisiliu · 2021-01-05T07:22:55Z

@septs

septs · 2021-01-05T07:38:44Z

export const signing = 'https://service.maskbook.com/arweave-remote-signing'

Does Mask maintain the signing system in private and keep SK?

as arweave does not provide remote signing method,
but this repo include some non public limit settings, not can fully open source.
(if you idea access this codebase, need to get @yisiliu approve)

it's core logic like:

  const [data, parsed] = await readRequest(ctx.req.read());
  const signed = await instance.crypto.sign(keyfile, data);
  const id = await instance.crypto.hash(signed);
  ctx.body = {
    signature: bufferTob64Url(signed),
    id: bufferTob64Url(id),
  };

wrote here response body to transaction.setSignature, signed by remote

Maskbook/packages/maskbook/src/plugins/FileService/arweave/remote-signing.ts

Lines 6 to 13 in 75de7fe

    
           export async function sign(transaction: Transaction) { 
        
               const response = await fetch(signing, { 
        
                   method: 'POST', 
        
                   // Temporary workaround for https://github.com/msgpack/msgpack-javascript/issues/145 
        
                   body: Uint8Array.from(await makeRequest(transaction)), 
        
               }) 
        
               transaction.setSignature(await response.json()) 
        
           }

septs · 2021-01-05T07:47:23Z

@haihongS plz send a email to septs@dimension.im request the codebase, thanks on @yisiliu approved

sherlock-shi-x · 2021-01-05T08:25:19Z

@haihongS plz send a email to septs@dimension.im request the codebase, thanks on @yisiliu approved

Thx, and I can continue the idea without this part right now.

I read some codes and tried file service in twitter, here is my understanding of process:

User A clicks the FileService plugin in twitter and drops a file into it

-> mask builds an arweave transaction
-> signed by mask's signing system
-> broadcast to arweave network through endpoint:arweave.net
-> waiting for the confirmation of arweave network

-> display arweave link: https://arweave.net/MASK_FILE_HASH#KEY
-> arweave redirect: https://balabala.arweave.net/MASK_FILE_HASH#KEY
-> request real file download link: https://arweave.net/REAL_FILE_HASH

plz point out the mistakes above if it has @septs

sherlock-shi-x · 2021-01-05T08:39:09Z

Besides, I noticed the lack of the whole encryption, which means, I can scan the history of transactions in mask's account and find the real file hash without the key.

Two sequent transactions, one for key check and the other represents the real file.

septs · 2021-01-05T09:34:33Z

Besides, I noticed the lack of the whole encryption, which means, I can scan the history of transactions in mask's account and find the real file hash without the key.

key using hmac-sha256 verify user key correctness.

see https://github.com/DimensionDev/Maskbook-FileService-LandingPage/blob/1d87424b21ad3815e4987cd9c2e7bee2ec176112/src/components/Password.tsx#L77-L83

Two sequent transactions, one for key check and the other represents the real file.

first transaction
1. generate (un)encrypted file with metadata ¹
2. upload to arwavae
two transaction
1. inject metadata (include real file link, filename, etc) ² into Landing page ³
2. upload to arwavae

notes:

https://github.com/DimensionDev/common-protocols/blob/master/specs/0-BSFOW.md

Maskbook/packages/maskbook/src/plugins/FileService/arweave/index.ts

Lines 52 to 67 in 75de7fe

    
           export async function uploadLandingPage(metadata: LandingPageMetadata) { 
        
               const encodedMetadata = JSON.stringify({ 
        
                   name: metadata.name, 
        
                   size: metadata.size, 
        
                   link: `https://arweave.net/${metadata.txId}`, 
        
                   signed: await makeFileKeySigned(metadata.key), 
        
                   createdAt: new Date().toISOString(), 
        
               }) 
        
               const response = await fetch(landing) 
        
               const text = await response.text() 
        
               const replaced = text.replace('__METADATA__', encodedMetadata) 
        
               const data = encodeText(replaced) 
        
               const transaction = await makePayload(data, 'text/html') 
        
               await instance.transactions.post(transaction) 
        
               return transaction.id 
        
           }

https://github.com/DimensionDev/Maskbook-FileService-LandingPage
https://en.wikipedia.org/wiki/Landing_page

sherlock-shi-x · 2021-01-07T10:33:32Z

awesome codes, and now we need to talk about the solution for this issue.

purpose

offer a cdn for file service at this moment
accelerate other components for mask later

concern

the robustness of service

solution

keep the original arweave link, but give users an alternative link

I have noticed the two key point links in file service, one is landingTxID and the other is payloadTxID

so, I need to add alter link in Maskbook-FileService-LandingPage. Cuz every encoding in file service will fetch data from landing page, hot-update will be much easy.

Then, we can offer the entry atler landing-page link in twitter, which will help users who can't reach arweave endpoint.

@septs what's your idea?

sherlock-shi-x · 2021-01-13T03:13:02Z

we have added a 10MB test file in
https://arweave.net/MQvwlLNyJcE5-A7wo6oUE0j4QwlXfXxYNbSlwR70ZAg#FkNfuQppBpeyGs7N

also cache link for landingPage
https://coldcdn.com/api/cdn/bronil/MQvwlLNyJcE5-A7wo6oUE0j4QwlXfXxYNbSlwR70ZAg#FkNfuQppBpeyGs7N

plz check the speed.

sherlock-shi-x · 2021-01-13T06:54:57Z

In chrome incognito mode, I tested two files.

one is from original arweave endpoint(8.7M):
https://arweave.net/9JgBCn8uaToqiTbugQ43yRuQioLbvbeTA5mSLEbnYt8#sample

another is the cached one(9.4M):
https://arweave.net/VzLiXPYJLyAmWoQbH05W5P5SOaqWesQCFg7a935iKYE#MrRBRMeAnB8EQBE4

The results is much ideal.

The strategy keeps the same as traditional cloud service provider. When the first request arrived, the system will redirect to origin source and schedule a job, which can cache file to the user's neighbourhood. As soon as the caching ready, new request will be distributed to cached nodes according to load-balance strategy.

sherlock-shi-x · 2021-01-13T07:07:03Z

Hope it will be convenient to users and save the cost of mask. ^...^

gitcoinbot · 2021-01-15T01:52:02Z

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work for 500.0 DAI (500.0 USD @ $1.0/DAI) has been submitted by:

Learn more on the Gitcoin Issue Details page
Want to chip in? Add your own contribution here.
Questions? Checkout Gitcoin Help or the Gitcoin's Discord
$1,001,251.69 more funded OSS Work available on the Gitcoin Issue Explorer

Tedko · 2021-12-10T19:08:52Z

consider done @yisiliu

Tedko · 2021-12-10T19:09:14Z

should close gitcoin bounty if this is consider done @yisiliu

yisiliu added Kind: Network Bounty: Open Long-term Task labels Dec 31, 2020

yisiliu assigned sherlock-shi-x Dec 31, 2020

sherlock-shi-x mentioned this issue Jan 5, 2021

Add file acceleration support for Maskbook daqnext/pm#1

Open

sounak98 mentioned this issue Jan 11, 2021

feat: add nft plugin #2170

Merged

sherlock-shi-x mentioned this issue Jan 14, 2021

feat: add CDN for the arweave file service #2245

Merged

Tedko closed this as completed Dec 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bounty] Add CDN for the file service #2160

[Bounty] Add CDN for the file service #2160

yisiliu commented Dec 31, 2020

gitcoinbot commented Dec 31, 2020

sherlock-shi-x commented Dec 31, 2020

gitcoinbot commented Dec 31, 2020

yisiliu commented Dec 31, 2020

sherlock-shi-x commented Dec 31, 2020 •

edited

Loading

yisiliu commented Dec 31, 2020

sherlock-shi-x commented Dec 31, 2020

yisiliu commented Dec 31, 2020

sherlock-shi-x commented Dec 31, 2020

yisiliu commented Dec 31, 2020

sherlock-shi-x commented Dec 31, 2020

gitcoinbot commented Jan 3, 2021

sherlock-shi-x commented Jan 5, 2021

sherlock-shi-x commented Jan 5, 2021

yisiliu commented Jan 5, 2021

septs commented Jan 5, 2021

septs commented Jan 5, 2021 •

edited

Loading

sherlock-shi-x commented Jan 5, 2021

sherlock-shi-x commented Jan 5, 2021

septs commented Jan 5, 2021

sherlock-shi-x commented Jan 7, 2021 •

edited

Loading

sherlock-shi-x commented Jan 13, 2021 •

edited

Loading

sherlock-shi-x commented Jan 13, 2021

sherlock-shi-x commented Jan 13, 2021 •

edited

Loading

gitcoinbot commented Jan 15, 2021

Tedko commented Dec 10, 2021

Tedko commented Dec 10, 2021

[Bounty] Add CDN for the file service #2160

[Bounty] Add CDN for the file service #2160

Comments

yisiliu commented Dec 31, 2020

Catalogue

Part 1: Problem Description

Part 2: Resources

Part 3: Contact and Support

gitcoinbot commented Dec 31, 2020

sherlock-shi-x commented Dec 31, 2020

gitcoinbot commented Dec 31, 2020

yisiliu commented Dec 31, 2020

sherlock-shi-x commented Dec 31, 2020 • edited Loading

Ok, let's start it.

The usual flow:

CDN cache

yisiliu commented Dec 31, 2020

sherlock-shi-x commented Dec 31, 2020

yisiliu commented Dec 31, 2020

sherlock-shi-x commented Dec 31, 2020

yisiliu commented Dec 31, 2020

sherlock-shi-x commented Dec 31, 2020

gitcoinbot commented Jan 3, 2021

sherlock-shi-x commented Jan 5, 2021

sherlock-shi-x commented Jan 5, 2021

yisiliu commented Jan 5, 2021

septs commented Jan 5, 2021

septs commented Jan 5, 2021 • edited Loading

sherlock-shi-x commented Jan 5, 2021

sherlock-shi-x commented Jan 5, 2021

septs commented Jan 5, 2021

sherlock-shi-x commented Jan 7, 2021 • edited Loading

purpose

concern

solution

sherlock-shi-x commented Jan 13, 2021 • edited Loading

sherlock-shi-x commented Jan 13, 2021

sherlock-shi-x commented Jan 13, 2021 • edited Loading

gitcoinbot commented Jan 15, 2021

Tedko commented Dec 10, 2021

Tedko commented Dec 10, 2021

sherlock-shi-x commented Dec 31, 2020 •

edited

Loading

septs commented Jan 5, 2021 •

edited

Loading

sherlock-shi-x commented Jan 7, 2021 •

edited

Loading

sherlock-shi-x commented Jan 13, 2021 •

edited

Loading

sherlock-shi-x commented Jan 13, 2021 •

edited

Loading