Uploading files smaller than ~42kb does not work

[kdhillon]

Version: Using version 1.11.6.

Description and Repro steps: Uploading files smaller than 42kb fails with TypeError: Cannot read properties of undefined (reading 'proof').
I've tried several files between 40kb and 50kb. Files above 42kb always succeed. Files smaller than ~42kb always fail.

Using the arbundles package there is a similar bug only affecting files <42kb. The library would say all files were uploaded correctly, but when fetching files <42kb from arweave, a text file with text OK is returned instead of the actual file:
https://arweave.net/xNvPXixbejvmmytAIrFenHvjVSsB7UuRCHYE1ZsYt3I

It seems like other users have experienced something similar in the past: metaplex-foundation/metaplex#635

Severity
Our use case (Arpeggi Labs) is users uploading small, arbitrary audio files, so we need to support files of all sizes.
Unfortunately, we've had to put up a gate in our app to prevent users from uploading small files and tell users that Arweave doesn't support them.

[rosmcmahon]

there is absolutely no size restriction (big or small) to data uploaded to Arweave.

looking at your example above, it doesn't look like you used arweave-js to upload this data. in fact, this looks like a bundlr devnet transaction:
http://devnet.bundlr.network/tx/xNvPXixbejvmmytAIrFenHvjVSsB7UuRCHYE1ZsYt3I/data

i would suggest you take this up with Metaplex or Bundlr support.

[rosmcmahon]

hi @kdhillon, if you join Bundlr discord we are discussing it now in #developer. there was a Metaplex bug exactly like this 10 months ago anyhow. [edit: which is what you have linked to above actually]

[kdhillon]

Thanks for the quick response!

IThe link I included was from a bundlr upload, sorry for the confusion -- but I am reproducing this with arweave-js library consistently. After some more testing, I've discovered it only occurs when using the chunked uploader, but not the normal uploader. Here's the code:

      import Arweave from 'arweave'
 ...
      // This works for files > 42kb but not for < 42kb. 
      let uploader = await this.arweave.transactions.getUploader(transaction)
      while (!uploader.isComplete) {
        await uploader.uploadChunk()
      }

 // Meanwhile, this works for all files
 await this.arweave.transactions.post(transaction)

For now we can implement a workaround by only using the chunked uploader for larger files, but I still wanted to point out this is a reproducible issue with the chunked uploader.

[rosmcmahon]

Here's an 11kb file i just uploaded using the latest arweave-js https://arweave.net/pStCwYvcurPRS7i4MQ1gUTZUyixJfj3SvdUbNLOgVNI

This is the entire code:

import Arweave from 'arweave'
import { readFileSync } from 'fs'
const arweave  = Arweave.init({host: 'arweave.net'})
const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms))
const jwk = JSON.parse(readFileSync(new URL('./secret.json', import.meta.url), 'utf-8'))


const tx = await arweave.createTransaction({
	data: readFileSync(new URL('./assets/example.jpg', import.meta.url)), // 11kb file
})
tx.addTag('Content-Type', 'image/jpeg')
await arweave.transactions.sign(tx, jwk)

let uploader = await arweave.transactions.getUploader(tx);
while (!uploader.isComplete) {
	await uploader.uploadChunk()
	console.log(`lastResponseStatus: ${uploader.lastResponseStatus}`)
	console.log(`${uploader.pctComplete}% complete, ${uploader.uploadedChunks}/${uploader.totalChunks}`);
}

Can you give any more detail as to your set up? (and also example txids?) This is not something that has ever been seen before using just arweave-js. I've personally uploaded probably 1000s of <25kb test files using just arweave-js.

FYI, the post function uses chunk uploader also under the hood... Maybe you can see something it's doing that your code is not picking up on?

[rosmcmahon]

i should add, i'm using ts-node-esm under nodejs@16 to run the previous code.

we do try and do comprehensive testing before arweave-js releases, but we can't cover everything. e.g. framework integration

[rosmcmahon]

if you have anything further to add to this issue you can reopen it @kdhillon.