Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is the size same as file statSync size #19

Open
brightchip opened this issue Oct 19, 2021 · 4 comments
Open

Is the size same as file statSync size #19

brightchip opened this issue Oct 19, 2021 · 4 comments

Comments

@brightchip
Copy link

I tried to read chunk by chunk and upload to remote server. but the final one seems exceed the file size

I'm using "read-chunk": "^3.2.0", because new version doesn't support require for which My project has to use it

firstChunk result {"d":{"StartUpload":"5242880"}}
uploading continue chunks 5242880 10485760 51387344

...
other chun result {"d":{"ContinueUpload":"47185920"}}
finished uplading... 47185920 52428800 51387344

   const stats = fs.statSync(path);
    const fileSizeInBytes = stats.size;
    console.log('fileSizeInBytes', fileSizeInBytes)

    for (offset = 0; offset < fileSizeInBytes; offset += chunkSize) {
        const chunk = await ReadChunk(path, offset, chunkSize);

        if (firstChunk) {
            console.log('uploading firstChunk', path, offset, offset + chunkSize, fileSizeInBytes)
       
        } else if (offset >= fileSizeInBytes - chunkSize) {
            console.log('finished uplading...', offset, offset + chunkSize, fileSizeInBytes)
        

        } else {
            console.log('uploading continue chunks', offset, offset + chunkSize, fileSizeInBytes)
    }
@sindresorhus
Copy link
Owner

This is most likely a logic bug in your code. However, I'm happy to take a closer look if you submit a failing tests.

@rawpixel-vincent
Copy link

rawpixel-vincent commented Oct 27, 2021

for (offset = 0; offset < fileSizeInBytes; offset += chunkSize) {
         const limit = offset + chunkSize <= fileSizeInBytes ? chunkSize : fileSizeInBytes - offset;
        const chunk = await ReadChunk(path, offset, limit);

the last loop could go over the fileSizeInBytes.
what does read-chunk in that case ?

the test would be

const buf = Buffer.from('12345');
const chunk = await ReadChunk(buf, 0, 10);
should(chunk.length === 5);

@rawpixel-vincent
Copy link

mkdir -p test-issue19 && cd test-issue19 && rm -f test.mjs && echo "1" > test.txt && yarn add read-chunk && echo "import {readChunk} from 'read-chunk';(async function() {let length = (await readChunk('test.txt', {length: 1000, startPosition: 1})).length;console.log(length);})()" > test.mjs && node test.mjs && cd - => 1

mkdir -p test-issue19 && cd test-issue19 && rm -f test.mjs && echo "12" > test.txt && yarn add read-chunk && echo "import {readChunk} from 'read-chunk';(async function() {let length = (await readChunk('test.txt', {length: 1000, startPosition: 1})).length;console.log(length);})()" > test.mjs && node test.mjs && cd - => 2

mkdir -p test-issue19 && cd test-issue19 && rm -f test.mjs && echo "123" > test.txt && yarn add read-chunk && echo "import {readChunk} from 'read-chunk';(async function() {let length = (await readChunk('test.txt', {length: 2, startPosition: 1})).length;console.log(length);})()" > test.mjs && node test.mjs && cd - => 2

works fine

@rawpixel-vincent
Copy link

rawpixel-vincent commented Nov 5, 2021

with 3.2.0
mkdir -p test-issue19 && cd test-issue19 && rm -f test.js && echo "1" > test.txt && yarn add read-chunk@3.2.0 && echo "const readChunk = require('read-chunk');(async function() {let length = (await readChunk('test.txt',1,1000)).length;console.log(length);})()" > test.js && node test.js && cd - => 1

mkdir -p test-issue19 && cd test-issue19 && rm -f test.js && echo "12" > test.txt && yarn add read-chunk@3.2.0 && echo "const readChunk = require('read-chunk');(async function() {let length = (await readChunk('test.txt',1,1000)).length;console.log(length);})()" > test.js && node test.js && cd - => 2

mkdir -p test-issue19 && cd test-issue19 && rm -f test.js && echo "123" > test.txt && yarn add read-chunk@3.2.0 && echo "const readChunk = require('read-chunk');(async function() {let length = (await readChunk('test.txt',1,2)).length;console.log(length);})()" > test.js && node test.js && cd - => 2

would need the input file used to test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants