Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NSFS | MPU | optimization #7434

Merged
merged 1 commit into from
Oct 2, 2023

Conversation

romayalon
Copy link
Contributor

@romayalon romayalon commented Aug 7, 2023

Explain the changes

Issues: Fixed #xxx / Gap #xxx

  1. Fixed NSFS Presto uploads large file as multiparts, calls complete with short timeout, causing retries and race on the mpu dir #7348

Testing Instructions:

  • Doc added/updated
  • Tests added

@romayalon romayalon requested a review from guymguym August 7, 2023 10:03
@romayalon romayalon force-pushed the romy-mpu-optimization branch 3 times, most recently from 8daded1 to bdfc30b Compare August 7, 2023 13:04
Copy link
Contributor

@liranmauda liranmauda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments but LGTM

src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
@romayalon romayalon force-pushed the romy-mpu-optimization branch 3 times, most recently from c99d6e7 to 617609a Compare August 14, 2023 15:49
@romayalon romayalon force-pushed the romy-mpu-optimization branch 2 times, most recently from 8955419 to d33c756 Compare August 21, 2023 09:07
src/native/fs/fs_napi.cpp Outdated Show resolved Hide resolved
src/util/chunk_fs.js Outdated Show resolved Hide resolved
src/util/chunk_fs.js Outdated Show resolved Hide resolved
src/test/unit_tests/sudo_index.js Outdated Show resolved Hide resolved
src/util/chunk_fs.js Outdated Show resolved Hide resolved
src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
Comment on lines 1420 to 1459
let open_mode = 'r+';
const md_open_mode = 'w+';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can add a single open mode if that can help here -

    { "w*", O_CREAT | O_RDWR },

Copy link
Contributor Author

@romayalon romayalon Aug 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead, what I did was to use * as truncate remover, let me know what you think

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd summon @eshelmarc to comment on that. WDYT on this mode. We don't want to truncate, but keep writing to the file in offsets that are multiplies of the part size. Romy said that O_APPEND is also not suitable so we wanted to open with just O_CREAT - Does this make sense?

src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
src/util/chunk_fs.js Outdated Show resolved Hide resolved
src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
@romayalon romayalon force-pushed the romy-mpu-optimization branch 4 times, most recently from 0e368bf to bf7b37b Compare August 28, 2023 07:00
@romayalon romayalon force-pushed the romy-mpu-optimization branch 5 times, most recently from 2741ae5 to 8917331 Compare October 1, 2023 06:57
Signed-off-by: Romy <35330373+romayalon@users.noreply.github.com>
@nimrod-becker nimrod-becker merged commit 9a15887 into noobaa:master Oct 2, 2023
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants