rgw: disable prefetch in rgw_file to fix 3x read amplification

[jettjaniak]

rgw: This PR disables prefetch in rgw_file (used by NFS-Ganesha).

Each call to rgw_read (rgw_file.cc) invokes three calls to RGWRados::get_obj_state with RGWObjState::prefetch_data = true. It results in great read amplification. If length argument in rgw_read call is smaller than rgw_max_chunk_size, then the amplification is threefold.

Tracker issue: https://tracker.ceph.com/issues/48289

Signed-off-by: Kajetan Janiak kjaniak@cloudferro.com

[mattbenjamin]

@kajetanJ thanks for spotting this; at the moment, I think it probably is correct to avoid prefetching in general, but we might be able to address this in future and also take advantage of range prefetch (@ofriedma) and some additional infrastructure to get benefit from prefetch in future? It's definitely not intended that there be multiple prefetch attempts, so we'd have to fix that, also

[mattbenjamin]

@kajetanJ can you create a tracker issue and attach it to his PR?

[mattbenjamin]

lgtm (but please update with tracker issue)

[jettjaniak]

@mattbenjamin my account on tracker is pending administrator approval. Would you like me to include all reproduction steps or just describe the issue?

[mattbenjamin]

@kajetanJ either is fine; what's your account name in tracker? I can try to expedite it

[jettjaniak]

@mattbenjamin it is kjaniak. Thanks.

[ofriedma]

I think it would be with less changes just to change line 1975 from true to false

RGWGetObj::get_data = false; // XXX

[mattbenjamin]

I think it would be with less changes just to change line 1975 from true to false

RGWGetObj::get_data = false; // XXX

I don't agree with this advice. @kajetanJ 's proposal is fine, and will inline just fine.

Matt

[jettjaniak]

@mattbenjamin Tracker issue added (link in the first comment).
@ofriedma

I think it would be with less changes just to change line 1975 from true to false

RGWGetObj::get_data = false; // XXX

it would cause problems in RGWGetObj::execute, rgw_op.cc:2220

  if (!get_data || ofs > end) {
    send_response_data(bl, 0, 0);
    return;
  }

[jettjaniak]

Could we backport this to Nautilus?

[dang]

Just mark the tracker with the backports you want.

[jettjaniak]

@dang it seems I can't edit an existing issue, so I added a note

[mattbenjamin]

done: we want octopus, nautilus, I believe

[mattbenjamin]

@dang tested by hand in a Nautilus backport, passed librgw_file_nfsns, the main unit file.

[mattbenjamin]

I've proved non-regression in unit tests run by hand, maybe worth confirming perf result