AIMD retries broken for multipart uploads (Missing Part error)

[westonpace]

Description

If there are 1 or 2 errors then object_store will retry the request and it will be fine. If there are 3 errors then object_store kicks the failure to the AIMD layer which cuts the rate and retries the request. If this second case happens the final write will fail.

When object_store handles retries we are fine. When AIMD kicks off a retry we are not. The object_store layer will see it as a new call to put_part and will assign it a new part number. The old part number will be a gap. When you try and complete the write you will get:

OSError: LanceError(IO): failed to shutdown object writer ... Generic Parts error: Missing part

There is no easy fix for this at the moment. I don't see any mechanism to tell object_store this is a retried request and should re-use the part (but maybe Claude can reverse engineer something). We also can't increase the object_store retries just for put_part. If we increase the retries globally then we hurt the AIMD sensitivity for the read path.

Steps to reproduce

There is a complete reproducer here: https://github.com/lance-format/lance/compare/claude/lance-multipart-upload-error-q09m9u

Expected behavior

No response

Lance version

9.0.0-rc.2

Language binding

Rust

Environment

All

Logs / traceback

[wjones127]

I wonder if we can replace the default retry mechanism by creating our own HttpConnector and wrapping with that. Then we won't have this multi-layer retry problem. For an example, see how we implemented retry-level counters in MeteringHttpConnector:

lance/rust/lance-io/src/object_store/metrics.rs

Lines 588 to 611 in bbc0477

	#[derive(Debug)]
	pub struct MeteringHttpConnector {
	base: String,
	inner: ReqwestConnector,
	}
	impl MeteringHttpConnector {
	pub fn new(base: String) -> Self {
	Self {
	base,
	inner: ReqwestConnector::default(),
	}
	}
	}
	impl HttpConnector for MeteringHttpConnector {
	fn connect(&self, options: &ClientOptions) -> object_store::Result<HttpClient> {
	let client = self.inner.connect(options)?;
	Ok(HttpClient::new(MeteringHttpService {
	base: self.base.clone(),
	inner: client,
	}))
	}
	}

lance/rust/lance-io/src/object_store/providers/aws.rs

Lines 98 to 103 in bbc0477

	#[cfg(feature = "metrics")]
	{
	builder = builder.with_http_connector(
	crate::object_store::metrics::MeteringHttpConnector::new(store_prefix),
	);
	}