Lance random take performace

[pengzhiwei2018]

I have do some test on the lance random take on local ssd disk, I test take 1000 rows one by one and take 1000 row at once, the performance has 40x differ, I suspect it related to the the scheduler cost between io and decode. Anyone have some suggestion about this? thanks.

main test code is here:

const test_cnt: u32 = 100;
let idx_vec = gen_random_ids(1000);
let batch_size = 1; 
let start_ts = std::time::Instant::now();
for i in 0..test_cnt {
   let chunnks = idx_vec.chunks(batch_size);
   for idx_chunk in chunnks {
     let mut ids = Vec::from(idx_chunk);
     let mut read_stream = lance_reader.take(ids).await;
    
     while let Some(batch_r) = read_stream.next().await {
       let batch = batch_r.unwrap();
     }
  }
 }
 let end_ts = std::time::Instant::now();
 println!("random read time: {:?}", (end_ts - start_ts) / test_cnt);

[pengzhiwei2018]

@westonpace can you help take a look at this issue

[Xuanwo]

Hi, @pengzhiwei2018. I'm not sure what type lance_reader is in this context. Could you share a more complete code snippet, including how lance_reader is created?

[pengzhiwei2018]

@Xuanwo here is the LanceReader, just a wrapper for lance's v2 FileReader. Thanks.

pub struct LanceReader {
	file_reader: Box<FileReader>,
}

impl LanceReader {
	pub async fn open(
		fs: Arc<dyn FileSystem>,
		path: &str
	) -> LanceReader {
		...
		let file_reader = FileReader::try_open(
			file_scheduler,
			None,
			Default::default(),
			&FileMetadataCache::with_capacity(1000, CapacityMode::Items),
			FileReaderOptions::default(),
		)
		.await
		.unwrap();

		Ok(LanceReader {
			file_reader: Box::new(file_reader),
		})
	}

	pub async fn take(&mut self, row_ids: Vec<u32>) -> Pin<Box<dyn RecordBatchStream>> {
		let indices = ReadBatchParams::Indices(arrow_array::UInt32Array::from(row_ids));
		self.file_reader.read_stream(indices, 1000, 1, FilterExpression::no_filter()).unwrap()
	}

[Xuanwo]

Thank you for giving more detailed context.

I suspect it related to the the scheduler cost between io and decode.

I believe this is true. For every take, we will need to create a new DecodeBatchScheduler. I'm guessing the cost is not optimal. A more detailed analysis should be done after we have a flamegraph of performance.

What is the schema of your lance table, and how much data does it contain? I might be able to help identify the bottleneck.

---

By the way, this is unrelated to the original question. Why do you want to retrieve only one row when you could process them in batches? Are you trying to test that whether lance has good random access performance?

[pengzhiwei2018]

We use lance in the pytorch dataset, We want to shuffle the data random and get the item by the random index one by one. If using batch load, there may cause oom issue in our case.

[Xuanwo]

Thank you for the explanation. This issue may also be related to #3983

[westonpace]

We use lance in the pytorch dataset, We want to shuffle the data random and get the item by the random index one by one. If using batch load, there may cause oom issue in our case.

The torch map dataset has a __getitems__ method that can be overridden so you can have both index-based sampling and batch reads. I would recommend this approach.

If you have so much data that a batch load is going to OOM then I would expect the scheduling overhead to be negligible. If the data is relatively small then you should be able to batch it. How big was the data when you measured a 40x difference?

[pengzhiwei2018]

We use lance in the pytorch dataset, We want to shuffle the data random and get the item by the random index one by one. If using batch load, there may cause oom issue in our case.

The torch map dataset has a __getitems__ method that can be overridden so you can have both index-based sampling and batch reads. I would recommend this approach.

If you have so much data that a batch load is going to OOM then I would expect the scheduling overhead to be negligible. If the data is relatively small then you should be able to batch it. How big was the data when you measured a 40x difference?

I use a 300MB data to do the test in local ssd. I have preload all the data to memory before take. after this, the differ is also 40x.

[pengzhiwei2018]

I generate a flame graph, it seems the rust malloc spend much time for take. I use the mimalloc instead of the default one. The performace has improved 3x. The DecodeBatchScheduler::try_new spend many time.

[Xuanwo]

Hi @pengzhiwei2018, how about we move our discussion to #3983 instead? Although that issue compares LMDB with Lance, we're both focused on random access performance. I think it would be a great place for us to collaborate.

[Xuanwo]

Closing this because the original performance question was continued and answered in #3983, while the remaining broader random-read work is tracked in #4374 and #5066-#5069. This does not claim that single-row reads have no overhead. @pengzhiwei2018, please reopen with a distinct current-version workload if those tracking issues do not cover it.