feat: Database subset

[evoxmusic]

See this issue

[evoxmusic]

HUGE performance improvement - 30ms per iteration instead of 900ms before

[evoxmusic]

I know it's a large PR but if you want to take a look at it @benny-n @fabriceclementz it's all good for me :)

[benny-n]

Looks great, very nice feature to have!

[fabriceclementz]

Good job @evoxmusic ! This is my favorite feature for the moment !
The possibility to implement many subset strategies is great and the code is expressive enough. 👍

I noticed some errors during the backup restore command

  Running `target/debug/replibyte -c ./examples/with-subset-and-transformer.yaml restore -v latest`
⠉ [00:00:00] [--------------------------------------------------------------------------------------------------------------------------------------------------------] 0B/25.24KiB (0s)
⠁ [00:00:00] [#################################################################################################################################################] 202.58KiB/25.24KiB (0s)
⠠ [00:00:02] [#################################################################################################################################################] 202.58KiB/25.24KiB (0s)
ERROR:  could not create unique index "pk_customers"
DETAIL:  Key (customer_id)=(TOMSP) is duplicated.
ERROR:  could not create unique index "pk_employees"
DETAIL:  Key (employee_id)=(6) is duplicated.
ERROR:  could not create unique index "pk_shippers"
⠙ [00:00:02] [#################################################################################################################################################] 202.58KiB/25.24KiB (0s)
ERROR:  there is no unique constraint matching given keys for referenced table "customers"
ERROR:  there is no unique constraint matching given keys for referenced table "employees"
ERROR:  there is no unique constraint matching given keys for referenced table "employees"
ERROR:  there is no unique constraint matching given keys for referenced table "customers"
ERROR:  there is no unique constraint matching given keys for referenced table "employees"
⠄ [00:00:02] [#################################################################################################################################################] 202.58KiB/25.24KiB (0s)
Restore successful!

It seems some rows are inserted twice so the primary key cannot be added.
I haven't look further for now.

[evoxmusic]

Yes I got the same issue and I will provide a fix

[evoxmusic]

I'm facing a small challenge on rows deduplication. Since RepliByte needs to be low on memory consumption, it's almost impossible to make row deduplication in memory. I am looking for a remediation.

[fabriceclementz]

I'm facing a small challenge on rows deduplication. Since RepliByte needs to be low on memory consumption, it's almost impossible to make row deduplication in memory. I am looking for a remediation.

I think It would require having all the rows in memory for deduplicates them?
Maybe you could create a temporary index in S3 like a HashMap data structure so you don't have to keep all rows in memory and could check if a row is already inserted.
I'm not sure It will be the most efficient solution nor even if it can help you!

[evoxmusic]

It's a good idea but my concern is the IO performance. I presume it will be not super performant. For the database subset I am already using the local disk because there is no choice. We can't process data and assume that users will have a lot of memory available. However, we can assume that they have some disk space and it is a requirement for database subset. I am working on a function to dedup specific lines from a file. I will push my code tomorrow (I made a Rust conf tonight - I am super tired)

[evoxmusic]

@fabriceclementz @benny-n finally done!! 😄 I am going to merge, but I will need to also explain that they are some requirements to use the database subset feature since it needs some disk space to do the processing. We can improve that part in an incremental way

[fabriceclementz]

Nice 👍 I will try this soon !