Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

index: add db-based implementation (e.g. sqlite) #208

Closed
efiop opened this issue Oct 18, 2022 · 1 comment · Fixed by #248
Closed

index: add db-based implementation (e.g. sqlite) #208

efiop opened this issue Oct 18, 2022 · 1 comment · Fixed by #248
Assignees
@efiop

Comments

@efiop
Copy link
Member

efiop commented Oct 18, 2022

We currently use in-memory prefix trie, but for large enough indexes it will be much nicer to have an ability to use a proper db and also use it to do operations like diff more efficiently (e.g. directly in sql query instead of fetching and comparing in python).
@efiop efiop self-assigned this Nov 29, 2022
efiop added a commit to efiop/dvc-data that referenced this issue Jan 5, 2023
@efiop
index: use sqltrie
1ee691e 
Fixes iterative#208
@efiop efiop mentioned this issue Jan 5, 2023
Merged
efiop added a commit to efiop/dvc-data that referenced this issue Jan 6, 2023
@efiop
index: use sqltrie
381e358 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 6, 2023
@efiop
index: use sqltrie
6e66b3f 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 6, 2023
@efiop
index: use sqltrie
1e1e97b 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 6, 2023
@efiop
index: use sqltrie
f9ef0c6 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 6, 2023
@efiop
index: use sqltrie
790865f 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 6, 2023
@efiop
index: use sqltrie
86bffd5 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 6, 2023
@efiop
index: use sqltrie
d8986cb 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 7, 2023
@efiop
index: use sqltrie
6cc8032 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 8, 2023
@efiop
index: use sqltrie
9a654e5 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 8, 2023
@efiop
index: use sqltrie
4a6c7e9 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 10, 2023
@efiop
index: use sqltrie
78cb984 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 11, 2023
@efiop
index: use sqltrie
4c41d55 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 11, 2023
@efiop
index: use sqltrie
bf927d0 
Fixes iterative#208
efiop added a commit to efiop/dvc-data that referenced this issue Jan 12, 2023
@efiop
index: use sqltrie
41a011b 
Fixes iterative#208
@efiop efiop mentioned this issue Jan 14, 2023
Closed
efiop added a commit to efiop/dvc-data that referenced this issue Jan 15, 2023
@efiop
index: md5: accept legacy state
3636e21 
Temporary connector before persistent index iterative#208 is fully supported.

Pre-requisite for iterative/dvc#8761
@efiop efiop mentioned this issue Jan 15, 2023
Merged
efiop added a commit that referenced this issue Jan 15, 2023
@efiop
index: md5: accept legacy state (#258)
a569ef3 
Temporary connector before persistent index #208 is fully supported.

Pre-requisite for iterative/dvc#8761
efiop added a commit that referenced this issue Jan 15, 2023
@efiop
index: use sqltrie (#248)
d226e2e 
Fixes #208
efiop added a commit to iterative/dvc that referenced this issue Jan 16, 2023
@efiop
index: use persistent data index
2dc1a74 
Related iterative/dvc-data#208
@efiop
Copy link
Member Author

efiop commented Jan 16, 2023

For the record, DataIndex now has an open method that allows using sqlite-based prefix tree. The current implementation is not perfect by any means(e.g. sqltrie is pretty raw and will need fixes/optimizations along the way), but sets course in a proper direction.

This was referenced Jan 16, 2023
Merged
Closed
efiop added a commit to iterative/dvc that referenced this issue Jan 30, 2023
@efiop
index: use persistent data index
833c450 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Jan 31, 2023
@efiop
index: use persistent data index
85c8ede 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Jan 31, 2023
@efiop
index: use persistent data index
527f025 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 3, 2023
@efiop
index: use persistent data index
25f7708 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 3, 2023
@efiop
index: use persistent data index
38be12d 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 3, 2023
@efiop
index: use persistent data index
9875c1d 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 4, 2023
@efiop
index: use persistent data index
b59a74c 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 4, 2023
@efiop
index: use persistent data index
8a5569d 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 5, 2023
@efiop
index: use persistent data index
91da9fe 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 6, 2023
@efiop
index: use persistent data index
ed70ae4 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 6, 2023
@efiop
index: use persistent data index
f694311 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 6, 2023
@efiop
index: use persistent data index
b1daf13 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 6, 2023
@efiop
index: use persistent data index
914b469 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 6, 2023
@efiop
index: use persistent data index
e75a9d1 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 7, 2023
@efiop
index: use persistent data index
62eff78 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 7, 2023
@efiop
index: use persistent data index
a0751ae 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 7, 2023
@efiop
index: use persistent data index
c9ebba0 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 7, 2023
@efiop
index: use persistent data index
27aa530 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 7, 2023
@efiop
index: use persistent data index
22bc53e 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 7, 2023
@efiop
index: use persistent data index
17fb0f3 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 8, 2023
@efiop
index: use persistent data index
a926fae 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 9, 2023
@efiop
index: use persistent data index
871957d 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 9, 2023
@efiop
index: use persistent data index
3735388 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 9, 2023
@efiop
index: use persistent data index
b765b37 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 9, 2023
@efiop
index: use persistent data index
c2f1113 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 12, 2023
@efiop
index: use persistent data index
e2a484d 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 12, 2023
@efiop
index: use persistent data index
b998e11 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 13, 2023
@efiop
index: experimental persistent data index
7241147 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 13, 2023
@efiop
index: experimental persistent data index
6560412 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 13, 2023
@efiop
index: experimental persistent data index
04e063d 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 13, 2023
@efiop
index: experimental persistent data index
191f040 
Related iterative/dvc-data#208
efiop added a commit to iterative/dvc that referenced this issue Feb 13, 2023
@efiop
index: experimental persistent data index
86a6fb1 
Related iterative/dvc-data#208
daavoo pushed a commit to iterative/dvc that referenced this issue Feb 13, 2023
@efiop @daavoo
index: experimental persistent data index
c5bd17d 
Related iterative/dvc-data#208
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@efiop efiop

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

index: use sqltrie efiop/dvc-data
1 participant
@efiop