Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Init ColBERTv2 managed index #9656

Merged
merged 3 commits into from Dec 21, 2023
Merged

Init ColBERTv2 managed index #9656

merged 3 commits into from Dec 21, 2023

Conversation

hatianzhang
Copy link
Contributor

@hatianzhang hatianzhang commented Dec 21, 2023 •
edited

Description

test ColBERT v2 model as a managed index for benchmarking purpose
https://arxiv.org/abs/2112.01488

Fixes # (issue)

Type of Change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Added new unit/integration tests
  • Added new notebook (that tests end-to-end)
  • I stared at the code and made sure it makes sense

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added Google Colab support for the newly added notebooks.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran make format; make lint to appease the lint gods

@dosubot dosubot bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Dec 21, 2023
@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@logan-markewich
Copy link
Collaborator

Do you mind also removing /experimental/colbert_index and /examples/paul_graham_essay/ColbertIndex.ipynb ? This will replace that

@logan-markewich
Copy link
Collaborator

Oh, or did it show up as a move/rename? I think the old notebook is still there at least

logan-markewich
logan-markewich reviewed Dec 21, 2023
View reviewed changes
llama_index/indices/managed/colbert_index/base.py
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this would make more sense as just a retriever? Similar to BM25Retriever ? Thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

colbertv2 indeed build index. it has its own indexer, retriever

@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Dec 21, 2023
@hatianzhang
Copy link
Contributor Author

hatianzhang commented Dec 21, 2023 •
edited

Oh, or did it show up as a move/rename? I think the old notebook is still there at least

removed old notebook in this pr. the colbert files are renamed

hatianzhang
hatianzhang commented Dec 21, 2023
View reviewed changes
llama_index/indices/managed/colbert_index/base.py
@@ -121,12 +124,9 @@ def _build_index_from_nodes(self, nodes: Sequence[BaseNode]) -> IndexDict:
kmeans_niters=self.kmeans_niters,
)
indexer = Indexer(checkpoint=self.model_name, config=config)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@logan-markewich their indexer

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea I suppose. It just feels very similar to BM25 since you can't add or delete data from it? It doesn't fit our definition of index very well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh they do have CRUD for index data: https://github.com/stanford-futuredata/ColBERT/blob/117098348e5196ede1c8e396c1c14f24d9a8754e/colbert/index_updater.py

the previous pr does not support it. I will add them for another pr

this pr is mostly for testing retrieval performance

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 21, 2023
@hatianzhang hatianzhang merged commit b1adb30 into main Dec 21, 2023
8 checks passed
@hatianzhang hatianzhang deleted the hz/hz_colbertv2 branch December 21, 2023 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@logan-markewich logan-markewich logan-markewich approved these changes

Assignees
No one assigned
Labels
lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@hatianzhang @logan-markewich