Make transformers optional + allow pinecone-text[dense] #266
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t[dense] 1. `transformers` shouldn't be a required dependency. It was accidentaly added as such. 2. Added option to intall pinecone-text with dense dependency, under a new extra named `torch` (feels like the right name, since this is the heaviest dependency).
igiloh-pinecone
commented
Jan 24, 2024
7 tasks
Now that I made `transformers` optional, needed to add error handling so it won't crash on import
I added make command that installs canopy with a selected list of extras, passed by an env var
Should be the first command after lint
izellevy
reviewed
Jan 28, 2024
| @@ -9,6 +9,10 @@ inputs: | |||
| description: "Whether to install canopy library, or dependencies only" | |||
| required: true | |||
| default: "true" | |||
| extras: | |||
| description: "Extra dependencies to install" | |||
There was a problem hiding this comment.
Maybe we should explain it is space separated
There was a problem hiding this comment.
Taking a list also is an option
izellevy
reviewed
Jan 28, 2024
| @@ -37,8 +41,12 @@ runs: | |||
| - name: Install dependencies | |||
| shell: bash | |||
| if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true' | |||
| run: poetry install --no-interaction --no-root --all-extras --with dev | |||
| env: | |||
There was a problem hiding this comment.
Do we need --with dev? By default it is with dev isnt it?
There was a problem hiding this comment.
I wasn't sure, and I prefer to be explicit...
igiloh-pinecone
commented
Jan 28, 2024
Improve the text description of the new `Extras` field
izellevy
reviewed
Jan 28, 2024
izellevy
approved these changes
Jan 28, 2024
Not really needed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
transformersshouldn't be a required dependency. It was accidentally added as such.pinecone-textwith its[dense]extra dependencySolution
transformersoptional, only installed with a newanyscaleextratorchextra (torch is the heaviest installation, so it made more sense as the extra's name thantransformersorsentence-transformers, which might be ambiguous)Type of Change
Might be slightly breaking for users who currently use the Anyscale Tokenizer, and would need to re-install with the extra.
Test Plan
No affect on current tests