Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix tiktoken pinning #85

Merged
merged 5 commits into from
Oct 3, 2023
Merged

Fix tiktoken pinning #85

merged 5 commits into from
Oct 3, 2023

Conversation

tarrade
Copy link
Contributor

@tarrade tarrade commented Sep 14, 2023

Checklist

  • Used a personal fork of the feedstock to propose changes
  • Bumped the build number (if the version is unchanged)
  • Reset the build number to 0 (if the version changed)
  • Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
  • Ensured the license file is being packaged.

Fix issue with upper limit for tiktoken >=0.3.2,<0.5.0 instead of 0.4.0

@conda-forge-webservices
Copy link
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Copy link
Member

@pavelzw pavelzw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/langchain-ai/langchain/blob/v0.0.288/libs/langchain/pyproject.toml#L34
Upstream, the dependency is ^0.3.2, i.e., >=0.3.2,<0.4.0. How about you change it upstream? Otherwise pip check will fail.

@tarrade
Copy link
Contributor Author

tarrade commented Sep 14, 2023

@conda-forge-admin, please rerender

@github-actions
Copy link
Contributor

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/langchain-feedstock/actions/runs/6184569452.

@tarrade
Copy link
Contributor Author

tarrade commented Sep 14, 2023

@pavelzw I am not an expert but what I saw is that ^0.3.2 means any version greater that 0.3.2. I like the way conda-forge is doing by putting an upper. The latest version is 0.5.0 so the limit should be 0.6.0. Did I misundertood something ?

My issue is that with pip I can install the latest version of langchain with the latest version of tiktoken but not with conda-forge

@pavelzw
Copy link
Member

pavelzw commented Sep 14, 2023

https://python-poetry.org/docs/dependency-specification/#caret-requirements

image

The version constraints in this feedstock are automatically transferred from the pyproject.toml of the langchain project.
Being able to install langchain with tiktoken 0.5.0 seems like an issue with pip. The correct behavior should be that it's not installable together.

If you find that tiktoken 0.5.0 works well with langchain, please open a PR in the langchain repo similar to langchain-ai/langchain#2865 for example. As soon as this PR would be merged and released, the dependency specification of this repo will be updated.

@tarrade
Copy link
Contributor Author

tarrade commented Sep 14, 2023

Thanks for the explanation. I will check again and follow your recommendation

@tarrade
Copy link
Contributor Author

tarrade commented Sep 14, 2023

I created 2 conda env with the following: conda create --name test_conda python=3.9 pip then in each identical env I installed tagged version of langchain and tiktoken with pip and conda

With pip it is working fine:

 pip install langchain==0.0.288 tiktoken==0.5.0
 Installing collected packages: urllib3, typing-extensions, tenacity, regex, PyYAML, packaging, numpy, mypy-extensions, multidict, idna, greenlet, frozenlist, charset-normalizer, certifi, attrs, async-timeout, annotated-types, yarl, typing-inspect, SQLAlchemy, requests, pydantic-core, numexpr, marshmallow, aiosignal, tiktoken, pydantic, dataclasses-json, aiohttp, langsmith, langchain
Successfully installed PyYAML-6.0.1 SQLAlchemy-2.0.20 aiohttp-3.8.5 aiosignal-1.3.1 annotated-types-0.5.0 async-timeout-4.0.3 attrs-23.1.0 certifi-2023.7.22 charset-normalizer-3.2.0 dataclasses-json-0.5.14 frozenlist-1.4.0 greenlet-2.0.2 idna-3.4 langchain-0.0.288 langsmith-0.0.37 marshmallow-3.20.1 multidict-6.0.4 mypy-extensions-1.0.0 numexpr-2.8.6 numpy-1.25.2 packaging-23.1 pydantic-2.3.0 pydantic-core-2.6.3 regex-2023.8.8 requests-2.31.0 tenacity-8.2.3 tiktoken-0.5.0 typing-extensions-4.7.1 typing-inspect-0.9.0 urllib3-2.0.4 yarl-1.9.

For conda, it is failing because of incompatibility:

conda install langchain=0.0.288 tiktoken=0.5.0
UnsatisfiableError: The following specifications were found to be incompatible with each other:

If I don't specify the version if tiktoken, the following is working:
conda install langchain=0.0.288 tiktoken

and the following version of tiktoken is installed:
tiktoken 0.3.3 py39h690f698_0 conda-forge

This is consistent with the specification for conda-forge.

I don't know poetry but it seems that the package that are optional are not checked (I didn't find some proper explanation but here what I can see:

 pip show langchain     
Name: langchain
Version: 0.0.288
Summary: Building applications with LLMs through composability
Home-page: https://github.com/langchain-ai/langchain
Author: 
Author-email: 
License: MIT
Location: /home/jupyter/.conda-env/test_pip/lib/python3.9/site-packages
Requires: aiohttp, async-timeout, dataclasses-json, langsmith, numexpr, numpy, pydantic, PyYAML, requests, SQLAlchemy, tenacity

and these packages are the one that are required:

[tool.poetry.dependencies]
python = ">=3.8.1,<4.0"
pydantic = ">=1,<3"
SQLAlchemy = ">=1.4,<3"
requests = "^2"
PyYAML = ">=5.3"
numpy = "^1"
tenacity = "^8.1.0"
aiohttp = "^3.8.3"
numexpr = "^2.8.4"
langsmith = "~0.0.21"
dataclasses-json = "^0.5.7"
sync-timeout = {version = "^4.0.0", python = "<3.11"}

and the one with {version = "...", optional = true} seems to be ignore

I don't know poetry but it is clear that the package build for pypi and conda-forge don't use the same information. The package on pypi ignore the package that are marked as "optional" while the package on conda-forge enforce them.

I am seeing that Rich is using the same strategy to include option package https://github.com/conda-forge/rich-feedstock/blob/main/recipe/meta.yaml

It seems that the "optional" are using in group when using poetry only. Did conda-forge decided to go and included all dependencies ?

I will test again the latest version of langchain and tiktoken and if it still work, I will do a PR as you suggested.

I am very confuse and don't really why poetry is interpret in different way

@tarrade
Copy link
Contributor Author

tarrade commented Sep 14, 2023

@pavelzw you said that you have a tool that automatically convert the pyproject.toml dependency part for conda-forge package ?
I did something quick and dirty by comparing package version in langchain and conda-feedstock and I am still seeing some differences:
image
I don't see any conda package cassio and the one for marqo is incorrect for example

@pavelzw
Copy link
Member

pavelzw commented Sep 15, 2023

Hm yeah, that's a bit odd...
The last time it (or at least a part of it) worked was in #70.
The relevant line for this "tool" is this one.

inspection: update-grayskull

It uses grayskull under the hood.
Maybe this is a bug in grayskull?

@pavelzw
Copy link
Member

pavelzw commented Sep 15, 2023

$ grayskull pypi langchain
$ cat langchain/meta.yaml | grep marqo
    - marqo >=1.2.4,<2.0.0

So grayskull seems to work but for some reason the bot.inspection: update-grayskull is not doing what it's supposed to...

@pavelzw
Copy link
Member

pavelzw commented Sep 15, 2023

I don't see any conda package cassio

This is because nobody cared yet to port it to conda-forge. Since this is only an optional dependency, this doesn't break this build here.

@tarrade
Copy link
Contributor Author

tarrade commented Sep 15, 2023

PR is here langchain-ai/langchain#10640, if/when approved, I will update this PR

@tarrade tarrade marked this pull request as draft September 19, 2023 08:07
baskaryan added a commit to langchain-ai/langchain that referenced this pull request Sep 29, 2023
…"^0.3.2" => "">=0.3.2,<0.6.0" and python "^3.9" =>">=3.9") (#11006)

- **Description:**
be able to use langchain with other version than tiktoken 0.3.3 i.e
0.5.1
  - **Issue:**
cannot installed the conda-forge version since it applied all optional
dependency:
       conda-forge/langchain-feedstock#85  
replace "^0.3.2" by "">=0.3.2,<0.6.0" and "^3.9" by python=">=3.9"
      Tested with python 3.10, langchain=0.0.288 and tiktoken==0.5.0

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
ShorthillsAI added a commit to shorthills-ai/langchain that referenced this pull request Oct 3, 2023
* Support using async callback handlers with sync callback manager (langchain-ai#10945)

The current behaviour just calls the handler without awaiting the
coroutine, which results in exceptions/warnings, and obviously doesn't
actually execute whatever the callback handler does

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->

* LangServe (langchain-ai#11046)

Adds LangServe package

* Integrate Runnables with Fast API creating Server and a RemoteRunnable
client
* Support multiple runnables for a given server
* Support sync/async/batch/abatch/stream/astream/astream_log on the
client side (using async implementations on server)
* Adds validation using annotations (relying on pydantic under the hood)
-- this still has some rough edges -- e.g., open api docs do NOT
generate correctly at the moment
* Uses pydantic v1 namespace

Known issues: type translation code doesn't handle a lot of types (e.g.,
TypedDicts)

---------

Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>

* Add input/output schemas to runnables (langchain-ai#11063)

This adds `input_schema` and `output_schema` properties to all
runnables, which are Pydantic models for the input and output types
respectively. These are inferred from the structure of the Runnable as
much as possible, the only manual typing needed is
- optionally add type hints to lambdas (which get translated to
input/output schemas)
- optionally add type hint to RunnablePassthrough

These schemas can then be used to create JSON Schema descriptions of
input and output types, see the tests

- [x] Ensure no InputType and OutputType in our classes use abstract
base classes (replace with union of subclasses)
- [x] Implement in BaseChain and LLMChain
- [x] Implement in RunnableBranch
- [x] Implement in RunnableBinding, RunnableMap, RunnablePassthrough,
RunnableEach, RunnableRouter
- [x] Implement in LLM, Prompt, Chat Model, Output Parser, Retriever
- [x] Implement in RunnableLambda from function signature
- [x] Implement in Tool

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->

* Expose loads and dumps in load namespace

* Async support for OpenAIFunctionsAgentOutputParser (langchain-ai#11140)

* milvus collections (langchain-ai#11148)

Description: There was no information about Milvus collections in the
documentation, so I am adding that.
Maintainer: @eyurtsev

* Xata chat memory FIX (langchain-ai#11145)

- **Description:** Changed data type from `text` to `json` in xata for
improved performance. Also corrected the `additionalKwargs` key in the
`messages()` function to `additional_kwargs` to adhere to `BaseMessage`
requirements.
- **Issue:** The Chathisroty.messages() will return {} of
`additional_kwargs`, as the name is wrong for `additionalKwargs` .
  - **Dependencies:**  N/A
  - **Tag maintainer:** N/A
  - **Twitter handle:** N/A

My PR is passing linting and testing before submitting.

* Fixed Typo Error in Update get_started.mdx file by addressing a minor typographical error. (langchain-ai#11154)

Fixed Typo Error in Update get_started.mdx file by addressing a minor
typographical error.

This improvement enhances the readability and correctness of the
notebook, making it easier for users to understand and follow the
demonstration. The commit aims to maintain the quality and accuracy of
the content within the repository.
please review the change at your convenience.

@baskaryan , @hwaking

* Implement better reprs for Runnables

* x

* x

* x

* x

* Fix stop key of TextGen. (langchain-ai#11109)

The key of stopping strings used in text-generation-webui api is
[`stopping_strings`](https://github.com/oobabooga/text-generation-webui/blob/main/api-examples/api-example.py#L51),
not `stop`.
<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->

* LangServe: Clean up init files (langchain-ai#11174)

Clean up init files

* mypy

* Lint

* Lint

* Expose lc_id as a classmethod (langchain-ai#11176)

* Expose LC id as a class method 
* User should not need to know that the last part of the id is the class
name

* Update Bedrock service name to "bedrock-runtime" and model identifiers (langchain-ai#11161)

- **Description:** Bedrock updated boto service name to
"bedrock-runtime" for the InvokeModel and InvokeModelWithResponseStream
APIs. This update also includes new model identifiers for Titan text,
embedding and Anthropic.

Co-authored-by: Mani Kumar Adari <maniadar@amazon.com>

* LangServe: Add release workflow (langchain-ai#11178)

Add release workflow to langserve

* LangServe: Update langchain requirement for publishing (langchain-ai#11186)

Update langchain requirement for publishing

* temporarily skip embedding empty string test (langchain-ai#11187)

* Fix anthropic secret key when passed in via init (langchain-ai#11185)

Fixes anthropic secret key when passed via init

langchain-ai#11182

* add anthropic scheduled tests and unit tests (langchain-ai#11188)

* Rm additional file check for scheduled tests (langchain-ai#11192)

cc @obi1kenobi Causing issues with GHA creds
https://github.com/langchain-ai/langchain/actions/runs/6342674950/job/17228926776

* Add source metadata to OutlookMessageLoader (langchain-ai#11183)

Description: Add "source" metadata to OutlookMessageLoader

This pull request adds the "source" metadata to the OutlookMessageLoader
class in the load method. The "source" metadata is required when
indexing with RecordManager in order to sync the index documents with a
source.

Issue: None

Dependencies: None

Twitter handle: @ATelders

Co-authored-by: Arthur Telders <arthur.telders@roquette.com>

* [OpenSearch] Add Self Query Retriever Support to OpenSearch (langchain-ai#11184)

### Description
Add Self Query Retriever Support to OpenSearch

### Maintainers
@rlancemartin, @eyurtsev, @navneet1v

### Twitter Handle
@OpenSearchProj

Signed-off-by: Naveen Tatikonda <navtat@amazon.com>

* [ElasticsearchStore] Improve migration text to ElasticsearchStore (langchain-ai#11158)

We noticed that as we have been moving developers to the new
`ElasticsearchStore` implementation, we want to keep the
ElasticVectorSearch class still available as developers transition
slowly to the new store.

To speed up this process, I updated the blurb giving them a better
recommendation of why they should use ElasticsearchStore.

* update docs nav (langchain-ai#11146)

* Add langserve version (langchain-ai#11195)

Add langserve version

* [Feat] Add optional client-side encryption to DynamoDB chat history memory (langchain-ai#11115)

**Description:** Added optional client-side encryption to the Amazon
DynamoDB chat history memory with an AWS KMS Key ID using the [AWS
Database Encryption SDK for
Python](https://docs.aws.amazon.com/database-encryption-sdk/latest/devguide/python.html)
**Issue:** langchain-ai#7886
**Dependencies:**
[dynamodb-encryption-sdk](https://pypi.org/project/dynamodb-encryption-sdk/)
**Tag maintainer:**  @hwchase17 
**Twitter handle:** [@jplock](https://twitter.com/jplock/)

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>

* Shared Executor (langchain-ai#11028)

* LLMonitor Callback handler: fix bug (langchain-ai#11128)

Here is a small bug fix for the LLMonitor callback handler. I've also
added user identification capabilities.

* Add support for MongoDB Atlas $vectorSearch vector search (langchain-ai#11139)

Adds support for the `$vectorSearch` operator for
MongoDBAtlasVectorSearch, which was announced at .Local London
(September 26th, 2023). This change maintains breaks compatibility
support for the existing `$search` operator used by the original
integration (langchain-ai#5338) due to
incompatibilities in the Atlas search implementations.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>

* add from_existing_graph to neo4j vector (langchain-ai#11124)

This PR adds the option to create a Neo4jvector instance from existing
graph, which embeds existing text in the database and creates relevant
indices.

* Add `add_graph_documents` support for FalkorDBGraph  (langchain-ai#11122)

Adding `add_graph_documents` support for FalkorDBGraph and extending the
`Neo4JGraph` api so it can support `cypher.py`

* FIx eval prompt (langchain-ai#11087)

**Description:** fixes a common typo in some of the eval criteria.

* Expanded version range for networkx, fixed sample notebook (langchain-ai#11094)

## Description
Expanded the upper bound for `networkx` dependency to allow installation
of latest stable version. Tested the included sample notebook with
version 3.1, and all steps ran successfully.
---------

Co-authored-by: Bagatur <baskaryan@gmail.com>

* docs: Mendable Search Improvements (langchain-ai#11199)

Improvements to the Mendable UI, more accurate responses, and bug fixes.

* Change type annotations from LLMChain to Chain in MultiPromptChain (langchain-ai#11082)

- **Description:** The types of 'destination_chains' and 'default_chain'
in 'MultiPromptChain' were changed from 'LLMChain' to 'Chain'. and
removed variables declared overlapping with the parent class
- **Issue:** When a class that inherits only Chain and not LLMChain,
such as 'SequentialChain' or 'RetrievalQA', is entered in
'destination_chains' and 'default_chain', a pydantic validation error is
raised.
-  -  codes
```
retrieval_chain = ConversationalRetrievalChain(
        retriever=doc_retriever,
        combine_docs_chain=combine_docs_chain,
        question_generator=question_gen_chain,
    )
    
    destination_chains = {
        'retrieval': retrieval_chain,
    }
    
    main_chain = MultiPromptChain(
        router_chain=router_chain,
        destination_chains=destination_chains,
        default_chain=default_chain,
        verbose=True,
    )
```

✅ `make format`, `make lint` and `make test`

* fix: short-circuit black and mypy calls when no changes made (langchain-ai#11051)

Both black and mypy expect a list of files or directories as input.
As-is the Makefile computes a list files changed relative to the last
commit; these are passed to black and mypy in the `format_diff` and
`lint_diff` targets. This is done by way of the Makefile variable
`PYTHON_FILES`. This is to save time by skipping running mypy and black
over the whole source tree.

When no changes have been made, this variable is empty, so the call to
black (and mypy) lacks input files. The call exits with error causing
the Makefile target to error out with:

```bash
$ make format_diff
poetry run black
Usage: black [OPTIONS] SRC ...

One of 'SRC' or 'code' is required.
make: *** [format_diff] Error 1
```

This is unexpected and undesirable, as the naive caller (that's me! 😄 )
will think something else is wrong. This commit smooths over this by
short circuiting when `PYTHON_FILES` is empty.

* Callback integration for Trubrics (langchain-ai#11059)

After contributing to some examples in the
[langsmith-cookbook](https://github.com/langchain-ai/langsmith-cookbook)
with @hinthornw, here is a PR that adds a callback handler to use
LangChain with [Trubrics](https://github.com/trubrics/trubrics-sdk).

* Support add_embeddings for opensearch (langchain-ai#11050)

- **Description:**
      -  Make running integration test for opensearch easy
- Provide a way to use different text for embedding: refer to langchain-ai#11002 for
more of the use case and design decision.
  - **Issue:** N/A
  - **Dependencies:** None other than the existing ones.

* chore: add support for TypeScript code splitting (langchain-ai#11160)


- **Description:** Adds typescript language to `TextSplitter`

---------

Co-authored-by: Jacob Lee <jacoblee93@gmail.com>

* fix trubrics lint issue (langchain-ai#11202)

* SearchApi integration (langchain-ai#11023)

Based on the customers' requests for native langchain integration,
SearchApi is ready to invest in AI and LLM space, especially in
open-source development.

- This is our initial PR and later we want to improve it based on
customers' and langchain users' feedback. Most likely changes will
affect how the final results string is being built.
- We are creating similar native integration in Python and JavaScript.
- The next plan is to integrate into Java, Ruby, Go, and others.
- Feel free to assign @SebastjanPrachovskij as a main reviewer for any
SearchApi-related searches. We will be glad to help and support
langchain development.

* Synthetic Data generation (langchain-ai#9472)

---------

Co-authored-by: William Fu-Hinthorn <13333726+hinthornw@users.noreply.github.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>

* LangServe: Relax requirements (langchain-ai#11198)

Relax requirements

* Add last_edited_time and created_time props to NotionDBLoader (langchain-ai#11020)

# Description

Adds logic for NotionDBLoader to correctly populate `last_edited_time`
and `created_time` fields from [page
properties](https://developers.notion.com/reference/page#property-value-object).

There are no relevant tests for this code to be updated.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>

* `LlamaCppEmbeddings`: adds `verbose` parameter, similar to `llms.LlamaCpp` class (langchain-ai#11038)

## Description

As of now, when instantiating and during inference, `LlamaCppEmbeddings`
outputs (a lot of) verbose when controlled from Langchain binding - it
is a bit annoying when computing the embeddings of long documents, for
instance.

This PR adds `verbose` for `LlamaCppEmbeddings` objects to be able
**not** to print the verbose of the model to `stderr`. It is natively
supported by `llama-cpp-python` and directly passed to the library – the
PR is hence very small.

The value of `verbose` is `True` by default, following the way it is
defined in [`LlamaCpp` (`llamacpp.py`
#L136-L137)](https://github.com/langchain-ai/langchain/blob/c87e9fb2ce0ae617e3b2edde52421c80adef54cc/libs/langchain/langchain/llms/llamacpp.py#L136-L137)

## Issue

_No issue linked_

## Dependencies

_No additional dependency needed_

## To see it in action

```python
from langchain.embeddings import LlamaCppEmbeddings

MODEL_PATH = "<path_to_gguf_file>"

if __name__ == "__main__":
    llm_embeddings = LlamaCppEmbeddings(
        model_path=MODEL_PATH,
        n_gpu_layers=1,
        n_batch=512,
        n_ctx=2048,
        f16_kv=True,
        verbose=False,
    )
```

Co-authored-by: Bagatur <baskaryan@gmail.com>

* Support new version of tiktoken that are working with langchain (tag "^0.3.2" => "">=0.3.2,<0.6.0" and python "^3.9" =>">=3.9") (langchain-ai#11006)

- **Description:**
be able to use langchain with other version than tiktoken 0.3.3 i.e
0.5.1
  - **Issue:**
cannot installed the conda-forge version since it applied all optional
dependency:
       conda-forge/langchain-feedstock#85  
replace "^0.3.2" by "">=0.3.2,<0.6.0" and "^3.9" by python=">=3.9"
      Tested with python 3.10, langchain=0.0.288 and tiktoken==0.5.0

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>

* Typo fix to MathpixPDFLoader - changed processed_file_format default … (langchain-ai#10960)

…from mmd to md. langchain-ai#7282

<!-- 
- **Description:** minor fix to a breaking typo - MathPixPDFLoader
processed_file_format is "mmd" by default, doesn't work, changing to
"md" fixes the issue,
- **Issue:** 7282
(langchain-ai#7282),
  - **Dependencies:** none,
  - **Tag maintainer:** @hwchase17,
  - **Twitter handle:** none
 -->

Co-authored-by: jare0530 <7915+jare0530@users.noreply.ghe.oculus-rep.com>

* Fix web-base loader (langchain-ai#11135)

Fix initialization

langchain-ai#11095

* Updated `LocalAIEmbeddings` docstring to better explain why `openai` (langchain-ai#10946)

Fixes my misgivings in
langchain-ai#10912

* Add support for project metadata in run_on_dataset (langchain-ai#11200)

* Add from_embeddings for opensearch (langchain-ai#10957)

* Skip for py3.8

* Skip in py3.8

* skip more

* Even more

* Enable creating Tools from any Runnable

* Fix invocation

* Lint

* Lint

* Add RunnableGenerator

* Add tests

* Lint

* Add a streaming json parser

* Implement str one

* WIP Add tests§

* Implement diff

* Implement diff

* Backwards compat

* Clean warnings: replace type with isinstance and fix syntax (langchain-ai#11219)

Clean warnings: replace type with `isinstance` and fix on notebook
syntax syntax

* Add async tests and comments

* Update fireworks features (langchain-ai#11205)

Description
* Update fireworks feature on web page

Issue - Not applicable
Dependencies - None
Tag maintainer - @baskaryan

* mongodb doc loader init (langchain-ai#10645)

- **Description:** A Document Loader for MongoDB
  - **Issue:** n/a
  - **Dependencies:** Motor, the async driver for MongoDB
  - **Tag maintainer:** n/a
  - **Twitter handle:** pigpenblue

Note that an initial mongodb document loader was created 4 months ago,
but the [PR ](langchain-ai#4285
never pulled in. @leo-gan had commented on that PR, but given it is
extremely far behind the master branch and a ton has changed in
Langchain since then (including repo name and structure), I rewrote the
branch and issued a new PR with the expectation that the old one can be
closed.

Please reference that old PR for comments/context, but it can be closed
in favor of this one. Thanks!

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>

* Suppress warnings in interactive env that stem from tab completion (langchain-ai#11190)

Suppress warnings in interactive environments that can arise from users 
relying on tab completion (without even using deprecated modules).

jupyter seems to filter warnings by default (at least for me), but
ipython surfaces them all

* OpenAI gpt-3.5-turbo-instruct cost information (langchain-ai#11218)

Added pricing info for `gpt-3.5-turbo-instruct` for OpenAI and Azure
OpenAI.

Co-authored-by: Attila Tőkés <atokes@rws.com>

* Fix typo in gradient.ipynb (langchain-ai#11206)

Enviroment -> Environment

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->

* Make test deterministic

* bump 305 (langchain-ai#11224)

* Using langchain input types (langchain-ai#11204)

Using langchain input type

* Make tests stricter, remove old code, fix up pydantic import when using v2 (langchain-ai#11231)

Make tests stricter, remove old code, fix up pydantic import when using v2 (langchain-ai#11231)

* Combine with existing json output parsers

* Lint

* Keep exceptions when not in streaming mode

* Update json.py

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>

* Update json.py

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>

* Lint

* Remove flawed test

- It is not possible to access properties on classes, only on instances, therefore this test is not something we can implement

* Implement RunnablePassthrough.assign(...) (langchain-ai#11222)

Passes through dict input and assigns additional keys

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->

* Add type to message chunks (langchain-ai#11232)

* Ignore aadd (langchain-ai#11235)

* fix code injection vuln (langchain-ai#11233)

- **Description:** Fix a code injection vuln by adding one more keyword
into the filtering list
  - **Issue:** N/A
  - **Dependencies:** N/A
  - **Tag maintainer:** 
  - **Twitter handle:**

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>

* Bump deps in langserve (langchain-ai#11234)

Bump deps in langserve lockfile

* Update DeepSparse LLM (langchain-ai#11236)

**Description:** Adds streaming and many more sampling parameters to the
DeepSparse interface

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>

* docs: `integrations/memory` consistency (langchain-ai#10255)

- updated titles and descriptions of the `integrations/memory` notebooks
into consistent and laconic format;
- removed
`docs/extras/integrations/memory/motorhead_memory_managed.ipynb` file as
a duplicate of the
`docs/extras/integrations/memory/motorhead_memory.ipynb`;
- added `integrations/providers` Integration Cards for `dynamodb`,
`motorhead`.
- updated `integrations/providers/redis.mdx` with links
- renamed several notebooks; updated `vercel.json` to reroute new names.

* docs: `document_transformers` consistency (langchain-ai#10467)

- Updated `document_transformers` examples: titles, descriptions, links
- Added `integrations/providers` for missed document_transformers

* docs: updated `YouTube` and `tutorial` video links (langchain-ai#10897)

updated `YouTube` and `tutorial` videos with new links.
Removed couple of duplicates.
Reordered several links by view counters
Some formatting: emphasized the names of products

* minor fix: remove redundant code from OpenAIFunctionsAgent (langchain-ai#11245)

minor fix: remove redundant code from OpenAIFunctionsAgent (langchain-ai#11245)

* rename repo namespace to langchain-ai (langchain-ai#11259)

### Description
renamed several repository links from `hwchase17` to `langchain-ai`.

### Why
I discovered that the README file in the devcontainer contains an old
repository name, so I took the opportunity to rename the old repository
name in all files within the repository, excluding those that do not
require changes.

### Dependencies
none

### Tag maintainer
@baskaryan

### Twitter handle
[kzk_maeda](https://twitter.com/kzk_maeda)

* Fix typo in docstring (langchain-ai#11256)

Description : Remove meaningless 's' in docstring

* Create new RunnableSerializable class in preparation for configurable runnables

- Also move RunnableBranch to its own file

* Lint

* Lint

* Lint

* Lint

* Move RunnableWithFallbacks to its own file

* Lint

* Lint

* Lint

* Update quickstart.mdx to add backtick after `ChatMessages`  (langchain-ai#11241)

While going through the documentation I found this small issue and
wanted to contribute!

<!-- Thank you for contributing to LangChain! -->

* Remove extra spaces (langchain-ai#11283)

### Description
When I was reading the document, I found that some examples had extra
spaces and violated "Unexpected spaces around keyword / parameter equals
(E251)" in pep8. I removed these extra spaces.
  
### Tag maintainer
@eyurtsev 
### Twitter handle
[billvsme](https://twitter.com/billvsme)

* Add base docker image and ci script for building and pushing (langchain-ai#10927)

* bump 306 (langchain-ai#11289)

* Small changes to runnable docs (langchain-ai#11293)

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->

* Add Google GitHub Action creds file to gitignore. (langchain-ai#11296)

Should resolve the issue here:
https://github.com/langchain-ai/langchain/actions/runs/6342767671/job/17229204508#step:7:36

After this merges, we can revert
langchain-ai#11192

* Add pending deprecation warning (langchain-ai#11133)

This PR uses 2 dedicated LangChain warnings types for deprecations
(mirroring python's built in deprecation and pending deprecation
warnings).

These deprecation types are unslienced during initialization in
langchain achieving the same default behavior that we have with our
current warnings approach. However, because these warnings have a
dedicated type, users will be able to silence them selectively (I think
this is strictly better than our current handling of warnings).

The PR adds a deprecation warning to llm symbolic math.

---------

Co-authored-by: Predrag Gruevski <2348618+obi1kenobi@users.noreply.github.com>

* Make numexpr optional (langchain-ai#11049)

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>

* Bump min version of numexpr (langchain-ai#11302)

Bump min version

* Bedrock scheduled tests (langchain-ai#11194)

* Fix closing bracket in length-based selector snippet (langchain-ai#11294)

**Description:**

Fix a forgotten closing bracket in the length-based selector snippet

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>

* Fix line break in docs imports (langchain-ai#11270)

It is just a straightforward docs fix.

* add LLMBashChain to experimental (langchain-ai#11305)

Add LLMBashChain to experimental

* Add .configurable_fields() and .configurable_alternatives() to expose fields of a Runnable to be configured at runtime (langchain-ai#11282)

* Upgrade `langchain` dependency versions to resolve dependabot alerts. (langchain-ai#11307)

* Add scoring chain (langchain-ai#11123)

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->

* Make Google PaLM classes serialisable (langchain-ai#11121)

Similarly to Vertex classes, PaLM classes weren't marked as
serialisable. Should be working fine with LangSmith.

---------

Co-authored-by: Erick Friis <erick@langchain.dev>

* Mark Vertex AI classes as serialisable (langchain-ai#10484)

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
- Tag maintainer: for a quicker response, tag the relevant maintainer
(see below),
- Twitter handle: we announce bigger features on Twitter. If your PR
gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. These live is docs/extras
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->

---------

Co-authored-by: Erick Friis <erick@langchain.dev>

* Adds Tavily Search API retriever (langchain-ai#11314)

@baskaryan @efriis

* Update clarifai.mdx

---------

Signed-off-by: Naveen Tatikonda <navtat@amazon.com>
Co-authored-by: Nuno Campos <nuno@langchain.dev>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>
Co-authored-by: William FH <13333726+hinthornw@users.noreply.github.com>
Co-authored-by: Apurv Agarwal <apoorvagarwal00@gmail.com>
Co-authored-by: Nan LI <linanenv@gmail.com>
Co-authored-by: Nuno Campos <nuno@boringbits.io>
Co-authored-by: Akio Nishimura <akionux@gmail.com>
Co-authored-by: mani2348 <itsmanikumar@gmail.com>
Co-authored-by: Mani Kumar Adari <maniadar@amazon.com>
Co-authored-by: Arthur Telders <72456061+ATelders@users.noreply.github.com>
Co-authored-by: Arthur Telders <arthur.telders@roquette.com>
Co-authored-by: Naveen Tatikonda <navtat@amazon.com>
Co-authored-by: Joseph McElroy <joseph.mcelroy@elastic.co>
Co-authored-by: Justin Plock <jplock@users.noreply.github.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Hugues <me@hugh.sh>
Co-authored-by: Noah Stapp <noah@noahstapp.com>
Co-authored-by: Tomaz Bratanic <bratanic.tomaz@gmail.com>
Co-authored-by: Guy Korland <gkorland@gmail.com>
Co-authored-by: Piotr Mardziel <piotrm@gmail.com>
Co-authored-by: Piyush Jain <piyushjain@duck.com>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: Michael Kim <59414764+xcellentbird@users.noreply.github.com>
Co-authored-by: Michael Landis <michael@momentohq.com>
Co-authored-by: Jeff Kayne <43336277+jeffkayne@users.noreply.github.com>
Co-authored-by: Kenneth Choe <kenneth.choe@gmail.com>
Co-authored-by: Fynn Flügge <fynnfluegge@gmx.de>
Co-authored-by: Jacob Lee <jacoblee93@gmail.com>
Co-authored-by: Donatas Remeika <dremeika@users.noreply.github.com>
Co-authored-by: PaperMoose <rbrandt810@gmail.com>
Co-authored-by: Noah Czelusta <83324596+swimninja247@users.noreply.github.com>
Co-authored-by: Clément Sicard <33360172+ClementSicard@users.noreply.github.com>
Co-authored-by: Dr. Fabien Tarrade <tarrade@users.noreply.github.com>
Co-authored-by: jreinjr <jason.w.reinhardt@gmail.com>
Co-authored-by: jare0530 <7915+jare0530@users.noreply.ghe.oculus-rep.com>
Co-authored-by: James Braza <jamesbraza@gmail.com>
Co-authored-by: Cynthia Yang <zixinyang92@gmail.com>
Co-authored-by: Jon Saginaw <saginawj@users.noreply.github.com>
Co-authored-by: Attila Tőkés <62890262+attila-tokes@users.noreply.github.com>
Co-authored-by: Attila Tőkés <atokes@rws.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
Co-authored-by: Haozhe <17514803+hazzel-cn@users.noreply.github.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
Co-authored-by: Leonid Ganeline <leo.gan.57@gmail.com>
Co-authored-by: Dayuan Jiang <34411969+DayuanJiang@users.noreply.github.com>
Co-authored-by: Kazuki Maeda <kzk.maeda0711@gmail.com>
Co-authored-by: Yeonji-Lim <57888020+Yeonji-Lim@users.noreply.github.com>
Co-authored-by: James Odeyale <jamesodeyale01@gmail.com>
Co-authored-by: zhengkai <994171686@qq.com>
Co-authored-by: Predrag Gruevski <2348618+obi1kenobi@users.noreply.github.com>
Co-authored-by: Oleg Sinavski <2086260+olegsinavski@users.noreply.github.com>
Co-authored-by: João Carabetta <joao.carabetta@gmail.com>
Co-authored-by: CG80499 <94075036+CG80499@users.noreply.github.com>
Co-authored-by: David Duong <david@duong.cz>
Co-authored-by: Erick Friis <erick@langchain.dev>
@tarrade
Copy link
Contributor Author

tarrade commented Oct 3, 2023

Hi @pavelzw ,

The modification for tiktoken was introduced in langcain v0.0.305 https://github.com/langchain-ai/langchain/releases/tag/v0.0.305

Now, I checked langchain-feedstock for v0.0.306 and I don't see the modification being propagated properly:
tiktoken >=0.3.2,<0.4.0
https://github.com/conda-forge/langchain-feedstock/blob/main/recipe/meta.yaml

I am happy to do a PR to fix it manually but since you told me it should be done automatically by getting the info directly from libs/langchain/pyproject.toml just wondering if something wrong happen in the last automatic PR

@pavelzw
Copy link
Member

pavelzw commented Oct 3, 2023

Thanks for keeping track of this @tarrade! Yeah, in a perfect world, this should have gotten updated automatically by the bot but for some reason, this is not the case...
Happy to merge a PR that fixes the tiktoken pinnings.

recipe/meta.yaml Outdated Show resolved Hide resolved
@pavelzw pavelzw marked this pull request as ready for review October 3, 2023 14:23
@pavelzw pavelzw changed the title Fix issue with upper limit for tiktoken 0.5.0 instead of 0.4.0 Fix tiktoken pinning Oct 3, 2023
@pavelzw pavelzw added the automerge Merge the PR when CI passes label Oct 3, 2023
@pavelzw
Copy link
Member

pavelzw commented Oct 3, 2023

Thanks for the PR @tarrade. If you find other discrepancies between this feedstock and the pyproject.toml from upstream, feel free to create another PR here.

@github-actions github-actions bot merged commit ff3cf3f into conda-forge:main Oct 3, 2023
3 checks passed
@github-actions
Copy link
Contributor

github-actions bot commented Oct 3, 2023

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

  • linter: passed
  • azure: passed

Thus the PR was passing and merged! Have a great day!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge Merge the PR when CI passes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants