Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AnyscaleLLM #447

Merged
merged 12 commits into from
Mar 20, 2024
Merged

Add AnyscaleLLM #447

merged 12 commits into from
Mar 20, 2024

Conversation

davidberenstein1957
Copy link
Member

Closes #446

import time

from distilabel.llm.anyscale import AnyscaleLLM

if __name__ == "__main__":

    start_time = time.time()
    llm = AnyscaleLLM(api_key="esecret_*****")
    llm.load()
    output = llm.generate(
        inputs=[
            [
                {
                    "role": "user",
                    "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
                },
            ]
        ]
    )
    print("--- %s seconds ---" % (time.time() - start_time))
    exit()

@davidberenstein1957 davidberenstein1957 added this to the 1.0.0 milestone Mar 19, 2024
@davidberenstein1957 davidberenstein1957 self-assigned this Mar 19, 2024
@davidberenstein1957 davidberenstein1957 linked an issue Mar 19, 2024 that may be closed by this pull request
@davidberenstein1957 davidberenstein1957 changed the title 446 anyscalellm Add AnyscaleLLM Mar 19, 2024
@davidberenstein1957 davidberenstein1957 marked this pull request as ready for review March 19, 2024 21:15
Copy link
Member

@alvarobartt alvarobartt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @davidberenstein1957!

src/distilabel/llm/anyscale.py Outdated Show resolved Hide resolved
src/distilabel/llm/anyscale.py Outdated Show resolved Hide resolved
src/distilabel/llm/anyscale.py Outdated Show resolved Hide resolved
src/distilabel/llm/anyscale.py Outdated Show resolved Hide resolved
src/distilabel/llm/anyscale.py Outdated Show resolved Hide resolved
src/distilabel/llm/anyscale.py Outdated Show resolved Hide resolved
tests/unit/llm/test_anyscale.py Outdated Show resolved Hide resolved
tests/unit/llm/test_anyscale.py Outdated Show resolved Hide resolved
@alvarobartt alvarobartt mentioned this pull request Mar 20, 2024
davidberenstein1957 and others added 7 commits March 20, 2024 09:34
Co-authored-by: Alvaro Bartolome <alvaro@argilla.io>
Co-authored-by: Alvaro Bartolome <alvaro@argilla.io>
Co-authored-by: Alvaro Bartolome <alvaro@argilla.io>
Co-authored-by: Alvaro Bartolome <alvaro@argilla.io>
Copy link
Member

@alvarobartt alvarobartt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, great work! Thanks for also removing the model defaults, feel free to merge core-refactor in your branch, delete the load method from AnyscaleLLM and then we can merge, thanks again!

Anyscale LLM implementation running the async API client of OpenAI because of duplicate API behavior.

Attributes:
model: the model name to use for the LLM, e.g., `google/gemma-7b-it`. [Supported models](https://docs.endpoints.anyscale.com/text-generation/supported-models/google-gemma-7b-it).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
model: the model name to use for the LLM, e.g., `google/gemma-7b-it`. [Supported models](https://docs.endpoints.anyscale.com/text-generation/supported-models/google-gemma-7b-it).
model: the model name to use for the LLM, e.g., `google/gemma-7b-it`. [Supported models](https://docs.endpoints.anyscale.com/text-generation/supported-models).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly this is not defined haha

Comment on lines 32 to 54
def load(
self, api_key: Optional[str] = None, base_url: Optional[str] = None
) -> None:
"""Loads the `AsyncOpenAI` client to benefit from async requests."""

try:
from openai import AsyncOpenAI
except ImportError as ie:
raise ImportError(
"OpenAI Python client is not installed which is required for `AnyscaleLLM`. Please install it using"
" `pip install openai`."
) from ie

self.api_key = self._handle_api_key_value(
self_value=self.api_key, load_value=api_key, env_var="OPENAI_API_KEY"
)
self.base_url = base_url or self.base_url

self._aclient = AsyncOpenAI(
api_key=self.api_key.get_secret_value(),
base_url=self.base_url,
max_retries=6,
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to define this function, as there's already an update on core-refactor, so feel free to merge the changes from that branch and remove this function instead 👍🏻

@davidberenstein1957 davidberenstein1957 merged commit 87afbc5 into core-refactor Mar 20, 2024
4 checks passed
@davidberenstein1957 davidberenstein1957 deleted the 446-anyscalellm branch March 20, 2024 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Add AnyScaleLLM integration
2 participants