Skip to content

Conversation

@endocrimes
Copy link

@endocrimes endocrimes commented Sep 19, 2023

This is an initial pass at adding LLM Inferencing support to they Python SDK. Demo in the cloud: https://spin-py-hello-llm-2zjy1qai.fermyon.app

Quirks:

  • There isn't a "good" way to expose a rust enum to python, so we make the model a string and map it to internal sdk constants / the other associated enum type

endocrimes and others added 5 commits September 15, 2023 10:49
Signed-off-by: Danielle Lancashire <dani@builds.terrible.systems>
Signed-off-by: Danielle Lancashire <dani@builds.terrible.systems>
Signed-off-by: Danielle Lancashire <dani@builds.terrible.systems>
Now that we've updated to the latest Rust Spin SDK, which uses a newer
`wit-bindgen`, we need to call `__wasm_call_ctors` explicitly in order to
initialize `wasi-libc` (e.g. grab environment variables, etc.).  Previously,
this was done implicitly, but that caused other problems, so `wit-bindgen` does
it explicitly, which has the side effect of telling `wasm-ld` _not_ to do it
implicitly.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Signed-off-by: Danielle Lancashire <dani@builds.terrible.systems>
@endocrimes endocrimes requested a review from dicej September 19, 2023 14:52
Signed-off-by: Danielle Lancashire <dani@builds.terrible.systems>
Copy link
Collaborator

@dicej dicej left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for doing this!

BTW, this is how wasmtime-py and componentize-py would represent a variant type like InferencingModel in Python:

from typing import Union
from dataclasses import dataclass

@dataclass
class InferencingModelLlama2Chat:
    pass

@dataclass
class InferencingModelCodellamaInstruct:
    pass

@dataclass
class InferencingModelOther:
    value: str

InferencingModel = Union[InferencingModelLlama2Chat, InferencingModelCodellamaInstruct, InferencingModelOther]

I think using a string in this case is fine, but for future reference the above is also an option.

@endocrimes
Copy link
Author

huh til about dataclasses! (I haven't seriously python'd in a very long time 😅, and the pyo3 issues were unhelpful 😂 )

@endocrimes endocrimes merged commit 7568c95 into main Sep 19, 2023
@endocrimes endocrimes deleted the dani/wip-llm-support branch September 19, 2023 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants