Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SpeziLLMFog that performs dynamic LLM job dispatch to fog nodes #52

Merged
merged 18 commits into from
Mar 28, 2024

Conversation

philippzagar
Copy link
Member

@philippzagar philippzagar commented Mar 26, 2024

Add SpeziLLMFog that performs dynamic LLM job dispatch to fog nodes

♻️ Current situation & Problem

Currently, SpeziLLM doesn't support a privacy friendly execution of LLM inference jobs within the local network.

⚙️ Release Notes

  • Add SpeziLLMFog that performs dynamic LLM job dispatch to fog nodes.
  • Include a FogNode component that easily starts up a complete fog node ready to perform LLM inference requests via docker compose.
  • Introduce custom LLMContext type that holds the internal state of the LLMSession, instead of relying on the SpeziChat models.

📚 Documentation

Added proper readmes, docs, and in-line comments

✅ Testing

Manual testing (for now)

📝 Code of Conduct & Contributing Guidelines

By submitting creating this pull request, you agree to follow our Code of Conduct and Contributing Guidelines:

@philippzagar philippzagar added the enhancement New feature or request label Mar 26, 2024
@philippzagar philippzagar self-assigned this Mar 26, 2024
@philippzagar philippzagar changed the title Add SpeziLLMFog that includes dynamic LLM job dispatch to jobs within the fog layer Add SpeziLLMFog that performs dynamic LLM job dispatch to fog nodes Mar 26, 2024
@philippzagar
Copy link
Member Author

@PSchmiedmayer The Markdown link check is supposed to fail, as I'm already using SPM docs that are published as soon as this PR is merged.

Copy link

codecov bot commented Mar 27, 2024

Codecov Report

Attention: Patch coverage is 34.86239% with 213 lines in your changes are missing coverage. Please review.

Project coverage is 32.03%. Comparing base (dc37b91) to head (6629605).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #52      +/-   ##
==========================================
+ Coverage   30.22%   32.03%   +1.82%     
==========================================
  Files          66       67       +1     
  Lines        2893     2932      +39     
==========================================
+ Hits          874      939      +65     
+ Misses       2019     1993      -26     
Files Coverage Δ
Sources/SpeziLLM/Helpers/LLMContext+Init.swift 0.00% <ø> (ø)
Sources/SpeziLLM/LLMSession.swift 42.86% <ø> (ø)
Sources/SpeziLLM/Mock/LLMMockSession.swift 80.00% <100.00%> (ø)
Sources/SpeziLLM/Models/LLMError.swift 0.00% <ø> (ø)
...rces/SpeziLLM/Models/LLMState+OperationState.swift 90.00% <ø> (ø)
Sources/SpeziLLM/Models/LLMState.swift 52.64% <ø> (ø)
Sources/SpeziLLM/Views/LLMChatView.swift 92.11% <100.00%> (+0.44%) ⬆️
Sources/SpeziLLM/Views/LLMChatViewSchema.swift 100.00% <ø> (ø)
...ocal/Configuration/LLMLocalContextParameters.swift 0.00% <ø> (ø)
...eziLLMLocal/Configuration/LLMLocalParameters.swift 0.00% <ø> (ø)
... and 25 more

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc37b91...6629605. Read the comment docs.

@philippzagar
Copy link
Member Author

philippzagar commented Mar 27, 2024

@PSchmiedmayer Please adjust the merge checks of the repo, as I added additional testing jobs for iPad, Vision, and macOS

Copy link
Member

@PSchmiedmayer PSchmiedmayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for all the work that went into this @philippzagar; greatly appreciated!

It looks great, happy to see this merged and completing the major LLM features we wanted to implement 🚀

FogNode/README.md Show resolved Hide resolved
FogNode/avahi/Dockerfile Outdated Show resolved Hide resolved
FogNode/avahi/Dockerfile-Sidecar Outdated Show resolved Hide resolved
FogNode/docker-compose.dev.yml Outdated Show resolved Hide resolved
Sources/SpeziLLMFog/LLMFogPlatform.swift Outdated Show resolved Hide resolved
Sources/SpeziLLMFog/LLMFogSession+Setup.swift Show resolved Hide resolved
@philippzagar
Copy link
Member Author

@PSchmiedmayer Please merge the PR, the failure of the markdown check is intended as I'm already referencing soon-to-be-published SPI docs in the README etc. :)
Also, please adjust the merge checks of the repo as I adjusted the CI jobs to build on all supported platforms in DEBUG and RELEASE mode

@PSchmiedmayer PSchmiedmayer merged commit d0d1834 into main Mar 28, 2024
18 of 19 checks passed
@PSchmiedmayer PSchmiedmayer deleted the feat/fog-layer branch March 28, 2024 19:01
@PSchmiedmayer
Copy link
Member

Thank you @philippzagar for the PR and all the advancements! Thanks for adding all the additional automated tests! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants