Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

epic: Jan supports multiple CPU/GPU Hardware optimizations #905

Closed
7 of 8 tasks
tikikun opened this issue Dec 7, 2023 · 8 comments
Closed
7 of 8 tasks

epic: Jan supports multiple CPU/GPU Hardware optimizations #905

tikikun opened this issue Dec 7, 2023 · 8 comments
Assignees
@hiro-v @hiento09
Labels
engineering: Jan Inference Layer Jan can serve models locally: with correct data structs, APIs, multi-inference engines, multi-model roadmap: Jan Home Server Jan can be run via Helm or Docker server-side roadmap: Observability System Monitor, logs, observability and hardware insights type: epic A major feature or initiative
Milestone
v0.4.7

Comments

@tikikun
Copy link
Contributor

tikikun commented Dec 7, 2023 •
edited by Van-QA

Problem

Windows users encounter elevated CPU usage (ex: #887), prompting the need for hardware optimizations in Nitro/Jan across various Windows architectures (CPU, GPU).

Success Criteria

  • Achieve compatibility for Jan on Windows with 16GB of RAM, accommodating systems lacking a GPU.
  • Enable Jan to operate seamlessly on Windows systems featuring Nvidia GPUs.

Out-of-scope

Not support Intel, AMD GPU for now, although we still detect these device.

Tasklist

Design & FE

Implementation

Additional context
@tikikun tikikun added the type: epic A major feature or initiative label Dec 7, 2023
@tikikun tikikun self-assigned this Dec 7, 2023
@tikikun
Copy link
Contributor Author

tikikun commented Dec 7, 2023

https://www.npmjs.com/package/systeminformation

@tikikun
Copy link
Contributor Author

tikikun commented Dec 8, 2023 •
edited

Intel optimization:
Instructions set:
AVX2 - Consumer grade
AVX512 - Before gen 12 or Consumer grade
AVX_VNNI - Consumer grade with OneAPI
AVX512_VNNI - Server grade

@tikikun
Copy link
Contributor Author

tikikun commented Dec 8, 2023

Windows WSL
#912

oneAPI for Windows
#911

AMD GPU for Windows
#913

AMD CPU for Windows
#914

Intel GPU (maybe crossing with oneAPI)
#915

@dan-jan dan-jan changed the title epic: Hardware optimizations epic: Jan supports multiple CPU/GPU Hardware optimizations Dec 9, 2023
@dan-jan
Copy link
Contributor

dan-jan commented Dec 9, 2023

@tikikun I've renamed this Epic and added more detail to the "Top" post, please take a look.

I will be creating a separate "Jan should work on Windows" epic, that will be more Product-focused (there will be some overlap)

@dan-jan dan-jan added this to the Jan on Windows milestone Dec 11, 2023
@dan-jan dan-jan changed the title epic: Jan supports multiple CPU/GPU Hardware optimizations feat: Jan supports multiple CPU/GPU Hardware optimizations Dec 12, 2023
@0xSage 0xSage added the engineering: Jan Inference Layer Jan can serve models locally: with correct data structs, APIs, multi-inference engines, multi-model label Dec 18, 2023
@0xSage 0xSage changed the title feat: Jan supports multiple CPU/GPU Hardware optimizations epic: Jan supports multiple CPU/GPU Hardware optimizations Jan 5, 2024
@imtuyethan imtuyethan assigned tikikun and unassigned tikikun Jan 17, 2024
@imtuyethan imtuyethan added the roadmap: Observability System Monitor, logs, observability and hardware insights label Jan 17, 2024
@imtuyethan
Copy link
Contributor

imtuyethan commented Jan 17, 2024 •
edited by hiro-v

Archive the original comment of Alan:

Problem

Success Criteria

  • Jan can run on Windows with 16gb of RAM, no GPU (both Intel and AMD processors)
  • Jan can run on Windows with Intel, AMD, Nvidia GPUs

Tasks

Additional context

@imtuyethan imtuyethan mentioned this issue Jan 29, 2024
Closed
@imtuyethan imtuyethan added the roadmap: Jan Home Server Jan can be run via Helm or Docker server-side label Jan 30, 2024
@hiro-v hiro-v assigned hiento09 and unassigned tikikun Feb 16, 2024
@dan-jan
Copy link
Contributor

dan-jan commented Feb 16, 2024

@Van-QA to decompose this into the engineering: Jan Inference Layer roadmap

@Van-QA
Copy link
Contributor

Van-QA commented Feb 16, 2024

Only 1 feat and 1 epic left, I will close this epic once the remaining ticket #915 completed.
On the other hand, I will follow up with the epic #913 in the engineering: Jan Inference Layer roadmap
image

@hiro-v hiro-v mentioned this issue Feb 16, 2024
Merged
10 tasks
@Van-QA
Copy link
Contributor

Van-QA commented Feb 19, 2024

as discussed with @hiro-v, the remaining feat #915 is nice to have, and will handled in the follow up epic #2092

@Van-QA Van-QA modified the milestones: Jan Windows MVP, v0.4.7 Feb 19, 2024
@hiro-v hiro-v mentioned this issue Feb 20, 2024
Closed
3 tasks
@hiro-v hiro-v closed this as completed Feb 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hiro-v hiro-v

@hiento09 hiento09

Labels
engineering: Jan Inference Layer Jan can serve models locally: with correct data structs, APIs, multi-inference engines, multi-model roadmap: Jan Home Server Jan can be run via Helm or Docker server-side roadmap: Observability System Monitor, logs, observability and hardware insights type: epic A major feature or initiative
Projects
Jan Project
Archived in project
Milestone
v0.4.7
Development

No branches or pull requests
7 participants
@tikikun @hiro-v @Van-QA @0xSage @imtuyethan @dan-jan @hiento09