Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Cloudflare Workers #2

Open
hellojwilde opened this issue Jun 15, 2023 · 0 comments
Open

Add support for Cloudflare Workers #2

hellojwilde opened this issue Jun 15, 2023 · 0 comments

Comments

@hellojwilde
Copy link
Owner

Background

There was a fantastic idea from a Redditor about making EnergeticAI run well in Cloudflare Workers, so you can have super fast inference at the edge, without the need to distribute your model weights:

https://www.reddit.com/r/tensorflow/comments/1493uoq/comment/jo6axc9/?utm_source=reddit&utm_medium=web2x&context=3

Goal

This task is to add support for Cloudflare Workers to EnergeticAI.

Approach

Given that Cloudflare Workers have even more restrictive bundle limits than AWS Lambda, I suspect the way to do this would be to distribute sharded model weights in Cloudflare KV, and then fetch from that in parallel on function invocation. On paper at least KV values should be colocated with the functions enough that this should be fast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant