Skip to content

Andthattoo/migration #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
andthattoo merged 63 commits into from
Oct 19, 2025
Merged

Andthattoo/migration #2

andthattoo merged 63 commits into from
Oct 19, 2025

Conversation

@andthattoo
Copy link
Member

@andthattoo andthattoo commented Oct 19, 2025 •
edited
Loading

Migration from initial repo.

Workflow:
Run shards + API via CLI.

  • Discovery as subprocess for memory bloat prevention during profile
  • Add Streaming gRPC
  • Move Embedding + LM head to start|end shards
  • Add Shard config class
  • Use compression/ folder for compression
  • Bump MLX to 0.29.2
  • Submodule updates
  • Cleaner logs
  • Fixed serialization
  • Repacking for offloaded layers

erhant and others added 30 commits October 5, 2025 18:39
@erhant
utils and protos
42d039f
@erhant
api and shard inits
242cb8c
@erhant
migr complete, todo testing
100d165
@erhant
some import fixes
87f35b5
@erhant
topo fixes, more testing and docs
46f0547
@erhant
first completed flow
20bf053
@erhant
fix next service seleciton w.r.t k
5623526
@erhant
add script for loading model at once
96806b0
@erhant
change back next_service from List[str] to str
3b1b6d9
@erhant
add topology GETter
ff85090
@erhant
small fix
852cc86
@erhant
unified topology type
74724bc
@erhant
add error handle for load model
2944165
@erhant
bump dnet
aa5acd7
@erhant
disable timeout
70705c6
@erhant
add downloader scirpt
a7cc148
@erhant
bump dnet-p2p
2c7b87b
@erhant
docfix
b5bc45e
@erhant
format fixes, slight rfks, bump dnet-p2p
d473f57
@erhant
better model sharing between api and shard
d49d985
@erhant
added tb-based ordering for solver, added typings, bump dnet-p2p with…
f1952ae 
… fixes
@erhant
more typing and serialization fixes
d405e3b
@erhant
some todos, slight optim
cd5caf7
@erhant
added unload_model endpoint to API
a38f656
@andthattoo
add make path for submodules
d947b0f
@andthattoo
bumped mlx-lm to 0.28.2 and mlx is forced to 0.29.2
cc63de6
@andthattoo
updated protos
3f72c9d
@andthattoo
compression utils added
cb426d9
@andthattoo
removed compression to compression/
8f5137d
@andthattoo
remove np import
7425a20
@andthattoo
embedding_size
bb3429b
@andthattoo
Removed redundant methods from qwen3 ring impl
8e8bdb5
@andthattoo
add missing cache import
7585a2b
@andthattoo
- no longer ened sampling, lm_project related code in create_generate…
53685bd 
…_step_for_ring_with_grpc

- added streaming support
@andthattoo
shard/node migrations
7839e34
@andthattoo
lazy logs, api shouldn't load embeddings + lm head
b67a2fe
@andthattoo
variable mismatches
6f198f3 
receive-activation migration fixed
compression missing func added
@andthattoo
fixes for wiretypes
2e2c16e 
asnycio.Queue is now placed
@andthattoo
shard fixes, imports, logs
efcd718
@andthattoo
api fixes, ApiServicers replaced with ShardApiServicer that deals wit…
f6ece27 
…h tokens instead of activations
@andthattoo
fixed bf16 serialization
13849d5
@andthattoo
added manual topology for tests
5fd63f7
@andthattoo
lazy logs update
18b8475
@andthattoo
testing script for two shards at one device
619d53a
@andthattoo
logging error fix
f4c76cd
@andthattoo
removed debug logs for layer assignment
1737a42 
shard instance name standardized, removed mDNS suffix
@andthattoo
halda_solve signature matched to dsolver-0.1.1
e89941c
@andthattoo
.git updates, submodules follow master branches
212c67f
@andthattoo
halda import fix
7baecfb
@andthattoo
config replaced with env variables
b4de68d
@andthattoo
weird claude comments removed
f31f22c
@andthattoo
reformatting + ruff fixes
7c3d196
@andthattoo
log cleaning + kvcache config added removed env vars to controle kv c…
74c1c8d 
…ache quantization
@andthattoo
kv_ttl added to KVCacheConfig
4a483ad
@andthattoo
dsolver -> distilp
26a4d8b
@andthattoo
submodule update
b78d83f
@andthattoo
Run profile functions within discovery as seperate processes to ensur…
c319cf7 
…e memory re-claim and prevent bloating
@andthattoo
re-formatting
43ed742
@andthattoo andthattoo merged commit 41a82b3 into master Oct 19, 2025
Yuvrajxms09 pushed a commit to Yuvrajxms09/dnet that referenced this pull request Dec 21, 2025
@andthattoo
Merge pull request firstbatchxyz#2 from firstbatchxyz/andthattoo/migr…
3716568 
…ation

Andthattoo/migration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@andthattoo @erhant