Skip to content

[Unity] Add LoadParamOnWorker0 function in shard loader#16093

Merged
junrushao merged 2 commits intoapache:unityfrom
jinhongyii:loader_add_nonshard_func
Nov 10, 2023
Merged

[Unity] Add LoadParamOnWorker0 function in shard loader#16093
junrushao merged 2 commits intoapache:unityfrom
jinhongyii:loader_add_nonshard_func

Conversation

@jinhongyii
Copy link
Contributor

@jinhongyii jinhongyii commented Nov 9, 2023

In DistIR compilation flow, shard loading is implemented in a Relax func, like

a = LoadParamOnWorker0(loader, index=0)
b = broadcast(a)
c = LoadParamOnWorker0(loader, index=1)
d = scatter_from_worker0(c)

LoadWholeParamOnWorker0 loads the unsharded param on worker0, and for other workers returns an empty array.

This PR implements LoadWholeParamOnWorker0

@jinhongyii
Copy link
Contributor Author

cc: @junrushao

@junrushao
Copy link
Member

The implementation looks good, but not sure what the best name tbh. @tqchen maybe you could suggest?

@tqchen
Copy link
Member

tqchen commented Nov 9, 2023

How about LoadParamOnWorker0. And for future sharded version we can say LoadShardedParams

@jinhongyii
Copy link
Contributor Author

sounds good

@jinhongyii jinhongyii changed the title [Unity] Add LoadWholeParamOnWorker0 function in shard loader [Unity] Add LoadParamOnWorker0 function in shard loader Nov 9, 2023
@junrushao
Copy link
Member

Sounds good! Let’s do that then

@jinhongyii jinhongyii force-pushed the loader_add_nonshard_func branch from d90eed4 to a1f198b Compare November 9, 2023 17:44
@junrushao junrushao merged commit bc10f76 into apache:unity Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants