Version v0.1.10 Release Today!

[binmakeswell]

What's Changed

Embedding

• [embedding] cache_embedding small improvement (cache_embedding small improvement #1564) by CsRic
• [embedding] polish parallel embedding tablewise ([embedding] polish parallel embedding tablewise #1545) by Jiarui Fang
• [embedding] freq_aware_embedding: add small functions for caller application (freq_aware_embedding: add small functions for caller application #1537) by CsRic
• [embedding] fix a bug in table wise sharding ([embedding] fix a bug in table wise sharding #1538) by Jiarui Fang
• [embedding] tablewise sharding polish ([embedding] tablewise sharding polish #1535) by Jiarui Fang
• [embedding] add tablewise sharding for FAW (Tablewise cache #1526) by CsRic

Nfc

• [NFC] polish test component gpt code style ([NFC] polish test component gpt code style #1567) by アマデウス
• [NFC] polish doc style for ColoTensor ([NFC] polish doc style for ColoTensor #1457) by Jiarui Fang
• [NFC] global vars should be upper case ([NFC] fx checkpointing global vars should be upper case #1456) by Jiarui Fang

Pipeline/tuning

• [pipeline/tuning] improve dispatch performance both time and space cost ([pipeline/tuning] improve dispatch performance both time and space cost #1544) by Kirigaya Kazuto

Fx

• [fx] provide a stable but not accurate enough version of profiler. ([fx] provide a stable but not accurate enough version of profiler. #1547) by Super Daniel
• [fx] Add common node in model linearize ([fx] Add common node in model linearize #1542) by Boyuan Yao
• [fx] support meta tracing for aten level computation graphs like functorch. ([fx] support meta tracing for aten level computation graphs like functorch. #1536) by Super Daniel
• [fx] Modify solver linearize and add corresponding test ([fx] Modify solver linearize and add corresponding test #1531) by Boyuan Yao
• [fx] add test for meta tensor. ([fx] add test for meta tensor. #1527) by Super Daniel
• [fx]patch nn.functional convolution ([fx]patch nn.functional convolution #1528) by YuliangLiu0306
• [fx] Fix wrong index in annotation and minimal flops in ckpt solver ([fx] Fix wrong index in annotation and minimal flops in ckpt solver #1521) by Boyuan Yao
• [fx] hack torch_dispatch for meta tensor and autograd. ([fx] hack __torch_dispatch__ for meta tensor and autograd. #1515) by Super Daniel
• [fx] Fix activation codegen dealing with checkpointing first op ([fx] Fix activation codegen dealing with checkpointing first op #1510) by Boyuan Yao
• [fx] fix the discretize bug ([fx] fix the discretize bug #1506) by Boyuan Yao
• [fx] fix wrong variable name in solver rotor ([fx] fix wrong variable name in solver rotor #1502) by Boyuan Yao
• [fx] Add activation checkpoint solver rotor ([fx] Add activation checkpoint solver rotor #1496) by Boyuan Yao
• [fx] add more op patches for profiler and error message for unsupported ops. ([fx] add more op patches for profiler and error message for unsupported ops. #1495) by Super Daniel
• [fx] fixed adapative pooling size concatenation error ([fx] fixed adapative pooling size concatenation error #1489) by Frank Lee
• [fx] add profiler for fx nodes. ([fx] add profiler for fx nodes. #1480) by Super Daniel
• [fx] Fix ckpt functions' definitions in forward ([fx] Fix ckpt functions' definitions in forward #1476) by Boyuan Yao
• [fx] fix MetaInfoProp for incorrect calculations and add detections for inplace op. ([fx] fix MetaInfoProp for incorrect calculations and add detections for inplace op. #1466) by Super Daniel
• [fx] add rules to linearize computation graphs for searching. ([fx] add rules to linearize computation graphs for searching. #1461) by Super Daniel
• [fx] Add use_reentrant=False to checkpoint in codegen ([fx] Add use_reentrant=False to checkpoint in codegen #1463) by Boyuan Yao
• [fx] fix test and algorithm bugs in activation checkpointing. ([fx] fix test and algorithm bugs in activation checkpointing. #1451) by Super Daniel
• [fx] Use colossalai checkpoint and add offload recognition in codegen ([fx] Use colossalai checkpoint and add offload recognition in codegen #1439) by Boyuan Yao
• [fx] fix the false interpretation of algorithm 3 in https://arxiv.org/abs/1604.06174. ([fx] fix the false interpretation of algorithm 3 in https://arxiv.org/abs/1604.06174. #1446) by Super Daniel

Autoparallel

• [autoparallel]add backward cost info into strategies ([autoparallel]add backward cost info into strategies #1524) by YuliangLiu0306
• [autoparallel] support fucntion in operator handler ([autoparallel] support fucntion in operator handler #1529) by YuliangLiu0306
• [autoparallel] change the merge node logic ([autoparallel] change the merge node logic #1533) by YuliangLiu0306
• [autoparallel] added liveness analysis ([autoparallel] added liveness analysis #1516) by Frank Lee
• [autoparallel] add more sharding strategies to conv ([autoparallel] add more sharding strategies to conv #1487) by YuliangLiu0306
• [autoparallel] add cost graph class ([autoparallel] add cost graph class #1481) by YuliangLiu0306
• [autoparallel] added namespace constraints ([autoparallel] added namespace constraints #1490) by Frank Lee
• [autoparallel] integrate auto parallel with torch fx ([autoparallel] integrate auto parallel with torch fx #1479) by Frank Lee
• [autoparallel] added dot handler ([autoparallel] added dot handler #1475) by Frank Lee
• [autoparallel] introduced baseclass for op handler and reduced code redundancy ([autoparallel] introduced baseclass for op handler and reduced code redundancy #1471) by Frank Lee
• [autoparallel] standardize the code structure ([autoparallel] standardize the code structure #1469) by Frank Lee
• [autoparallel] Add conv handler to generate strategies and costs info for conv ([autoparallel] Add conv handler to generate strategies and costs info #1467) by YuliangLiu0306

Utils

• [utils] refactor parallel layers checkpoint and bcast model on loading checkpoint ([utils] refactor parallel layers checkpoint and bcast model on loading checkpoint #1548) by ver217
• [utils] optimize partition_tensor_parallel_state_dict ([utils] optimize partition_tensor_parallel_state_dict #1546) by ver217
• [utils] Add use_reetrant=False in utils.activation_checkpoint ([utils] Add use_reetrant=False in utils.activation_checkpoint #1460) by Boyuan Yao
• [utils] Impl clip_grad_norm for ColoTensor and ZeroOptimizer ([utils] Impl clip_grad_norm for ColoTensor and ZeroOptimizer #1442) by ver217

Hotfix

• [hotfix] change namespace for meta_trace. ([hotfix] change namespace for meta_trace. #1541) by Super Daniel
• [hotfix] fix init context ([hotfix] fix init context #1543) by ver217
• [hotfix] avoid conflict of meta registry with torch 1.13.0. ([hotfix] avoid conflict of meta registry with torch 1.13.0. #1530) by Super Daniel
• [hotfix] fix coloproxy typos. ([hotfix] fix coloproxy typos. #1519) by Super Daniel

Pipeline/pipleline_process_group

• [pipeline/pipleline_process_group] finish PipelineProcessGroup to manage local abd global rank in TP,DP and PP ([pipeline/pipleline_process_group] finish PipelineProcessGroup to manage local abd global rank in TP,DP and PP #1508) by Kirigaya Kazuto

Doc

• [doc] docstring for FreqAwareEmbeddingBag ([doc] docstring for FreqAwareEmbeddingBag #1525) by Jiarui Fang
• [doc] update readme with the new xTrimoMultimer project ([doc] update readme with the new xTrimoMultimer project #1477) by Sze-qq
• [doc] update docstring in ProcessGroup ([doc] update docstring in ProcessGroup #1468) by Jiarui Fang
• [Doc] add more doc for ColoTensor. ([Doc] add more doc for ColoTensor. #1458) by Jiarui Fang

Autoparellel

• [autoparellel]add strategies constructor ([autoparellel]add strategies constructor #1505) by YuliangLiu0306

Faw

• [FAW] cpu caching operations ([FAW] cpu caching operations #1520) by Jiarui Fang
• [FAW] refactor reorder() for CachedParamMgr ([FAW] refactor reorder() for CachedParamMgr #1514) by Jiarui Fang
• [FAW] LFU initialize with dataset freq ([FAW] LFU initialize with dataset freq #1513) by Jiarui Fang
• [FAW] shrink freq_cnter size (shrink freq_cnter size. #1509) by CsRic
• [FAW] remove code related to chunk ([FAW] remove code related to chunk #1501) by Jiarui Fang
• [FAW] add more docs and fix a warning ([FAW] add more docs and fix a warning #1500) by Jiarui Fang
• [FAW] FAW embedding use LRU as eviction strategy intialized with dataset stats ([FAW] FAW embedding use LRU as eviction strategy intialized with dataset stats #1494) by CsRic
• [FAW] LFU cache for the FAW by CsRic
• [FAW] init an LFU implementation for FAW ([FAW] init an LFU implementation for FAW #1488) by Jiarui Fang
• [FAW] reorganize the inheritance struct of FreqCacheEmbedding (reorganize the inheritance struct of FreqCacheEmbedding #1448) by Geng Zhang

Pipeline/rpc

• [pipeline/rpc] update outstanding mechanism | optimize dispatching strategy ([pipeline/rpc] update outstanding mechanism | optimize dispatching strategy #1497) by Kirigaya Kazuto
• [pipeline/rpc] implement distributed optimizer | test with assert_close ([pipeline/rpc] implement distributed optimizer | test with assert_close #1486) by Kirigaya Kazuto
• [pipeline/rpc] support interleaving | fix checkpoint bug | change logic when dispatch data in work_list to ensure steady 1F1B ([pipeline/rpc] support interleaving | fix checkpoint bug | change logic when dispatch data in work_list to ensure steady 1F1B #1483) by Kirigaya Kazuto
• [pipeline/rpc] implement a demo for PP with cuda rpc framework ([pipeline/rpc] implement a demo for PP with cuda rpc framework #1470) by Kirigaya Kazuto

Tensor

• [tensor]add 1D device mesh ([tensor]add 1D device mesh #1492) by YuliangLiu0306
• [tensor] support runtime ShardingSpec apply ([tensor] support runtime ShardingSpec apply #1453) by YuliangLiu0306
• [tensor] shape consistency generate transform path and communication cost ([tensor] shape consistency generate transform path and communication cost #1435) by YuliangLiu0306
• [tensor] added linear implementation for the new sharding spec ([tensor] added linear implementation for the new sharding spec #1416) by Frank Lee

Fce

• [FCE] update interface for frequency statistics in FreqCacheEmbedding ([FCE] update interface for frequency statistics in FreqCacheEmbedding #1462) by Geng Zhang

Workflow

• [workflow] added TensorNVMe to compatibility test ([workflow] added TensorNVMe to compatibility test #1449) by Frank Lee

Test

• [test] fixed the activation codegen test ([test] fixed the activation codegen test #1447) by Frank Lee

Engin/schedule

• [engin/schedule] use p2p_v2 to recontruct pipeline_schedule ([engin/schedule] use p2p_v2 to recontruct pipeline_schedule #1408) by Kirigaya Kazuto

Full Changelog: v0.1.10...v0.1.9

---

This discussion was created from the release Version v0.1.10 Release Today!.