{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":585017139,"defaultBranch":"main","name":"ColossalAI","ownerLogin":"ericxsun","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2023-01-04T05:19:10.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/1772912?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1672809551.819573","currentOid":""},"activityList":{"items":[{"before":"8e412a548e5366d1c42bcf386bd185091bd0c280","after":"3f2be80530ad4ba332b0f375d0493c46fcebe298","ref":"refs/heads/main","pushedAt":"2024-06-03T03:31:53.000Z","pushType":"push","commitsCount":308,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"fix (#5765)","shortMessageHtmlLink":"fix (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2325044650\" data-permission-text=\"Title is private\" data-url=\"https://github.com/hpcaitech/ColossalAI/issues/5765\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/hpcaitech/ColossalAI/pull/5765/hovercard\" href=\"https://github.com/hpcaitech/ColossalAI/pull/5765\">hpcaitech#5765</a>)"}},{"before":"df5e9c53cf23d44656470cc319ee0b470c40712f","after":"8e412a548e5366d1c42bcf386bd185091bd0c280","ref":"refs/heads/main","pushedAt":"2024-04-03T10:20:50.000Z","pushType":"push","commitsCount":3,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[shardformer] Sequence Parallelism Optimization (#5533)\n\n* sequence parallel optimization\r\n\r\n* validate sequence parallel in llama (code to be polished)\r\n\r\n* shardformer api writing\r\n\r\n* integrate sequence parallel in ShardFormer\r\n\r\n* fix pp bugs and sp bugs for LlaMa model\r\n\r\n* integrating ring-based sequence parallelism into ShardFormer\r\n\r\n* [sequence parallelism]: Add fused megatron function\r\n\r\n* integrating ring-based sequence parallelism into ShardFormer\r\n\r\n---------\r\n\r\nCo-authored-by: linsj20 <linsj20@mails.tsinghua.edu.cn>\r\n\r\n* fix bugs when useing sp and flashattention together\r\n\r\n* fix operation function name\r\n\r\n* support flash attention for ulysses-style sp\r\n\r\n* clarify sp process group\r\n\r\n* fix compatibility bugs in moe plugin\r\n\r\n* fix fused linear bugs\r\n\r\n* fix linear layer test\r\n\r\n* support gpt model all-to-all sp\r\n\r\n* modify shard data dimension (meant to be dim=-1)\r\n\r\n* support megtron-style sp and distributed attn for llama model\r\n\r\n* [shardformer] add megatron sp to llama\r\n\r\n* support llama7B 128k with distributed attention\r\n\r\n* [shardformer] robustness enhancement\r\n\r\n* add block attn\r\n\r\n* sp mode 1: keep input as a complete sequence\r\n\r\n* fix sp compatability\r\n\r\n* finish sp mode 3 support for gpt\r\n\r\n* using all_to_all_single when batch size is 1\r\n\r\n* support mode 2 sp in gpt2 (#5)\r\n\r\n* [shardformer] add megatron sp to llama\r\n\r\n* support llama7B 128k with distributed attention\r\n\r\n* [shardformer] robustness enhancement\r\n\r\n* add block attn\r\n\r\n* sp mode 1: keep input as a complete sequence\r\n\r\n* fix sp compatability\r\n\r\n* refactor ring implementation\r\n\r\n* support mode 2 sp in gpt2\r\n\r\n* polish code\r\n\r\n* enable distributed attn mask when using sp mode 2 and 3 in llama\r\n\r\n* automatically enable flash attn when using sp mode 2 and 3 in llama\r\n\r\n* inplace attn mask\r\n\r\n* add zero2 support for sequence parallel\r\n\r\n* polish code\r\n\r\n* fix bugs\r\n\r\n* fix gemini checkpoint io\r\n\r\n* loose tensor checking atol and rtol\r\n\r\n* add comment\r\n\r\n* fix llama layernorm grad\r\n\r\n* fix zero grad\r\n\r\n* fix zero grad\r\n\r\n* fix conflict\r\n\r\n* update split and gather auto grad func\r\n\r\n* sequence parallel: inside text split (#6)\r\n\r\n* polish code (part 1)\r\n\r\n* polish code (part 2)\r\n\r\n* polish code (part 2.5)\r\n\r\n* polish code (part 3)\r\n\r\n* sequence parallel: inside text split\r\n\r\n* miscellaneous minor fixes\r\n\r\n* polish code\r\n\r\n* fix ulysses style ZeRO\r\n\r\n* sequence parallel: inside text split\r\n\r\n* miscellaneous minor fixes\r\n\r\n* disaggregate sp group and dp group for  sp\r\n\r\n* fix llama and gpt sp\r\n\r\n* polish code\r\n\r\n* move ulysses grad sync to ddp (#9)\r\n\r\n* remove zero_stage and unbind the grad sync for alltoall sp\r\n\r\n* add 2d group creation test\r\n\r\n* move ulysses grad sync to ddp\r\n\r\n* add 2d group creation test\r\n\r\n* remove useless code\r\n\r\n* change shard config not to enable sp when enable_all_optimizations\r\n\r\n* add sp warnings for several model\r\n\r\n* remove useless code\r\n\r\n---------\r\n\r\nCo-authored-by: linsj20 <linsj20@mails.tsinghua.edu.cn>","shortMessageHtmlLink":"[shardformer] Sequence Parallelism Optimization (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2212681663\" data-permission-text=\"Title is private\" data-url=\"https://github.com/hpcaitech/ColossalAI/issues/5533\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/hpcaitech/ColossalAI/pull/5533/hovercard\" href=\"https://github.com/hpcaitech/ColossalAI/pull/5533\">hpcaitech#5533</a>)"}},{"before":"385e85afd460a1b9a947b09c9d0f7d2628c35ad2","after":"df5e9c53cf23d44656470cc319ee0b470c40712f","ref":"refs/heads/main","pushedAt":"2024-03-30T08:26:15.000Z","pushType":"push","commitsCount":21,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[ColossalChat] Update RLHF V2 (#5286)\n\n* Add dpo. Fix sft, ppo, lora. Refactor all\r\n\r\n* fix and tested ppo\r\n\r\n* 2 nd round refactor\r\n\r\n* add ci tests\r\n\r\n* fix ci\r\n\r\n* fix ci\r\n\r\n* fix readme, style\r\n\r\n* fix readme style\r\n\r\n* fix style, fix benchmark\r\n\r\n* reproduce benchmark result, remove useless files\r\n\r\n* rename to ColossalChat\r\n\r\n* use new image\r\n\r\n* fix ci workflow\r\n\r\n* fix ci\r\n\r\n* use local model/tokenizer for ci tests\r\n\r\n* fix ci\r\n\r\n* fix ci\r\n\r\n* fix ci\r\n\r\n* fix ci timeout\r\n\r\n* fix rm progress bar. fix ci timeout\r\n\r\n* fix ci\r\n\r\n* fix ci typo\r\n\r\n* remove 3d plugin from ci temporary\r\n\r\n* test environment\r\n\r\n* cannot save optimizer\r\n\r\n* support chat template\r\n\r\n* fix readme\r\n\r\n* fix path\r\n\r\n* test ci locally\r\n\r\n* restore build_or_pr\r\n\r\n* fix ci data path\r\n\r\n* fix benchmark\r\n\r\n* fix ci, move ci tests to 3080, disable fast tokenizer\r\n\r\n* move ci to 85\r\n\r\n* support flash attention 2\r\n\r\n* add all-in-one data preparation script. Fix colossal-llama2-chat chat template\r\n\r\n* add hardware requirements\r\n\r\n* move ci test data\r\n\r\n* fix save_model, add unwrap\r\n\r\n* fix missing bos\r\n\r\n* fix missing bos; support grad accumulation with gemini\r\n\r\n* fix ci\r\n\r\n* fix ci\r\n\r\n* fix ci\r\n\r\n* fix llama2 chat template config\r\n\r\n* debug sft\r\n\r\n* debug sft\r\n\r\n* fix colossalai version requirement\r\n\r\n* fix ci\r\n\r\n* add sanity check to prevent NaN loss\r\n\r\n* fix requirements\r\n\r\n* add dummy data generation script\r\n\r\n* add dummy data generation script\r\n\r\n* add dummy data generation script\r\n\r\n* add dummy data generation script\r\n\r\n* update readme\r\n\r\n* update readme\r\n\r\n* update readme and ignore\r\n\r\n* fix logger bug\r\n\r\n* support parallel_output\r\n\r\n* modify data preparation logic\r\n\r\n* fix tokenization\r\n\r\n* update lr\r\n\r\n* fix inference\r\n\r\n* run pre-commit\r\n\r\n---------\r\n\r\nCo-authored-by: Tong Li <tong.li352711588@gmail.com>","shortMessageHtmlLink":"[ColossalChat] Update RLHF V2 (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2089916679\" data-permission-text=\"Title is private\" data-url=\"https://github.com/hpcaitech/ColossalAI/issues/5286\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/hpcaitech/ColossalAI/pull/5286/hovercard\" href=\"https://github.com/hpcaitech/ColossalAI/pull/5286\">hpcaitech#5286</a>)"}},{"before":"ec7176cc954766bca10cf3a0d9a1dd5d6df0803d","after":"385e85afd460a1b9a947b09c9d0f7d2628c35ad2","ref":"refs/heads/main","pushedAt":"2024-03-12T05:55:01.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[hotfix] fix typo s/keywrods/keywords etc. (#5429)","shortMessageHtmlLink":"[hotfix] fix typo s/keywrods/keywords etc. (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2170805179\" data-permission-text=\"Title is private\" data-url=\"https://github.com/hpcaitech/ColossalAI/issues/5429\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/hpcaitech/ColossalAI/pull/5429/hovercard\" href=\"https://github.com/hpcaitech/ColossalAI/pull/5429\">hpcaitech#5429</a>)"}},{"before":"63c6b0f81dbf369712e947f69d24972bdf02d0d3","after":"ec7176cc954766bca10cf3a0d9a1dd5d6df0803d","ref":"refs/heads/main","pushedAt":"2024-03-11T02:54:04.000Z","pushType":"push","commitsCount":22,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"Merge branch 'hpcaitech:main' into main","shortMessageHtmlLink":"Merge branch 'hpcaitech:main' into main"}},{"before":"6d7b1ed7c8c0d4f0ece3a9d92ce3f836c1395b00","after":"63c6b0f81dbf369712e947f69d24972bdf02d0d3","ref":"refs/heads/main","pushedAt":"2024-02-27T06:28:18.000Z","pushType":"push","commitsCount":28,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"Merge branch 'hpcaitech:main' into main","shortMessageHtmlLink":"Merge branch 'hpcaitech:main' into main"}},{"before":"ec4a6d58797dee4a12c1b32a1a53542ced848066","after":"6d7b1ed7c8c0d4f0ece3a9d92ce3f836c1395b00","ref":"refs/heads/main","pushedAt":"2024-02-21T05:33:17.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"8c0faaef1e7f3a1c8d66150e9cd93d2fb871cd0c","after":"ec4a6d58797dee4a12c1b32a1a53542ced848066","ref":"refs/heads/main","pushedAt":"2024-02-18T07:08:29.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"d247b58ee68597ce8d2dde935246dc0779448d9e","after":"8c0faaef1e7f3a1c8d66150e9cd93d2fb871cd0c","ref":"refs/heads/main","pushedAt":"2024-02-06T14:39:37.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"2442c7d329ed279cdd486a06103a7dc990b6c140","after":"d247b58ee68597ce8d2dde935246dc0779448d9e","ref":"refs/heads/main","pushedAt":"2024-02-02T10:53:45.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"616854c19ac35a8f4c5d91883e5e91cf692cc7f9","after":"2442c7d329ed279cdd486a06103a7dc990b6c140","ref":"refs/heads/main","pushedAt":"2024-02-02T10:51:42.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"2cfbe7c7f0bc4e214d7b49a8b6d184a89329a473","after":"616854c19ac35a8f4c5d91883e5e91cf692cc7f9","ref":"refs/heads/main","pushedAt":"2024-02-02T08:34:12.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"c60189dd5c99856a5a2e0fa554f37eb1f88b06aa","after":"2cfbe7c7f0bc4e214d7b49a8b6d184a89329a473","ref":"refs/heads/main","pushedAt":"2024-02-02T08:05:53.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"c923a4553edeee632ce52c1f55f9969ec7b81155","after":"c60189dd5c99856a5a2e0fa554f37eb1f88b06aa","ref":"refs/heads/main","pushedAt":"2024-02-01T05:03:02.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"95708707c16201a33047cef3fed680d21b43fef9","after":"c923a4553edeee632ce52c1f55f9969ec7b81155","ref":"refs/heads/main","pushedAt":"2024-01-31T10:24:55.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"5e16b805ba0f899106bf432a43ef9447d1b408cb","after":"95708707c16201a33047cef3fed680d21b43fef9","ref":"refs/heads/main","pushedAt":"2024-01-31T10:09:07.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"957b29c1ce07cdd9d739e702c9ad81311c01bce2","after":"5e16b805ba0f899106bf432a43ef9447d1b408cb","ref":"refs/heads/main","pushedAt":"2024-01-31T09:33:06.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"29877ff752005c5c3007b1b2894802dacf5836b5","after":"957b29c1ce07cdd9d739e702c9ad81311c01bce2","ref":"refs/heads/main","pushedAt":"2024-01-31T02:13:23.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"100b2fbec300a84ce7072dd507605583c0ff3801","after":"29877ff752005c5c3007b1b2894802dacf5836b5","ref":"refs/heads/main","pushedAt":"2024-01-30T03:46:12.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"[fsdp] impl save/load shard model/optimizer","shortMessageHtmlLink":"[fsdp] impl save/load shard model/optimizer"}},{"before":"71321a07cf8a24a69fa3f8a5ebf2319cc937f1cd","after":"100b2fbec300a84ce7072dd507605583c0ff3801","ref":"refs/heads/main","pushedAt":"2024-01-30T03:24:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"feat(fsdp): impl save/load shard model/optimizer","shortMessageHtmlLink":"feat(fsdp): impl save/load shard model/optimizer"}},{"before":"32253315b4ad2b2a9c9c61b82049a98637b029a4","after":"71321a07cf8a24a69fa3f8a5ebf2319cc937f1cd","ref":"refs/heads/main","pushedAt":"2024-01-30T03:12:44.000Z","pushType":"push","commitsCount":1392,"pusher":{"login":"ericxsun","name":"QinLuo","path":"/ericxsun","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1772912?s=80&v=4"},"commit":{"message":"fix typo change dosen't to doesn't (#5308)","shortMessageHtmlLink":"fix typo change dosen't to doesn't (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2099649774\" data-permission-text=\"Title is private\" data-url=\"https://github.com/hpcaitech/ColossalAI/issues/5308\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/hpcaitech/ColossalAI/pull/5308/hovercard\" href=\"https://github.com/hpcaitech/ColossalAI/pull/5308\">hpcaitech#5308</a>)"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEWn9oRAA","startCursor":null,"endCursor":null}},"title":"Activity · ericxsun/ColossalAI"}