add sharding for gpt3 #1064

zhaoyinglia · 2021-09-23T02:52:08Z

PR types

New features

PR changes

Models

Description

add sharding parallel for gpt-3

ZeyuChen

Please shorter the import path of DygraphShardingOptimizer since it's not so friendly for developer.

ZeyuChen · 2021-09-23T03:28:09Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

@@ -30,26 +30,20 @@
 import lr
 from paddle.distributed import fleet
 from paddle.distributed.fleet.meta_parallel import get_rng_state_tracker
+from paddle.distributed.fleet.meta_optimizers.dygraph_optimizer.dygraph_sharding_optimizer import DygraphShardingOptimizer


It's too long for this import ,is it ok to update the init file to shorter the import path?

ZHUI · 2021-09-23T13:30:58Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

-    paddle.seed(basic_seed + dp_rank)
-
-    # local_seed/ global_seed is used to control dropout in ModelParallel
-    local_seed = basic_seed + 123 + mp_rank * 10 + pp_rank * 1000


seed 的东西可否都移入到这里 @zhaoyuchen2018

ZHUI · 2021-09-23T06:11:53Z

examples/language_model/gpt-3/dygraph/run_pretrain.py


 MODEL_CLASSES = {
    "gpt": (GPTForPretraining, GPTTokenizer),
    "gpt-cn": (GPTForPretraining, GPTChineseTokenizer),
 }


-def set_hyrbid_parallel_seed(basic_seed, dp_rank, mp_rank, pp_rank):
+def set_hyrbid_parallel_seed(basic_seed, idx):
    assert args.device != "cpu"


对，这里需要给一个 warning 吧， @ForFishes 后面可以加一下

ok， will fix in next pr.

ZHUI · 2021-09-23T06:12:46Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

@@ -174,7 +184,7 @@ def do_train(args):

    clip = None
    if args.grad_clip > 0:
-        clip = paddle.nn.ClipGradByGlobalNorm(clip_norm=args.grad_clip)


？为什么改掉ClipGradByGlobalNorm

需要支持 ClipGradByGlobalNorm，如果报错请查看 https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/distributed/fleet/meta_optimizers/dygraph_optimizer/hybrid_parallel_optimizer.py#L45 @zhaoyingli

已支持ClipGradByGlobalNorm。

ZHUI · 2021-09-23T06:13:57Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

@@ -227,8 +253,8 @@ def do_train(args):
                args,
                data_file,
                local_rank=local_rank,
-                data_world_size=args.dp_degree,
-                data_world_rank=dp_rank,
+                data_world_size=worker_num,


这里不能这么改，要考虑MP

data_world_size 只需要考虑读不同数据等rank？ dp 和 sharding group 读取不同的数据， mp 和 pp 读取相同的数据，所以data_world_size = dp_degree * sharding_degree

我看岔了，worker_index worker_num你们重新定义了，sorry！

examples/language_model/gpt-3/dygraph/run_pretrain.py

ZHUI · 2021-09-23T13:29:46Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

+    sharding_rank = hcg.get_sharding_parallel_rank()
+
+    sharding_size = hcg.get_sharding_parallel_world_size()
+    worker_index = dp_rank * sharding_size + sharding_rank


How about?

worker_index -> data_world_rank worker_num -> data_world_size

ZHUI · 2021-09-23T13:30:58Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

-    paddle.seed(basic_seed + dp_rank)
-
-    # local_seed/ global_seed is used to control dropout in ModelParallel
-    local_seed = basic_seed + 123 + mp_rank * 10 + pp_rank * 1000


seed 的东西可否都移入到这里 @zhaoyuchen2018

ZHUI · 2021-09-24T01:16:38Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

                if (global_step % args.save_steps == 0 or
-                        global_step >= args.max_steps) and dp_rank == 0:
+                        global_step >= args.max_steps) and worker_index == 0:


静态图是专门写了 save_persistable 来save，这里的sharding，需要其他专门的save支持吗？

ZHUI

LGTM

ZHUI · 2021-10-11T03:51:05Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

    }

    strategy.pipeline_configs = {
        "accumulate_steps": args.local_batch_size // args.micro_batch_size,
        "micro_batch_size": args.micro_batch_size
    }

+    strategy.tensor_parallel_configs = {"tensor_init_seed": 123, }


what's tensor_init_seed for ?

set control in tensor parallel

ZHUI · 2021-10-11T04:07:47Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

@@ -322,24 +348,25 @@ def do_train(args):
                    logger.info("Save model to %s" % output_dir)

                    if args.pp_degree > 1:
-                        model_to_save.save_state_dict(output_dir)
-                        if mp_rank * pp_rank == 1:
+                        if mp_rank == 0 and sharding_rank == 0 and pp_rank == 0:


所以目前的策略是，dp_rank == 0 下所有参数都保存。

现阶段，可以不加load 参数，但辛苦确认一下，load checkpoint会不会存在问题？

另外记一个合并参数的 TODO 吧

dp_rank==0下所有参数都保存这个逻辑没有问题，但是现在模型保存没有测试，所以可能会存在问题，比如是否存储了全量的参数。

ForFishes · 2021-10-11T04:15:31Z

examples/language_model/gpt-3/dygraph/run_pretrain.py


 MODEL_CLASSES = {
    "gpt": (GPTForPretraining, GPTTokenizer),
    "gpt-cn": (GPTForPretraining, GPTChineseTokenizer),
 }


-def set_hyrbid_parallel_seed(basic_seed, dp_rank, mp_rank, pp_rank):
+def set_hyrbid_parallel_seed(basic_seed, idx):
    assert args.device != "cpu"


ok， will fix in next pr.

ForFishes · 2021-10-11T04:16:49Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

    local_rank = int(os.getenv("PADDLE_RANK_IN_NODE", 0))

    # seed control in hybrid parallel
-    set_hyrbid_parallel_seed(args.seed, dp_rank, mp_rank, pp_rank)
+    set_hyrbid_parallel_seed(args.seed, data_world_rank)


所有seed都在这里设置吧

ForFishes · 2021-10-11T04:17:48Z

examples/language_model/gpt-3/dygraph/run_pretrain.py

                            tokenizer.save_pretrained(output_dir)
+                        model_to_save.save_state_dict(output_dir)


模型保存这里，没有详细的测试。需要记个TODO

ZHUI

LGTM

zhaoyingli added 2 commits September 23, 2021 10:03

add sharding for gpt-3

7b46a0f

del debug

e53dc0f

ZeyuChen requested a review from ZHUI September 23, 2021 03:27

ZeyuChen assigned ZHUI Sep 23, 2021

ZeyuChen reviewed Sep 23, 2021

View reviewed changes

ZHUI requested a review from ForFishes September 23, 2021 06:09

ZHUI requested changes Sep 23, 2021

View reviewed changes

ZHUI reviewed Sep 24, 2021

View reviewed changes

zhaoyingli added 2 commits September 29, 2021 16:24

add sharding save model

32212cf

update model save

81a44c8

ZHUI previously approved these changes Oct 11, 2021

View reviewed changes

ForFishes reviewed Oct 11, 2021

View reviewed changes

fix seed func

ae94f58

zhaoyinglia dismissed ZHUI’s stale review via ae94f58 October 11, 2021 07:35

set control in tensor parallel

7972503

ZHUI approved these changes Oct 11, 2021

View reviewed changes

ZHUI and others added 2 commits October 11, 2021 16:19

Merge branch 'develop' into gpt3-sharding

f214022

Merge branch 'develop' into gpt3-sharding

044264f

ZeyuChen merged commit 91d81c9 into PaddlePaddle:develop Oct 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add sharding for gpt3 #1064

add sharding for gpt3 #1064

zhaoyinglia commented Sep 23, 2021

ZeyuChen left a comment

ZeyuChen Sep 23, 2021

zhaoyinglia Oct 9, 2021

ZHUI Sep 23, 2021

ZHUI Sep 23, 2021

ForFishes Oct 11, 2021

ZHUI Sep 23, 2021

JZ-LIANG Sep 23, 2021

zhaoyinglia Oct 9, 2021

ZHUI Sep 23, 2021

JZ-LIANG Sep 23, 2021

ZHUI Sep 23, 2021 •

edited

Loading

ZHUI Sep 23, 2021

zhaoyinglia Oct 9, 2021

ZHUI Sep 23, 2021

ZHUI Sep 24, 2021

zhaoyinglia Oct 9, 2021

ZHUI left a comment

ZHUI Oct 11, 2021

ZHUI Oct 11, 2021

zhaoyinglia Oct 11, 2021

ZHUI Oct 11, 2021

zhaoyinglia Oct 11, 2021

ForFishes Oct 11, 2021

ForFishes Oct 11, 2021

zhaoyinglia Oct 11, 2021

ForFishes Oct 11, 2021

zhaoyinglia Oct 11, 2021

ZHUI left a comment

		tokenizer.save_pretrained(output_dir)
		model_to_save.save_state_dict(output_dir)

add sharding for gpt3 #1064

add sharding for gpt3 #1064

Conversation

zhaoyinglia commented Sep 23, 2021

PR types

PR changes

Description

ZeyuChen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZHUI Sep 23, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZHUI left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZHUI left a comment

Choose a reason for hiding this comment

ZHUI Sep 23, 2021 •

edited

Loading