Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Pipeline Parallel for PPO training and support generation with InferenceModel #7953

Merged
merged 56 commits into from
Jun 13, 2024
Merged

Add Pipeline Parallel for PPO training and support generation with InferenceModel #7953

merged 56 commits into from
Jun 13, 2024

Commits on Feb 2, 2024

  1. Add Pipeline Parallel for PPO training.

    @guoshengCS
    guoshengCS committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    c71fb92 View commit details
    Browse the repository at this point in the history

  2. Move new_ppo_trainer.py to ppo_trainer.py

    @guoshengCS
    guoshengCS committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    2f2ad5e View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2024

  1. Fix padding among batches of accumulation steps in _prepare_pipeline_… 

    …inputs_func.
    @guoshengCS
    guoshengCS committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    8e8143e View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2024

  1. Fix hcg using in TP generation

    @guoshengCS
    guoshengCS committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    e4d7781 View commit details
    Browse the repository at this point in the history

  2. Try to support generation in PP. And allow extra training args passed… 

    … from main from_pratrined.
    @guoshengCS
    guoshengCS committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    4d1641b View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2024

  1. Support PP generation.

    @guoshengCS
    guoshengCS committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    34d4cd1 View commit details
    Browse the repository at this point in the history

  2. Fix PP eval by unify prediction_step

    @guoshengCS
    guoshengCS committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    665fee2 View commit details
    Browse the repository at this point in the history

  3. Fix reward value showing error cased by BF16 dtype when eval

    @guoshengCS
    guoshengCS committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    a2e9702 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. fix all

    @ZHUI
    ZHUI committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    6c8441c View commit details
    Browse the repository at this point in the history

  2. Make non-PipelineParallel models use the same loss layer with PipeMod… 

    …el to unify.
    @guoshengCS
    guoshengCS committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    d295d11 View commit details
    Browse the repository at this point in the history

  3. add offload.

    @ZHUI
    ZHUI committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    38cc1a7 View commit details
    Browse the repository at this point in the history

  4. Use create_loss to unify Pipe and non-Pipe usage.

    @guoshengCS
    guoshengCS committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    6ff38c8 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. Add eval mode and offload level.

    @ZHUI
    ZHUI committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    6e49431 View commit details
    Browse the repository at this point in the history

  2. merge

    @ZHUI
    ZHUI committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    c421af7 View commit details
    Browse the repository at this point in the history

  3. fix all

    @ZHUI
    ZHUI committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    f6b5f97 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. support tp+pp

    @ZHUI
    ZHUI committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    63df4fd View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. fix data split.

    @ZHUI
    ZHUI committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    c9e5cad View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. Fix position_ids in generation/eval/train.

    @guoshengCS
    guoshengCS committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    5979507 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2024

  1. fix data group.

    @ZHUI
    ZHUI committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    16d886a View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. add tp rank guard

    @ZHUI
    ZHUI committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    1786357 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. Support rollout label data both with target length or source+target l… 

    …ength.
    @guoshengCS
    guoshengCS committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    3bc48cb View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. Merge remote-tracking branch 'guosheng/ppo-4d' into ppo-4d/support_uc

    @ZHUI
    ZHUI committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    5ae2c6f View commit details
    Browse the repository at this point in the history

  2. Move metric calculation to rl_step to avoid comm.

    @guoshengCS
    guoshengCS committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    1b50869 View commit details
    Browse the repository at this point in the history

  3. fix pad

    @ZHUI
    ZHUI committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    bc80256 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'guosheng/ppo-4d' into ppo-4d/support_uc

    @ZHUI
    ZHUI committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    986b407 View commit details
    Browse the repository at this point in the history

  5. fix create group.

    @ZHUI
    ZHUI committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    b3f22c2 View commit details
    Browse the repository at this point in the history

  6. no print

    @ZHUI
    ZHUI committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    8c7e612 View commit details
    Browse the repository at this point in the history

  7. Suppport inference model generation.

    @guoshengCS
    guoshengCS committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    2e3bf85 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. fix compatible for no eval model.

    @ZHUI
    ZHUI committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    df452d1 View commit details
    Browse the repository at this point in the history

  2. fix pp sync.

    @ZHUI
    ZHUI committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    d73b8a3 View commit details
    Browse the repository at this point in the history

  3. remove debug info

    @ZHUI
    ZHUI committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    e14e04b View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. Refacor PPO training using StepTrainer.

    @guoshengCS
    guoshengCS committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    8015c8b View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2024

  1. Open PolicyTrainer loss logging postprocess. More StepTrainer docs.

    @guoshengCS
    guoshengCS committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    860e61d View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2024

  1. more timer.

    @ZHUI
    ZHUI committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    afa1b53 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'guosheng/ppo-4d' into ppo-4d/support_uc

    @ZHUI
    ZHUI committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    80e47e6 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. fix bugs.

    @ZHUI
    ZHUI committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    757d3a7 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #1 from PaddlePaddle/ppo-4d/support_uc 

    PPP 4d/support uc
    @guoshengCS
    guoshengCS committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    6f2eff6 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. Add EMA and PPOMetric

    @guoshengCS
    guoshengCS committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    1448b73 View commit details
    Browse the repository at this point in the history

  2. add tests

    @ZHUI
    ZHUI committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    b809631 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. add unit test for rank guard.

    @ZHUI
    ZHUI committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    2f8d032 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #2 from ZHUI/ppo-4d-test 

    Add ppo 4d test
    @guoshengCS
    guoshengCS committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    edf28f2 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. Fix reshard zero3 and reshard infer.

    @guoshengCS
    guoshengCS committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    fbb9ac3 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'ppo-4d' of https://github.com/guoshengCS/PaddleNLP into… 

    … ppo-4d-reshard
    @guoshengCS
    guoshengCS committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    aebdc89 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. Revert PaddlePaddle#7818 for llama and remove position_ids for gen/tr… 

    …ain/eval to align.
    @guoshengCS
    guoshengCS committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    cb6e4ff View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. Move reload/clean/data_group to comm_utils and use guard to decorate … 

    …them.
    @guoshengCS
    guoshengCS committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    4ddd415 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. Offload sync and other data reuse fix.

    @guoshengCS
    guoshengCS committed May 7, 2024
    Configuration menu
    Copy the full SHA
    b68cb0d View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Clead code

    @guoshengCS
    guoshengCS committed May 9, 2024
    Configuration menu
    Copy the full SHA
    5e46ab6 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Update README

    @guoshengCS
    guoshengCS committed May 16, 2024
    Configuration menu
    Copy the full SHA
    d538917 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i… 

    …nto ppo-4d
    @guoshengCS
    guoshengCS committed May 27, 2024
    Configuration menu
    Copy the full SHA
    510ef03 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. Update ppo_trainer

    @guoshengCS
    guoshengCS committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    1feb5ee View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. format code

    @gongel
    gongel committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    c8b3c61 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Fix make_position_ids by 4d causal mask.

    @guoshengCS
    guoshengCS committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    c26583f View commit details
    Browse the repository at this point in the history

  2. Merge pull request #4 from gongel/ppo-4d 

    [PPO] Format code
    @guoshengCS
    guoshengCS committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    1b8e4a3 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'ppo-4d' of https://github.com/guoshengCS/PaddleNLP into… 

    … ppo-4d-eb
    @guoshengCS
    guoshengCS committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    9acd87c View commit details
    Browse the repository at this point in the history

  4. Fix nested_broadcast_tensor_with_empty import

    @guoshengCS
    guoshengCS committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    ffa4658 View commit details
    Browse the repository at this point in the history

  5. Update eval with make_attention_mask

    @guoshengCS
    guoshengCS committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    f1e66f2 View commit details
    Browse the repository at this point in the history