Skip to content

Fsdp advanced tutorial #1959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 63 commits into from
Jul 7, 2022
Merged

Fsdp advanced tutorial #1959

Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
7bc252b
WIP adding FSDP advanced tutorial on T5 training
HamidShojanazeri Jun 7, 2022
c1fa8d3
FSDP advanced featues in progress
HamidShojanazeri Jun 11, 2022
08da9bb
removing old flow charts
HamidShojanazeri Jun 11, 2022
919c9c3
clean up
HamidShojanazeri Jun 11, 2022
a0b1745
clean up
HamidShojanazeri Jun 11, 2022
f6c4e56
clean up
HamidShojanazeri Jun 11, 2022
e329e48
clean up
HamidShojanazeri Jun 11, 2022
d20dedb
intro updates
HamidShojanazeri Jun 20, 2022
a6e878d
Updates on features
HamidShojanazeri Jun 27, 2022
a390494
typos correction
HamidShojanazeri Jun 27, 2022
50f43d2
Update sharding strategy
HamidShojanazeri Jun 27, 2022
a0d7052
adding cross ref
HamidShojanazeri Jun 27, 2022
ca227bf
Update FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 27, 2022
958df1e
Update FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 27, 2022
8234f5b
added summary to the end
HamidShojanazeri Jun 27, 2022
2713757
added reference to code section for adding each feature
HamidShojanazeri Jun 27, 2022
5309ca5
update text grammar and minor details for top section
lessw2020 Jun 27, 2022
c86206c
removed activation checkpointing
HamidShojanazeri Jun 28, 2022
875b97e
various grammar cleanup and minor explanatory additions (upper section)
lessw2020 Jun 28, 2022
66ed452
update transformer wrapping policy and mixed precision
lessw2020 Jun 28, 2022
a5251b1
complete edits/updates for feature sections
lessw2020 Jun 29, 2022
21946be
add bolded summary
lessw2020 Jun 29, 2022
f48f6c7
updated the backward prefetch
HamidShojanazeri Jun 29, 2022
4904337
update the example link
HamidShojanazeri Jun 29, 2022
2b72377
Update FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 29, 2022
a59b0bf
update authors
HamidShojanazeri Jun 29, 2022
70318d5
Merge branch 'master' into FSDP_advanced_tutorial
HamidShojanazeri Jun 29, 2022
9b6e86b
minor spelling fixes
lessw2020 Jun 29, 2022
b544f17
typos fix
HamidShojanazeri Jun 29, 2022
7a6cf32
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 30, 2022
21ba97c
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 30, 2022
fbe84f1
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 30, 2022
a2494af
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 30, 2022
3781a2c
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 30, 2022
224256f
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 30, 2022
0bbb15e
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 30, 2022
22d5c9c
clean up
HamidShojanazeri Jun 30, 2022
39da6cb
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jun 30, 2022
7e4639a
addressing comments
HamidShojanazeri Jul 5, 2022
0857f7f
addressed backward prefetch comments
HamidShojanazeri Jul 5, 2022
49f562e
addressed the comment on model checkpoint saving
HamidShojanazeri Jul 5, 2022
09112a6
Addressing the zero2 comments and overview of FSDP
HamidShojanazeri Jul 5, 2022
e8dca4b
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
a606a9c
updates based on comments
HamidShojanazeri Jul 5, 2022
989e811
FSDP wrapper comments
HamidShojanazeri Jul 5, 2022
a7e0a03
updated with additional comments
HamidShojanazeri Jul 5, 2022
876d7c2
updated the bfloat16 memory
HamidShojanazeri Jul 5, 2022
e0aa2b4
updating the title separator
HamidShojanazeri Jul 5, 2022
6003fd6
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
3baf838
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
0725539
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
61fba4f
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
0f4e437
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
dc1adfa
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
e69008d
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
c03cf82
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
9d8f6da
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
210ac24
Update intermediate_source/FSDP_adavnced_tutorial.rst
HamidShojanazeri Jul 5, 2022
10871a9
addressing comments on code consistency
HamidShojanazeri Jul 7, 2022
c4581f9
remove CPUoffload import
HamidShojanazeri Jul 7, 2022
0e3062e
mixed precision re-wording
HamidShojanazeri Jul 7, 2022
ef1ea8f
addressing comments
HamidShojanazeri Jul 7, 2022
29caf67
Merge branch 'master' into FSDP_advanced_tutorial
HamidShojanazeri Jul 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading