Skip to content

Commit

Permalink
Hotfix/tutorial readme index (#1922)
Browse files Browse the repository at this point in the history
* [tutorial] removed tutorial index in readme

* [tutorial] removed tutorial index in readme
  • Loading branch information
FrankLeeeee committed Nov 12, 2022
1 parent 24cbee0 commit d43a671
Show file tree
Hide file tree
Showing 5 changed files with 16 additions and 17 deletions.
2 changes: 1 addition & 1 deletion examples/tutorial/auto_parallel/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Handson 3: Auto-Parallelism with ResNet
# Auto-Parallelism with ResNet

## Prepare Dataset

Expand Down
2 changes: 1 addition & 1 deletion examples/tutorial/hybrid_parallel/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Handson 1: Multi-dimensional Parallelism with Colossal-AI
# Multi-dimensional Parallelism with Colossal-AI


## Install Titans Model Zoo
Expand Down
2 changes: 1 addition & 1 deletion examples/tutorial/large_batch_optimizer/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Handson 4: Comparison of Large Batch Training Optimization
# Comparison of Large Batch Training Optimization

## Prepare Dataset

Expand Down
2 changes: 1 addition & 1 deletion examples/tutorial/opt/README.md
Original file line number Diff line number Diff line change
@@ -1 +1 @@
# Handson 5: Fine-tuning and Serving for OPT from Hugging Face
# Fine-tuning and Serving for OPT from Hugging Face
25 changes: 12 additions & 13 deletions examples/tutorial/sequence_parallel/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Handson 2: Sequence Parallelism with BERT
# Sequence Parallelism with BERT

In this example, we implemented BERT with sequence parallelism. Sequence parallelism splits the input tensor and intermediate
In this example, we implemented BERT with sequence parallelism. Sequence parallelism splits the input tensor and intermediate
activation along the sequence dimension. This method can achieve better memory efficiency and allows us to train with larger batch size and longer sequence length.

Paper: [Sequence Parallelism: Long Sequence Training from System Perspective](https://arxiv.org/abs/2105.13120)
Expand All @@ -16,15 +16,15 @@ First, let's prepare the WikiPedia dataset from scratch. To generate a preproces
For the preprocessing script, we thank Megatron-LM for providing a preprocessing script to generate the corpus file.

```python
# download raw data
# download raw data
mkdir data && cd ./data
wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2

# install wiki extractor
git clone https://github.com/FrankLeeeee/wikiextractor.git
pip install ./wikiextractor

# extractmodule
# extractmodule
wikiextractor --json enwiki-latest-pages-articles.xml.bz2
cat text/*/* > ./corpus.json
cd ..
Expand All @@ -34,7 +34,7 @@ mkdir vocab && cd ./vocab
wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt
cd ..

# preprocess some data
# preprocess some data
git clone https://github.com/NVIDIA/Megatron-LM.git
cd ./Megatron-LM
python tools/preprocess_data.py \
Expand Down Expand Up @@ -86,12 +86,12 @@ class Encoder(object):

## How to Train with Sequence Parallelism

We provided `train.py` for you to execute training. Before invoking the script, there are several
We provided `train.py` for you to execute training. Before invoking the script, there are several
steps to perform.

### Step 1. Set data path and vocab path

At the top of `config.py`, you can see two global variables `DATA_PATH` and `VOCAB_FILE_PATH`.
At the top of `config.py`, you can see two global variables `DATA_PATH` and `VOCAB_FILE_PATH`.

```python
DATA_PATH = <data-path>
Expand All @@ -106,7 +106,7 @@ For example, if your my-bert_text_sentence.bin is /home/Megatron-LM/my-bert_text
DATA_PATH = '/home/Megatron-LM/my-bert_text_sentence'
```

The `VOCAB_FILE_PATH` refers to the path to the vocabulary downloaded when you prepare the dataset
The `VOCAB_FILE_PATH` refers to the path to the vocabulary downloaded when you prepare the dataset
(e.g. bert-large-uncased-vocab.txt).

### Step 3. Make Dataset Helper
Expand All @@ -121,12 +121,12 @@ make
### Step 3. Configure your parameters

In the `config.py` provided, a set of parameters are defined including training scheme, model, etc.
You can also modify the ColossalAI setting. For example, if you wish to parallelize over the
You can also modify the ColossalAI setting. For example, if you wish to parallelize over the
sequence dimension on 8 GPUs. You can change `size=4` to `size=8`. If you wish to use pipeline parallelism, you can set `pipeline=<num_of_pipeline_stages>`.

### Step 4. Invoke parallel training

Lastly, you can start training with sequence parallelism. How you invoke `train.py` depends on your
Lastly, you can start training with sequence parallelism. How you invoke `train.py` depends on your
machine setting.

- If you are using a single machine with multiple GPUs, PyTorch launch utility can easily let you
Expand All @@ -137,7 +137,6 @@ machine setting.
```

- If you are using multiple machines with multiple GPUs, we suggest that you refer to `colossalai
launch_from_slurm` or `colossalai.launch_from_openmpi` as it is easier to use SLURM and OpenMPI
to start multiple processes over multiple nodes. If you have your own launcher, you can fall back
launch_from_slurm` or `colossalai.launch_from_openmpi` as it is easier to use SLURM and OpenMPI
to start multiple processes over multiple nodes. If you have your own launcher, you can fall back
to the default `colossalai.launch` function.

0 comments on commit d43a671

Please sign in to comment.