TopSpin, is a PyTorch based high-level Deep Learning training framework, designed for BERT-style deep learning models, as opposed to GPT3 based deep learning. User defined model can be launched in either single-GPU, multi-GPU, or multi-server, without changing one line of codes.
TopSpin supports many useful techniques in Deep Learning Training, such as gradient accumulation, learning rate warm-up and decay, early stop, mixed precision training, parameter regularizers like VAT automatic validation data evaluation in training, TensorBoard, True gradient for variable batch size in some NLP applications, and provide rich statistics in training log, like network time statistics, training time prediction.
TopSpin, besides, provides inherent AutoML support that searches possible model parameter combinations in one or multiple servers in parallel.
1. Install
2. Examples
>> python3 -m pip install TopSpin
>> python3 -m pip install -r requirements.txtEspecially we require to install protobuf==3.20.* designed in requirements.txt.
Regarding other packages, higher versions might be working too.
- Intent detection Task.
- The core problem is a short sentence classification, and we use TextCNN as its basic model.
- make_features.py
- param.py
- model.py
- train.py
- predict.py
- Debug mode >> python3 example/nlp/intent_detection/train.py
- AutoML mode >> python3 example/nlp/intent_detection/train_auto_starter.py
- Distributed mode >> python3 example/nlp/intent_detection/train_dist_starter.py
- Speaker change detection
- Given a multi-person dialog utterance, we detect the time points speaker changes.
- Object detection.
>> python3 example/nlp/intent_dection/train.py
By design, quck run mode is for debugging, though you could run for simple tasks which require only one GPU. TopSpin, in this mode, would find a free GPU behind your back and assign to your task. You don't need to set GPU id.
This mode is not supportive of GPU allocation, server allocation, task parallelization. Strongly encourage you guys to use mode2, auto run.
You just use TopSpin.ParameterRange to list candiate values. TopSpin would unfold all combinations and sent to GPU queue for paralleled training.
self.iter_num_update_optimizer = ParameterRange([1, 2, 3])
self.embedding_size = ParameterRange([128, 256, 512])
self.kernel_sizes = ParameterRange([[3, 4, 5], [1, 2, 3, 4, 5], [5, 6, 7]])
self.kernel_number = ParameterRange([128, 256])
self.dropout_ratio = ParameterRange([0.1, 0.3])Even you could AutoML training data.
self.train_files = ParameterRange(["train.1.pkl", "train.2.pkl"])The training entry is often defined in a seperate train_auto_starter.py
starter.start_train(
Param.get_instance(),
"example/nlp/intent_dection/train.py",
[starter.Server(None, [1, 3, 4, 6, 7]), # None means current log-in server.
starter.Server("192.168.1.10", [1, 3, 4, 6, 7])]
)Here distributed training means using more than one server to train ONE model, as opposed to
mulitiple models in AutoML.
The philosophy of distributed training is that the model is large enough that all GPUs from designed servers would be occupied. As a result, AutoML would be too costly to apply here.
The training entry is also defined in a seperate train_dist_starter.py.
Set your servers_file in your param.py, and use this starter function
starter.start_distributed_train(
Param.get_instance(),
"example/nlp/intent_dection/train.py",
)
1. >> ssh-keygen -t rsa
2. >> chmod 75[offline_smalldataset.py](src%2Ftopspin%2Fdataset%2Foffline_smalldataset.py)5 ~/.ssh
3. copy ~/.ssh/id_rsa.pub to your target server and save as ~/.ssh/authorized_keys
4. chmod 644 ~/.ssh/authorized_keys
You could use network shared disk.
>> sshfs {user}@{ip}:{data-folder} {local-folder} -o idmap=user -o allow_other
We actually use Network-attached Storage (NAS) system.
The distributed training is not our selling point. TopSpin is a training framework for DL models, which
integrates many proven effective techniques in training, as well as AutoML to greatly improve your training efficieny.
Meantime, it supports multiple-GPU and distributed training, even your daily work might not use them.
TopSpin would detect whether the server cluster has sufficient GPUs to run your task. If not, it would refuse to run.
TopSpin would detect whether your designated data path (or feature path, etc.) are available. If not, it would refuse to run.
In most cases, TopSpin would set this for you. Yet in some very
rare case, TopSpin can not set a correct value. One workable
solution is ls /sys/class/net) to try those value one by one.
A typical net_name is en0, eth0.
No any difference. Actually, TopSpin, in the debugging mode, provides many convenience behind your back, such as setting all potential multi-thread running as single-thread running, and setting GPU number to ONE, ignoring your preset number.
In order to speedup the data loading, could I save a copy of the data in each server, and how to set TopSpin?
Sure. You can do it, and it indeed speedups the data loading as well as the training. In this way you do not need to change any code. It runs perfectly.
Another way is to copy 1/n data to n servers respectively to save disk space. Then in the user's train.py, you need to update the parameters of get_batch_data(...) in Trainer.__init__(...). Change dist.get_rank() and dist.get_world_size() to dist.get_local_rank() and self._param.gpu_num respectively.
Before excuting GPU running, TopSpin would examine the availability of desired GPUs and accessibility of data folder.
Now the user defined function can return a dict,
return {
"loss": loss,
"batch_num": b_x.size(0), # depends on user's application.
"figure": figure # optional
}If your batch_num is just the number of sentences, then you could ignore
this value; otherwise, in some cases this number actually equating to word
number, you have to set it exactly.
Espeically, the key figure records all information to draw by TopSpin
automatically.
figure = {
"F1-score": 0.34,
"accuracy": 0.46
}as well as any information desired to show in a figure. TopSpin would append your training loss, as well as validation and testing data errors by default.
class ModelWrapperBase:
def __init__(self, param: ParamBase, user_model_cls):
...
model = user_model_cls(param)
...In comparison, lower versions permit passing a user model instance into
ModelWrapperBase.
===============================================================================
Since starting to develop 2 years ago, it has been evolved for 5 versions , including one for tensorflow 1.X, one for tensorflow 2.X, three versions for PyTorch. This README mainly introduce the most recent version, namely estimator4 , as it contains the richest functions among all versions.
We list only interfaces of main modules in estimator4, with their private functions omitted.
class ModelWrapperBase:
def __init__(self, param: ParamBase, model: torch.nn.Module):
...
def evaluate_file(self, data_file) -> float:
# return an evaluation result. Smaller, better.
...
def predict(self, *batch):
# Just invoke it
return self._model(*batch)
...class TrainerBase:
def __init__(self,
model_wrapper: ModelWrapperBase,
train_data_iter,
optimizer: typing.Union[Optimizer, None, str]=None):
...
def train_one_batch(self, *batch)-> tensor:
# Given a batch=[batch_x, batch_y], return a loss tensor
...
def early_stop(self, epoch_id, loss_history: list, vali_error_history: list):
# override it when necessary
return False # default
def train(self):
# Just invoke it
...class PredictorBase:
def __init__(self,
model_wrapper: ModelWrapperBase):
# You do NOT need to change it.
def evaluate_file(self, data_file) -> float:
# You do NOT need to change it.
with torch.no_grad():
return self._model_wrapper.evaluate_file(data_file)
def predict(self, batch_data):
# You do NOT need to change it.
with torch.no_grad():
return self._model_wrapper.predict(batch_data)Use ParameterRange to mark all parameters and set their candidate values,
then TopSpin would search all combinations for you in the backhand.
self.lr = ParameterRange([5e-4, 7e-4])
self.warmup_ratio = ParameterRange([0.1, 0.2, 0.3])
self.hidden_dim = ParameterRange([100, 200, 300])We assume a single task requires 2 GPUs, then we run it on my current server and use GPU 2 and 3.
param = MyParam.get_instance(),
param.gpu_num = 2
starter.start_train(
param,
"nlp_tasks/task00_intent/ver_2_2_7_9/train.py",
[starter.Server(None, [2, 3])]
)Let continue with MyParam definition that uses AutoML and it finally
generates
2x3x3=18 parameter variants. Then we have only one server with GPU 2, 3
available. Let's assume one task takes 1 hour, then we have to wait for 18
hours. The waiting looks boring ...
Luckily, Shuang brother say, my task is done, and you can use GPU 0, 1 of that server. We update our starting script like this.
param = MyParam.get_instance(),
param.gpu_num = 2
starter.start_train(
param,
"nlp_tasks/task00_intent/ver_2_2_7_9/train.py",
[starter.Server(None, [0, 1, 2, 3])]
)In this case, two tasks can be run in parallel. Hence, our waiting time is
reduced to 18/(4/2)=9 hours. Life looks a bit better.
More luckily, xiaozhao sister said, my task is also done, you can use RTX8000 that has 8 GPUs. We further update our script like this,
param = MyParam.get_instance(),
param.gpu_num = 2
starter.start_train(
param,
"nlp_tasks/task00_intent/ver_2_2_7_9/train.py",
[starter.Server(None, [0, 1, 2, 3]),
starter.Server("192.168.1.228", [0, 1, 2, 3, 4, 5, 6, 7])]
)Then our waiting time becomes only 18/((4+8)/2)=3 hours.
Life should be much better.
User: How to update my codes to a distributed version?
TopSpin: Sir, you are all set. Try a different starting command.
param = MyParam.get_instance(),
param.servers_file = "my_cluster_servers.txt" # You tell TopSpin this.
param.gpu_num = 8
starter.start_distributed_train(
param,
"patbert/ver_2_2_7_9/train.py",
)Add PAL_Frame folder to your Content Root.
If you run in your Mac terminal, then you need to add to ~/.profile.
export PYTHONPATH={your-PAL-frame-folder}/PAL_frame:$PYTHONPATH
export PYTHONPATH=./:$PYTHONPATH
Test the new settings
>> source ~/.profile
>> echo $PYTHONPATH
You should see the correct values.
Add the above codes to ~/.bashrc, and repeat the testing.
>> source ~/.bashrc
>> echo $PYTHONPATH
class Param(ParamBase):
def __init__(self):
super(Param, self).__init__("nlp_example.intent_dection")
# standard parameters derived from ParamBase
self.gpu_num = 2 # required GPU number.
self.use_gpu = False
self.use_amp = False # mixed precision based training
self.optimizer_name = "Adam"
self.lr = 5e-4
self.batch_size = 16 # for one worker
self.iter_num_update_optimizer = 1 # gradient accumulation
self.warmup_ratio = 0.1 # lr warm percentage of all steps.
self.model_kept_num = 5 # save you models
self.path_feat = "feat/nlp/intent_dection"
self.train_files = [f"{self.path_feat}/train.pkl"]
self.vali_file = f"{self.path_feat}/vali.pkl"
self.test_files = []
self.train_sample_num = 100
self.eval_gap_sample_num = 100 # sample number between two evaluations.
self.epoch_num = 5
# user model specific parameters
self.max_seq_len = 128
self.class_number = 13
self.embedding_size = 128
self.kernel_sizes = [3, 4, 5]
self.kernel_number = 128
self.dropout_ratio = 0.3
self.vocab_size = 60000class Model(nn.Module):
def __init__(self, param: Param):
super(Model, self).__init__()
self._embedding = nn.Embedding(param.vocab_size, param.embedding_size)
self._textcnn = nlp_torch.TextCNN(
kernels=param.kernel_sizes,
in_channel=1,
out_channel=param.kernel_number,
max_seq_len=param.max_seq_len,
dim=param.embedding_size,
dropout=param.dropout_ratio
)
self._textcnn_output_size = len(param.kernel_sizes) * param.kernel_number
self._dense = nlp_torch.Dense(
nn.Linear(self._textcnn_output_size, param.class_number)
)
def forward(self, word_ids):
word_ids = word_ids.unsqueeze(1)
embedding_out = self._embedding(word_ids)
textcnn_out = self._textcnn(embedding_out)
out = self._dense(textcnn_out)
pred_labels = torch.argmax(out, 1)
return out, pred_labelsclass ModelWrapper(ModelWrapperBase):
def __init__(self, param: Param):
super(ModelWrapper, self).__init__(
param, Model(param)
)
def evaluate_file(self, data_file: str):
'''
:param data_file:
:return: smaller better, such as WER, -F1-value, -Accuracy.
'''
start_time = time.time()
all_true_labels = []
all_pred_labels = []
for _, batch in get_batch_data(self._param, [data_file], 1, False):
batch = [e.to(self._device) for e in batch]
b_word_ids, b_labels = batch
logits, pred_labels = self.predict(b_word_ids)
all_true_labels.extend(b_labels.tolist())
all_pred_labels.extend(pred_labels.tolist())
result = Measure.calc_precision_recall_fvalue(
all_true_labels, all_pred_labels
)
total_time = time.time() - start_time
avg_time = total_time / (len(all_true_labels) + 1e-6)
weighted_f = result["weighted_f"]
Logger.info(
f"eval: "
f"file={data_file} weighted_f={weighted_f} result={result} "
f"total_time={total_time:.4f} secs avg_time={avg_time:.4f} sec/sample "
)
Logger.info(f"WEIGHTED_F : {weighted_f}")
return -weighted_f
def predict(self, b_word_ids):
logits, pred_labels = self._model(b_word_ids)
return logits, pred_labelsclass Trainer(TrainerBase):
def __init__(self, param):
super(Trainer, self).__init__(
ModelWrapper(param),
get_batch_data(param, param.train_files, param.epoch_num, True),
None
)
def train_one_batch(self, b_word_ids, b_label):
logits, pred_labels = self._model_wrapper.predict(b_word_ids)
return nn.functional.cross_entropy(logits, b_label, reduction="mean")>> python3 example/nlp/intent_dection/train.py
Run.
>> nohup python3 example/nlp/intent_dection/train_starter.py > log.any_name &
In current folder, log.any_name only tell you if your running is normal. You
should check work/{run_dir}/log/log.rank_0 to read outputs of your model.
{run_dir} can be found in log.any_name.
When your running fails, you can check work/{run_dir}/log/log.node_0, and
the problematic batch data and model are automatically saved into work/{run_dir }/bug/ for your bug replay.
There is a unique method to stop your training, regardless of any starting mode.
python3 palframe/pytorch/estimator4/stopper.py {ParamBase.path_work}
You could have different models run together in a server pool to use your
resources to an extreme.
Use RunManager class.
run_manager = RunManager(tasks, servers)
run_manager.run()See definition of Task and Server in palframe/pytorch/estimator4/starter.py.
class Server:
def __init__(self, ip, gpus):
...
class Task:
def __init__(self, param: ParamBase, source_script_and_params):
...
In the PAT-BERT project, one pretrained model would be tested in 10 downstream
NLP tasks, and each task has multiple set of parameter configurations.
We typically use RunManager to finish 216 tasks in 180 GPUs in just 16 hours.
>> python3 example/nlp/intent_dection/make_features.py
>> nohup python3 example/nlp/intent_dection/train_starter.py > log.any_name &
No any change for your codes.
Debug mode, for debugging only, which uses single GPU (or CPU).Auto mode, your common choice, which supports automatic GPUs and servers allocation.Distributed mode, for quite complex tasks that need more than one server.
So far we only support Debug mode, Distributed mode, and a single task in Auto mode. Todo: to support Auto mode fully.
In the construction function of the user's Parameter class, set the restore_training_from_path_work as the running path of your last run. Its default
value is None.
Capable of speeding up training by 80% without a performance degradation.
Effectively support extremely large dataset.
In the case when Distributed mode is unavailable.
The best evaluation score and its corresponding model are showed and stored.
>> tensorboard --logdir work/{run-log}/tensorboard --bind_all
If your GPUs have unbalanced running tasks, such as different batch sizes, you
could active this function to improve your final performance.
Besides setting param.true_gradient = True,
in train_one_batch, you should return (loss, batch num), instead of just
loss.
-
You model trainable parameter information
Besides total number of parameters are displayed, parameters are sorted by their sizees, percentage on overall parameters.
-
Training time prediction
-
Network efficiency estimate
A default PredictorBase is provided and we have preset some enviroment for you, such as set inference and no gradient.
Each deployment supports only GPU and is defined as param.gpus[0]. Thus, you should set proper param.gpus values.