Adjust patience and other NER training duration parameters

[FrancescoCasalegno]

We are currently training our NER models using the default parameters of the config.cfg for the duration and early stopping:

Search/data_and_models/pipelines/ner/config.cfg

Lines 88 to 90 in 04aa04b

	patience = 1600
	max_epochs = 0
	max_steps = 20000

But, as discussed in #343 (comment), we can probably set patience and max_steps to much smaller values without any negative impact on the accuracy of our models.

Also, we should look into the learning rate schedule parameters that should probably also be adapted.

[pafonta]

Hello @FrancescoCasalegno,

Also, we should look into the learning rate schedule parameters that should probably also be adapted.

Here is the section in config.cfg (from the PR introducing the transformer component):

Search/data_and_models/pipelines/ner/config.cfg

Lines 136 to 140 in f855db2

	[training.optimizer.learn_rate]
	@schedules = "warmup_linear.v1"
	warmup_steps = 250
	total_steps = 20000
	initial_rate = 0.00005

The documentation for warmup_linear.v1 is there:
https://thinc.ai/docs/api-schedules#warmup_linear

If one wants to use a fixed learning rate, one could do this by deleting the section [training.optimizer.learn_rate] and adding the parameter learn_rate as part of the section [training.optimizer].

The documentation for Adam.v1 is there:
https://thinc.ai/docs/api-optimizers#adam

Note:
Full documentation links for config.cfg: #328 (comment).

[pafonta]

I am having a look into this issue.

There are 2 things to consider, I think:

1. We use in Move to Transformer-based spaCy pipelines for all our NER models #351 the same training parameters as spaCy for their SOTA model, en_core_web_trf. So we might want to be careful about changing them.

2. If we use the scores from the training to select the new parameters, we might have erroneous conclusions. Indeed:
   a. We currently use the scores from our evaluation and on the test set to select the best NER approach.
   b. During training, the evaluation is done by spaCy on the dev set.
   c. We know spaCy evaluation on the dev set and our evaluation on the test set don't compare (see Performances of NER models could be extremely different between dev and test sets #321).
   d. So, the new parameters could do good according to spaCy and on dev but badly according to us and on test.

[pafonta]

Experiment

Goal:

• If we use a constant learning rate and train for a fixed and large number of steps / epochs, could we find from the collected data which configuration would reduce training time while keeping the current accuracy?

setup

Base:

• HEAD of Move to Transformer-based spaCy pipelines for all our NER models #351 (906d95d).

Modifications:

• Use a patience of 0. The training is then done for max_steps, i.e. 20,000.
• Use a constant learning rate of 5e-5, instead of a warmup_linear schedule (which has an initial learning rate of 5e-5).

Observations

1. As expected in point 2 above (Adjust patience and other NER training duration parameters #356 (comment)), we cannot make a correct decision on a number of maximum steps / epochs until Performances of NER models could be extremely different between dev and test sets #321 is sorted out. Indeed, the performances on the dev set and the test set could be in contradiction. For example, in the table below, for entity chemical, the dev F1 score improved by ~30 % while the test F1 score decreased by ~ 10 %. This could be over-fitting but, at the same time, this contradiction isn't there for all the entities. So, there should be another or an additional cause than over-fitting.

2. It doesn't seem there is a correlation between the best dev F1 and the number of steps or epochs. Therefore, we cannot say from the experiment data what would be the best trade-off between training time (i.e. number of steps / epochs) and the performances (i.e. F1 score).

Results

summary

entity	epoch of best dev F1	best dev F1	relative delta compared to #351 (best dev F1)	test F1	relative delta compared to #351 (test F1)
cell_compartment	1201	0.8525	3.28	0.8000	2.71
cell_type	266	0.6400	5.14	0.7059	3.02
chemical	1525	0.6842	28.95	0.4400	-12.63
drug	25	0.8333	4.16	0.7044	6.49
organism	1005	0.8966	0.48	0.5972	-8.66
protein	3464	0.8182	22.73	0.4783	-15.12

➡️ Entities disease, organ, and pathway were not considered due to time constraints.

[pafonta]

Solution

From the experiment, we could have an idea of the minimal number of steps / epochs needed to reach the current accuracy (i.e. from #351, 906d95d).

Based on when the current dev F1 is reached (experiment data below) when training with a constant learning rate without early stopping, we could try:

patience = 800  # steps
max_epochs = 1500  # epochs
max_steps = 0
...
learn_rate = 0.00005

Indeed, the current dev F1 is reached after a maximum of 1,200 epochs and the best dev F1 is reached after an addition of a maximum of 600 steps. Only the entities which got an improvement on the test F1 (i.e. cell_compartment, cell_type, and drug) were considered for the remaining steps.

experiment data

➡️ Here, F1 are F1 on the dev set.

entity	epoch when F1 >= #351 F1	step of best F1	step when F1 >= #351 F1	steps before best F1	best F1	F1 >= #351 F1	#351 F1
cell_compartment	1201	9800	9800	0	0.8525	0.8525	0.8254
cell_type	67	800	200	600	0.6400	0.6154	0.6087
chemical	82	15000	800	14200	0.6842	0.5455	0.5306
drug	25	200	200	0	0.8333	0.8333	0.8
organism	1005	9800	9800	0	0.8966	0.8966	0.8923
protein	334	10400	1000	9400	0.8182	0.7273	0.6667

[pafonta]

Validation

Using the proposed solution above (#356 (comment)):

• The training duration has been divided by 2. It is ~10 hours instead of ~23 hours.
• The accuracy has not changed (on the test set).
• However, on the dev set, the accuracy has decreased for more than 3 % (relative delta) for half the entities.

➡️ In conclusion, from the above and the data below:

1. The observation 1 done in Adjust patience and other NER training duration parameters #356 (comment), related to Performances of NER models could be extremely different between dev and test sets #321, is verified again.
2. We can validate the proposed solution (under the previous point).
3. We can lower the max_epochs to 500 epochs.

❓ Having so much epochs isn't "right" but tuning the hyperparameters is outside the scope of this ticket.

details & data

The accuracy is in average 0.7122 instead of 0.7151. That's a relative decrease of -0.41 %.

There are important relative decreases for disease (-25 %), organ (-7.9 %), organism (-4.9 %), and pathway (-3.6 %).

entity	epoch of F1	step of F1	F1	#351 F1	relative delta compared to #351	best F1	relative delta compared to best
cell_compartment	49	400	0.8197	0.8254	-0.69	0.8525	-3.85
cell_type	266	800	0.6400	0.6087	5.14	0.6400	0.00
chemical	122	1200	0.6512	0.5306	22.73	0.6842	-4.82
drug	24	200	0.8333	0.8000	4.16	0.8333	0.00
organism	41	400	0.8485	0.8923	-4.91	0.8966	-5.36
protein	333	1000	0.7273	0.6667	9.09	0.8182	-11.11
disease	133	400	0.4000	0.5333	-25.00	NA	NA
organ	24	200	0.7037	0.7636	-7.85	NA	NA
pathway	50	200	0.7857	0.8148	-3.57	NA	NA

➡️ NAs: entities disease, organ, and pathway were not considered in #356 (comment). Therefore, there is no "best F1".

[pafonta]

Supplementary material

I had made some plots which I haven't included yet to make the results more digestible.

However, I think they are of interest and it might be worth saving them here for further reference.

They could also be useful for #321.

Here are the plots:

best dev F1 vs epochs

plot

dev F1 & losses vs epochs

➡️ The dotted vertical black line is the epoch for which the best dev F1 is reached. It's exact value is in the plot title.

cell_compartment

cell_type

chemical

drug

organism

protein