(4/n) Data Refactor - Finetuning Scripts

[awaelchli]

Fixes #954
Fixes #951

[rasbt]

When running with 1 epoch and an epoch size of 50000 on Alpaca (default 64 global batch size and microbatch size of 1), that's currently how the learning rate looks like during the course of the training:

This looks good and is exactly what I would expect.

When I reduced the training epoch size to 1000 and adjusted the warmup steps 100 -> 10, then it seems to be doing something weird:

I think we need to adjust the code so that the scheduler steps here

    steps_per_epoch = len(train_dataloader) // train.gradient_accumulation_iters(devices)
    lr_max_steps = train.epochs * steps_per_epoch

are perhaps computed by the epoch_size * train.epochs so that this doesn't happen? We can use len(train_dataloader) // train.gradient_accumulation_iters(devices) then if train.epoch_size is None. What do you think?

[carmocca]

My comments would apply to all the other files too