some step take a long time in trainning

[FrozenZero]

here is the training log :
2023-05-04 08:32:44,031 44k INFO ====> Epoch: 6019, cost 53049.93 s
2023-05-04 08:32:58,036 44k INFO ====> Epoch: 6020, cost 14.01 s
2023-05-04 08:33:12,135 44k INFO ====> Epoch: 6021, cost 14.10 s
2023-05-04 08:33:26,165 44k INFO ====> Epoch: 6022, cost 14.03 s
2023-05-04 08:33:40,417 44k INFO ====> Epoch: 6023, cost 14.25 s
2023-05-04 08:33:54,734 44k INFO ====> Epoch: 6024, cost 14.32 s
2023-05-04 08:34:09,064 44k INFO ====> Epoch: 6025, cost 14.33 s

Has anyone encountered such trouble? One step of the model training process takes a particularly long time.

[34j]

Maybe your batch_size is too close to the limit

[FrozenZero]

Maybe your batch_size is too close to the limit
here is some training info:
GPU:RTX 3090,
batch_size:16
It uses 14G of graphics memory,and 10G free. i don't understand the 'limit' ,Can you explain to me what 'limit' means? thx.