Model training and optimization 🎯

[KarelZe]

• Decide whether I want to treat option trade classification as a proabilistic classification problem or not
• Bundle training in a docker contrainer that can run in any pod or on scc infrastructure
• Add parametrized script to run studies
• Get access to BwUniCluster2.0
• Write script to start studies
• Free memory with gc.collect()
• Set-up pre-commit hooks e. g., mypy
• Implement gradient boosting using catboost
• Implement TabTransformer using TabSurvey
• Try out TabTransformer closer to original paper without einops and default PyTorch implementations for attention. (see. e. g., here.
• Write custom DataSet for PyTorch
• Improve training performance of TabTransformer
• Look into data pipes / data loader 2 for PyTorch
• Implement save callback for PyTorch models
• PyTorch use multiple GPUs. (dataparallel) See here.
• Implement FTTransformer
• Add timing code
• Implement TabNet using TabSurvey
• Set up weights and biases integration as shown here or here
• Set up test cases for gbms
• Set up test cases for classical classifier
• Set up test cases for TabTransformer.
• Set up test cases for FTTransformer. See here
• Set up test cases for TabNet
• Simplify objective code through train() and test() method as done here.
• Add weighted loss for neural networks
• Adjust early stopping of neural networks to work with accuracy
• Migrate heroku db
• Save completed study objects to gcs and track in wandb
• Track saved models in wandb
• Visualize learning curves for best CatBoost model
• Research, if early stopping in neural nets should also be done based on the accuracy
• Add code to obtain feature importances from trained models
• Add shared embeddings to TabTransformer. See paper or here.
• Adress problem how to adress high dimensionality of categorical variables
• Research if there is a broader theory / concept to decay e. g. exponential smooting or weighted regression etc.
• Visualize decay parameter and find optimal decay factor
• Study samples / probabilites where the prediction is wrong
• Fully understand how the target value should be like for gradient boosting -1/1 or 0/1 like in neural nets?
• Add code to study learning curves e. g., in wandb
• Add code for pre-training
• Experiment with learning rate scheduling
• Add code for attention visualization. First viz done in Interpretability with SHAP and attention maps 🐇 #85, but still have to research best approach to combine maps over different attention heads and layers. Finally decided for a method proposed by Chefer et. al. See here and here.
• Get slurm running
• go through Google playbook
• Define rounds for what I want to optimize
• Do searches more systematically. See this article here. and here. and here
• Extend experiment tracking as shown here.
• normalize data
• Set up configuration for training
• Set up concrete action plan how to improve training
• pytorch 2.0 integration
• Differentiate into exploration and exploitation phase
• Think about folding the validation set into the training set and retrain the best configuration
• Check out retraining
• change early stopping criterion
• Set up option to fix some hyperparams through a config or so
• Verify hyperparameter search space
• Use batch size finder (Implemented in Automatically find maximum batch size🥐 #125)
• Add some option to generate results fast
• Implement a retraining
• Replace early stopping with checkpointing?
• Check if logits is the right word in code
• What samples does the model get wrong?
• Add training curves to wandb
• Check in wandb if the hyperparameter search space is chosen optimal
• Rerun studies with different initializations. See how it affects the results
• Add code to average results from different initializations
• Add code for visualizations e. g., hyperparameter search space, influence of randomness etc.
• Do code review with @pheusel or @lxndrblz

[KarelZe]

Most done by now. Won't work on remaining tasks. 🚀