What activation functions for regression neural networks?

[Beluker]

Body

I have recently started coding my first neural network and want to train it on a regression problem, e.g. computing sine of a number. Though I am not sure what activation functions to use, or what range I should have my weights and biases. Also do I train it any differently? Any help appreciated : )

Guidelines

• I have read and understood this category's guidelines before making this post.

[megamiii]

For activation functions, ReLU is generally a solid choice for the hidden layers because it helps avoid some common issues during training, like vanishing gradients. For the output of a regression problem, you might not want any activation function at all, or you could use a linear activation to keep the output range flexible and unbounded.

Your initial weights are best when they're small random numbers. Xavier or He initialization are good starting points.

When it comes to training your model, you'll likely use Mean Squared Error (MSE) as your loss function because it's standard for regression problems. Adam is a popular optimizer choice. For the training, I'd say it's a lot of experimentation, especially with hyperparameters like batch size and learning rate. You should adjust them based on your model's performance.

Good luck and enjoy!

[Beluker]

Thanks, this really helps!