use Fused AdamW as default

[lessw2020]

Currently Titan does not use Fused AdamW as default.
This PR makes Fused the new default.

After investigating current parallelisms using Llama 8B, I found an average speedup of 2.64% as follows:

Fused AdamW	FSDP, eager	2.24%
8B	FSDP, compile	1.63%
	TP	3.62%
	AsyncTP	3.26%
	CP	2.97%
(debug model)	PP	2.10%
Gains	Average	2.64%
	Min	1.63%
	Max	3.62%

Updated to add --optimizer.implementation with ["for-loop", "foreach" and "fused"] support.

Testing:
beyond verifying no issues with all parallelisms above, verified that fused/foreach/ for-loop is being set with the new default config:

[rank0]:Using foreach implementation for optimizer
[rank0]:foreach=True, fused=False

[rank0]:Using for-loop implementation for optimizer
[rank0]:foreach=False, fused=False

[rank0]:Using fused implementation for optimizer
[rank0]:foreach=False, fused=True

[tianyu-l]

Thanks for verifying the performance. I have a suggestion inline.

Also, regarding https://github.com/pytorch/torchtitan/blob/main/torchtitan/components/optimizer.py#L212-L213
can fused and foreach coexist?

[fegin]

fused and foreach cannot coexist.

[lessw2020]

updated based on PR feedback to ensure command line disable is supported (toml support was already there).

Added --optimizer.disable_fused support and verified both cases:
a - using disable_fused in cmd line:

rank0]:Using AdamW optimizer
[rank0]:optimizer_kwargs: {'lr': 0.0008, 'betas': (0.9, 0.95), 'weight_decay': 0.1, 'fused': False, 'foreach': True}

b - not using disable fused (uses default setting for fused):

rank0]:Using AdamW optimizer
[rank0]:optimizer_kwargs: {'lr': 0.0008, 'betas': (0.9, 0.95), 'weight_decay': 0.1, 'fused': True, 'foreach': False}

[tianyu-l]

LGTM.
Could you fix CI before merging?
In particular, please update the fused into a foreach test: https://github.com/pytorch/torchtitan/blob/main/tests/integration_tests.py#L267-L274
Also, there seem to be two optimizers used, which we should trim to one.