Question about sharding / TP

[njhill]

@OlivierDehaene @Narsil is it expected that the output should be the same (or very close) when using the TP implementation for a given model vs non-sharded/single GPU?

Am seeing quite different output, this is for example with flan-ul2 or flan-t5-xxl with 2 GPUs, using float16 for both single and double GPU cases.

This is using a different fork of the code - I'm still investigating and will also try with the latest from the main branch of this repo as-is, but would be very helpful to know generally what you observe / what's expected.

[Narsil]

@OlivierDehaene @Narsil is it expected that the output should be the same (or very close) when using the TP implementation for a given model vs non-sharded/single GPU?

Yes everything should be very close.

Very close is hard to quantify though. Pytorch version can screw things up, dtype can mess up (some models are more susceptible to bfloat16 to float16 change.

https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py#LL316C15-L316C71
huggingface/transformers#20287

Those might be relevant to your issue too. (Maybe the sharding accentuates some precision issue)

[njhill]

Thanks @Narsil!

[njhill]

@OlivierDehaene thanks for #356, I had also just discovered that bf16 works better for flan-t5-xxl sharded, per @Narsil's suggestion.

However it would be good if we could get it to work in fp16 so that V100's could be used, and since the results are more stable generally for variable length input.

It's interesting that:

• Non-sharded flan-t5-xxl works ok in fp16 (I know changes were made to the impl in the past to accommodate this)
• flan-ul2 actually appears to work ok when sharded in fp16. The output does begin to diverge slightly from a non-TP deployment but the differences are fairly minor

[Narsil]

💡 : Imagine a world, where we could change a model range values arbitrarily without worrying about dtypes :)

Since the model contains all of the weights, it should be doable to find invariants to allow some form of rescaling..