Fix Int8Tensor v2 to use float32 eps for better bfloat16 precision

[jcaip]

• Add explicit eps=torch.finfo(torch.float32).eps to Int8Tensor.from_hp
• Ensures consistent behavior with v1 AffineQuantizedTensor
• Fixes SQNR regression with bfloat16 inputs (bfloat16 eps=0.0078 vs float32 eps=1.19e-07)

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/3664

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures, 1 Cancelled Job, 5 Pending

As of commit 9976fef with merge base 4b3ebc4 ():

NEW FAILURES - The following jobs have failed:

• Run 1xH100 Tests / test (H100, linux.aws.h100, --pre torch torchvision torchaudio fbgemm-gpu-genai --index-url https... / linux-job (gh)
  RuntimeError: Command docker exec -t 40e03231c783125b563f17ecd5b26378cf873708d0f629d562630cfff3e8618d /exec failed with exit code 1
• Run Regression Tests (aarch64) / test-cpu-ops (macos-14) (gh)
  Process completed with exit code 1.
• Run Regression Tests / test (CPU 2.6, linux.4xlarge, torch==2.6.0 --index-url https://download.pytorch.org/whl/cpu, cpu) / linux-job (gh)
  RuntimeError: Command docker exec -t 6780129e175ac5d53b7d2df14c2a01ba822fe74f8eadf37e576c9a43b01944f6 /exec failed with exit code 1
• Run Regression Tests / test (CPU 2.7, linux.4xlarge, torch==2.7.1 --index-url https://download.pytorch.org/whl/cpu, cpu) / linux-job (gh)
  RuntimeError: Command docker exec -t d6e29844a14bee884fa0965402653b0af07db8cfdf2e6b614c717f0494b89e13 /exec failed with exit code 1
• Run Regression Tests / test (CPU 2.8, linux.4xlarge, torch==2.8.0 --index-url https://download.pytorch.org/whl/cpu, cpu) / linux-job (gh)
  RuntimeError: Command docker exec -t b5faa25b69198efdd85f1db6a7a0124f37620c9a2504dd725996f740478b64f1 /exec failed with exit code 1
• Run Regression Tests / test (CPU 2.9, linux.4xlarge, torch==2.9.1 --index-url https://download.pytorch.org/whl/cpu, cpu) / linux-job (gh)
  RuntimeError: Command docker exec -t 427dc24a062055498d5baff59c613f398803288cc5e762706cb266d3e3721997 /exec failed with exit code 1
• Run Regression Tests / test-nightly (CPU Nightly, linux.4xlarge, --pre torch --index-url https://download.pytorch.org/wh... / linux-job (gh)
  RuntimeError: Command docker exec -t 7f705b83ef0f69119acd338822303c85f35db7e622cdeb74c3f409a8aaa416ac /exec failed with exit code 1
• Run TorchAO Experimental MPS Tests / test-mps-ops / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• Run Regression Tests (aarch64) / test-cpu-ops (linux.arm64.2xlarge) (gh)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[amdfaa]

This is failing on both rocm and cuda.
https://hud.pytorch.org/hud/pytorch/ao/3350b2fd9ccae1e3868045ca90514f5d1ae95a84/1?per_page=50&name_filter=&mergeEphemeralLF=true

[jcaip]

@amdfaa thanks for flagging, should be able to fix this in a bit.