[Inductor] added aten.exponential_ decomp #91673

min-jean-cho · 2023-01-04T01:27:03Z

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire

pytorch-bot · 2023-01-04T01:27:05Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/91673

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 1ba5577:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

soumith · 2023-01-04T02:51:26Z

torch/_decomp/decompositions.py

@@ -1963,6 +1963,11 @@ def uniform(
    )


+@register_decomposition(aten.exponential_)
+def exponential_(self, rate=1, generator=None):
+    return self.copy_(-1/rate * torch.log(1 - torch.rand_like(self)))


don't you have to use generator in the torch.rand_like call if it is not None?

Thanks @soumith, looks like prim does not support generator yet, I see there is a TODO comment https://github.com/pytorch/pytorch/blob/master/torch/_prims/__init__.py#L2669-L2670.

For now, I think we can just add assert generator is None.

makes sense.

See comment #90869 (comment) , #91673 (comment). Pull Request resolved: #91833 Approved by: https://github.com/jansel

lezcano

This function also has an out-of place variant and an out= variant. A better way to implement it would be to implement the out-of-place variant and then generate the in-place via register_inplace and the out= via the out wrapper.

lezcano · 2023-01-12T23:41:39Z

torch/_decomp/decompositions.py

+@register_decomposition(aten.exponential_)
+def exponential_(self, rate=1, generator=None):
+    assert generator is None
+    return self.copy_(-1 / rate * torch.log1p(-torch.rand_like(self)))


Suggested change

return self.copy_(-1 / rate * torch.log1p(-torch.rand_like(self)))

return self.copy_(-1 / rate * torch.log(torch.rand_like(self)))

If x ~ U(0,1), 1-x ~ U(0,1).

This can go to refs?

Same for the other distributions, yep

@lezcano this won't work with triton (and generally with any fast log approximation) if compute is properly done in fp32, this is explained in the comments in eager code. Exponential distribution should not generate 0's because pdf at 0 is 0. Yet with half dtype fast log approximation would be truncated to 0:

In [28]: max_rand = torch.rand(10000000000, device="cuda").amax() In [29]: def fn(x): ...: return x.log().half() ...: In [30]: opt_fn = torch.compile(fn) /scratch/ngimel/work/pytorch/torch/_dynamo/eval_frame.py:372: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled.Consider setting `torch.set_float32_matmul_precision('high')` warnings.warn( In [31]: fn(max_rand) Out[31]: tensor(-5.9605e-08, device='cuda:0', dtype=torch.float16) #fine, eager log doesn't use fast approximation) In [33]: opt_fn(max_rand) Out[33]: tensor(-0., device='cuda:0', dtype=torch.float16) #fast log approx

Right. I guess we'd have similar issues even if we cast it to float?

Or this is device specific definition of exponential ?

The current formula you have implemented should do. This comment #91673 (review) and #91673 (comment) are still relevant though.

Apparently cpu incorrectly implements exponential in eager, but cuda exponential_ indeed doesn't contain zero, at least until large lambda would cause underflow:

In [5]: torch.empty(100000000, device="cuda").exponential_().min() Out[5]: tensor(5.9605e-08, device='cuda:0')

This is because CPU implementation uses MKL (

pytorch/aten/src/ATen/native/cpu/DistributionKernels.cpp

Line 149 in b8057aa

vdRngExponential(VSL_RNG_METHOD_EXPONENTIAL_ICDF, stream, len,

). Let me check with MKL folks whether the VSL vRngExponential excludes zero or not. If it doesn't, we should fix the MKL-based implementation from PyTorch side. cc @CaoE

The link above says f(x) is defined for x >= a, where a is a displacement parameter.

pytorch/aten/src/ATen/native/cpu/DistributionKernels.cpp

Lines 149 to 150 in b8057aa

vdRngExponential(VSL_RNG_METHOD_EXPONENTIAL_ICDF, stream, len,

(double *)(sample_ptr + begin), 0, 1./lambda);

Here a is 0, so probably that's why cpu generated 0.

ngimel · 2023-01-12T23:50:48Z

torch/_decomp/decompositions.py

@@ -1963,6 +1963,12 @@ def uniform(
    )


+@register_decomposition(aten.exponential_)


what about casting halfs to higher precision for intermediate computations?

Decorating it with pw_cast_for_opmath should be good enough right?

Oh thanks for the catch, pw_cast_for_opmath should do. I'm checking if other ELEMENTWISE_TYPE_PROMOTION_KIND than the default would be applicable.

lezcano

Cool!

min-jean-cho · 2023-01-18T04:44:19Z

@pytorchbot merge

pytorchmergebot · 2023-01-18T04:46:33Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

lezcano · 2023-01-18T08:38:09Z

Also, could you add the relevant OpInfo test?

…113195) Range of sampled random variable needs to be clarified for `torch.tensor.exponential_` whose supported interval is (0, inf) is different from [0, inf] of exponential distribution. Background: #37984 (comment), #48841 (comment), #91673 (comment) Pull Request resolved: #113195 Approved by: https://github.com/albanD

added aten.exponential_ decomp

e71d1ae

github-actions bot added ciflow/inductor module: inductor labels Jan 4, 2023

pytorchbot added the open source label Jan 4, 2023

soumith reviewed Jan 4, 2023

View reviewed changes

min-jean-cho mentioned this pull request Jan 7, 2023

[Inductor] assert generator for random, dropout #91833

Closed

min-jean-cho requested a review from jgong5 January 10, 2023 23:26

pytorchmergebot pushed a commit that referenced this pull request Jan 11, 2023

[Inductor] assert generator for random, dropout (#91833)

364f526

See comment #90869 (comment) , #91673 (comment). Pull Request resolved: #91833 Approved by: https://github.com/jansel

jgong5 approved these changes Jan 11, 2023

View reviewed changes

min-jean-cho marked this pull request as ready for review January 11, 2023 20:23

min-jean-cho requested review from ngimel, jansel and lezcano January 11, 2023 20:24

use log1p for numerical stability

d86b9cc

lezcano reviewed Jan 12, 2023

View reviewed changes

ngimel reviewed Jan 12, 2023

View reviewed changes

lezcano mentioned this pull request Jan 13, 2023

Replace log(1 + x) with log1p(x) #92114

Closed

min-jean-cho added 2 commits January 13, 2023 09:12

(1) outplace wrapper (2) cast halfs

595dd34

wrap to default type promotion kind

7866494

min-jean-cho requested review from jgong5, ngimel and lezcano January 14, 2023 09:21

jansel approved these changes Jan 14, 2023

View reviewed changes

lezcano approved these changes Jan 15, 2023

View reviewed changes

jgong5 approved these changes Jan 16, 2023

View reviewed changes

min-jean-cho added 2 commits January 17, 2023 13:25

Merge branch 'master' into minjean/inductor_decompose_aten_exponential

f42cb08

lint

1ba5577

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 18, 2023

pytorchmergebot added the Merged label Jan 18, 2023

pytorchmergebot closed this in fb50a4b Jan 18, 2023

min-jean-cho mentioned this pull request Nov 8, 2023

[doc] clarify the range of sampled rv for torch.tensor.exponential_ #113195

Closed

jingxu10 mentioned this pull request Nov 10, 2023

[doc] clarify the range of sampled rv for torch.tensor.exponential_ #113428

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Inductor] added aten.exponential_ decomp #91673

[Inductor] added aten.exponential_ decomp #91673

min-jean-cho commented Jan 4, 2023 •

edited by pytorch-bot bot

pytorch-bot bot commented Jan 4, 2023 •

edited

soumith Jan 4, 2023

min-jean-cho Jan 5, 2023

soumith Jan 5, 2023

lezcano left a comment

lezcano Jan 12, 2023

ngimel Jan 12, 2023

lezcano Jan 12, 2023

ngimel Jan 13, 2023 •

edited

lezcano Jan 13, 2023

min-jean-cho Jan 13, 2023

lezcano Jan 13, 2023

ngimel Jan 13, 2023

jgong5 Jan 14, 2023

min-jean-cho Jan 14, 2023

ngimel Jan 12, 2023

jgong5 Jan 14, 2023

min-jean-cho Jan 14, 2023

lezcano left a comment

min-jean-cho commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

lezcano commented Jan 18, 2023

	return self.copy_(-1 / rate * torch.log1p(-torch.rand_like(self)))
	return self.copy_(-1 / rate * torch.log(torch.rand_like(self)))

	vdRngExponential(VSL_RNG_METHOD_EXPONENTIAL_ICDF, stream, len,
	(double *)(sample_ptr + begin), 0, 1./lambda);

		@@ -1963,6 +1963,12 @@ def uniform(
		)


		@register_decomposition(aten.exponential_)

[Inductor] added aten.exponential_ decomp #91673

[Inductor] added aten.exponential_ decomp #91673

Conversation

min-jean-cho commented Jan 4, 2023 • edited by pytorch-bot bot

pytorch-bot bot commented Jan 4, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/91673

✅ No Failures

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lezcano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ngimel Jan 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lezcano left a comment

Choose a reason for hiding this comment

min-jean-cho commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

Merge started

lezcano commented Jan 18, 2023

min-jean-cho commented Jan 4, 2023 •

edited by pytorch-bot bot

pytorch-bot bot commented Jan 4, 2023 •

edited

ngimel Jan 13, 2023 •

edited