Implement ATen Distributions.cu + Poisson by rachtsingh · Pull Request #58 · probtorch/pytorch

rachtsingh · 2017-12-31T08:43:54Z

I got a bit into the weeds and the scope expanded, but I think this is the right way to do it. This PR (will) implement a pointwise Poisson sampling method for CPU/CUDA, and should be followed up quickly with ports of the Gamma / Dirichlet samplers, and other things we might want to use from distributions.c.

So far:

Renames Generator inside THCRandomTensor.h to be THCGenerator, and fix CUDAGenerator instantiation
Implement Poisson distribution + torch.poisson (Variable only for now, but we can wrap it).
Write tests for the distribution.
Implement a Philox-based sampler for rejection sampling.

rachtsingh · 2018-01-02T17:52:03Z

Ah, not quite ready for review yet, by the way - I still need to debug this CUDA issue and write tests.

I tried refactoring to use ATen instead (which is much cleaner), but I can't figure out how to sample a uniform given a CUDAGenerator reference (which doesn't seem to have a reference to any curandState*-types). If anyone has a pointer that would be great.

fritzo

Note you should register this in the EXAMPLES list in test_distributions.py. When you do so, tests will fail because you do not implement .entropy(). If you don't want to implement .entropy() (which as I understand has no closed form for Poisson), you should allow NotImplementedError in TestDistributionShapes.test_entropy_shape():

- actual_shape = dist.entropy().size()
+ try:
+     actual_shape = dist.entropy().size()
+ except NotImplementedError:
+     continue

fritzo · 2018-01-02T19:51:21Z

nit: Consider reverting unnecessary whitespace changes to ease review by PyTorch folks.

Thanks! I usually squash before review but I'll be more on top of it.

gchanan · 2018-01-03T15:59:28Z

On your question, I don't think we should add calls to get random numbers directly to the generator; the normal pattern is to pass in the generator to a function that generates the random numbers, right?

rachtsingh · 2018-01-03T21:10:35Z

Yeah, that sounds right. I'm not sure about casting the result of unsafeGetTH from void * to be able to sample though - is it ok to change the generator to output the specialized Generator when parsing native_functions.yaml?

EDIT: taken care of, I didn't realize how at::globalContext works. I poked around with CUDA pointers for a bit and realized it's the same as the type's context, at least on my machine. Let me know if there's a multi-GPU issue.

rachtsingh · 2018-01-06T23:56:30Z

+  //g if (!getApplyGrid(totalElements, grid)) {
+  //g   return false;
+  //g }
+  //g grid = dim3(1);


This is temporary - looking for advice on how to best limit the blockDim to 256 when calling CUDA_tensor_apply2

rachtsingh · 2018-01-06T23:59:14Z

Ok, I think it's probably ready for review. @fritzo, given the C++/CUDA changes, it's probably best to ask someone from PyTorch to review as well? I don't know what exactly the plans are for CUDAGenerator, etc.

fritzo

Python code looks good. just one comment on parameter name. I have not reviewed CUDA code; let's do that review on pytorch/pytorch.

fritzo · 2018-01-08T06:21:37Z

nit: preceding underscore usually denotes a private variable. It's probably safer to call this lam or lambda_. Alternatively it would be nice to call it rate which is both semantically meaningful and improves compatibility with tensorflow distributions.

For example self._lambda would be the obvious member name but that looks private. Instead self.lambda_ or self.rate are clearly public.

Both sound good to me. Let's use rate since that's more evocative? I.e. like mean/std in Normal already, and matches up with Exponential, which is the right idea.

fritzo · 2018-01-08T06:28:46Z

@apaszke How should we test the CUDA sampler? Do we use the same tests in test_distributions.py that we use for CPU samplers, or should these new tests live in test_cuda.py?

apaszke · 2018-01-08T09:03:57Z

I think test_distributions.py is ok. Just make unittest skip the tests if CUDA is unavailable (look at test_nn.py and test_autograd.py for examples).

apaszke · 2018-01-08T09:07:18Z

It's better to use tensor instead of zeros if you don't depend on the initial values of the tensor (it will be uninitialized)

apaszke · 2018-01-08T09:07:47Z

Is this unstable in fp32? fp64 math is extremely slow on non-Tesla GPUs, and we should avoid it

apaszke · 2018-01-08T09:08:13Z

Is this the right place for this copyright notice?

apaszke · 2018-01-08T09:10:00Z

Let's avoid adding more of the sampling methods to the global namespace. torch.distributions is the way to expose those samplers

rachtsingh · 2018-01-08T17:58:17Z

Ok, think I addressed comments (thanks for the review!), and am removing poisson from the global namespace [running tests locally right now]. I'll move this PR to pytorch/pytorch afterwards.

gchanan · 2018-01-09T17:01:35Z

if you are having trouble with half, try DISPATCH_ALL_FLOATING_TYPES; dispatch_all uses Half and DISPATCH_ALL_FLOATING_TYPES uses half; we should resolve this, probably always using Half (and providing device conversion functions?). CC @colesbury

Thanks, that works!

Ideally we would always use half for device functions and at::Half for host functions.

rachtsingh · 2018-01-09T19:47:47Z

After doing some checks, it looks like curand_poisson without using the more robust API uses either an inverse CDF sampler or the transformed rejection method, neither of which are accurate for low values of lambda. So, it fails the distribution tests (good thing they work!).

I think there's 3 options - (1) to merge as-is (by raising the threshold on the CUDA version), but to warn that for low values of lambda, the results may be biased, or (2) merge the CPU version, not the GPU version, or (3) punt on this until we get Philox-based PRNGs up, which was what I was planning to tackle next.

apaszke · 2018-01-09T19:55:40Z

I'm fine with all options. I'll leave the decision to @fritzo

fritzo · 2018-01-09T20:59:25Z

Option 1 sounds very reasonable. You could also file an issue at probtorch/pytorch noting that we should revisit this.

rachtsingh · 2018-01-15T21:59:10Z

OK, fixed the issues and added a Philox-based RNG. I don't think the sampling algorithm is the most accurate, but we can work on it in the future I think.

Also, I think there's a bug in the handling of failure_rate (note that I decreased that to make the tests pass), but I'll have to double check the math before fixing.

Let me know if this is good, I'll squash the commits to be 1 (or whatever you think is semantic), and then update the pytorch PR.

rachtsingh · 2018-01-18T16:47:15Z

(discussing in pytorch#4556)

alicanb mentioned this pull request Jan 2, 2018

Implement Poisson distribution #39

Closed

rachtsingh force-pushed the poisson branch from b0f67da to 00f3abd Compare January 2, 2018 17:48

fritzo reviewed Jan 2, 2018

View reviewed changes

rachtsingh force-pushed the poisson branch from 00f3abd to 420291b Compare January 3, 2018 11:22

rachtsingh changed the title ~~Implementation of Poisson distribution + torch.poisson(...)~~ Implement ATen Distributions.cu + Distributions.cpp Jan 3, 2018

rachtsingh mentioned this pull request Jan 3, 2018

Port gamma and dirichlet implementations to CUDA #61

Open

rachtsingh force-pushed the poisson branch from 420291b to c45daf3 Compare January 6, 2018 22:29

rachtsingh changed the title ~~Implement ATen Distributions.cu + Distributions.cpp~~ Implement ATen Distributions.cu + Poisson Jan 6, 2018

rachtsingh force-pushed the poisson branch from c45daf3 to 5c1251c Compare January 6, 2018 23:53

rachtsingh commented Jan 6, 2018

View reviewed changes

rachtsingh force-pushed the poisson branch from 5c1251c to 4ff1328 Compare January 7, 2018 01:54

fritzo reviewed Jan 8, 2018

View reviewed changes

apaszke reviewed Jan 8, 2018

View reviewed changes

rachtsingh force-pushed the poisson branch 3 times, most recently from 8bd52eb to 3d39052 Compare January 9, 2018 07:57

rachtsingh mentioned this pull request Jan 9, 2018

Implement Poisson distribution, added Distributions.cu, CUDAGenerator pytorch/pytorch#4556

Merged

rachtsingh force-pushed the poisson branch from 3b11b8b to 9a677c3 Compare January 9, 2018 09:56

gchanan reviewed Jan 9, 2018

View reviewed changes

Implemented Poisson in Distributions.cu and Distributions.cpp

9ede37d

rachtsingh added 4 commits January 15, 2018 16:56

Changed _lambda to rate

5ee0cca

Added CUDA tests, moved license message, and s/double/float/g

6fe78fe

Fixes - to squash

0667628

Ignore half issue -- to squash

a9591df

rachtsingh force-pushed the poisson branch from b1acc71 to 5a7d114 Compare January 15, 2018 21:57

rachtsingh added 2 commits January 16, 2018 15:26

Adds Philox-based RNG

e7c4fa1

Fixes from comments -- to squash

1a6d1e5

rachtsingh force-pushed the poisson branch from fc385ab to 1a6d1e5 Compare January 16, 2018 20:27

rachtsingh closed this Jan 18, 2018

Uh oh!

Conversation

rachtsingh commented Dec 31, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rachtsingh commented Jan 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fritzo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gchanan commented Jan 3, 2018

Uh oh!

rachtsingh commented Jan 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rachtsingh commented Jan 6, 2018

Uh oh!

fritzo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fritzo commented Jan 8, 2018

Uh oh!

apaszke commented Jan 8, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rachtsingh commented Jan 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rachtsingh commented Jan 9, 2018

Uh oh!

apaszke commented Jan 9, 2018

Uh oh!

fritzo commented Jan 9, 2018

Uh oh!

rachtsingh commented Jan 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rachtsingh commented Jan 18, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rachtsingh commented Dec 31, 2017 •

edited

Loading

rachtsingh commented Jan 2, 2018 •

edited

Loading

rachtsingh commented Jan 3, 2018 •

edited

Loading

rachtsingh commented Jan 8, 2018 •

edited

Loading

rachtsingh commented Jan 15, 2018 •

edited

Loading