[WIP] Add method for Dirichlet Processes

[souravsingh]

This is related to project for implementing non-parameteric Bayesian processes.

I may have made a mistake in writing the method.

@twiecki

[AustinRochford]

@souravsingh I'll write out some feedback and pointers for implementing this today or tomorrow.

[souravsingh]

@AustinRochford Can you give some feedback on the PR? I would love to have this feature merged into PyMC3.

[AustinRochford]

Apologies, work got more hectic than I thought, will try to give some feedback to get you unblocked tonight.

[AustinRochford]

@souravsingh there are two distributions we should eventually build here, and we should decide which one to attack first.

1. The (truncated) Dirichlet process, which is a discrete distribution. The implementation may be simpler, but efficient sampling is a bit more complex.
2. The (truncated) Dirichlet process mixture model. pymc3's Mixture class is useful here.

Let me know which one you want to attack first and we can work on a PR.

[souravsingh]

@AustinRochford I would like to start with Dirichlet process.

[AustinRochford]

Great.

This is a bit complex, since we need to store both the weights and atoms as two separate RVs. I think the place to start would be implementing a (truncated) stick breaking process distribution by filling in something similar to this class definition:

from pymc3.distributions import Continuous

class StickBreakingProcess(Continuous):
    def __init__(self, alpha, shape=None, *args, **kwargs):
        super(StickBreaking, self).__init__(*args, shape=shape, **kwargs)
        
        raise NotImplementedError
        
    def logp(self, value):
        raise NotImplementedError

I'm thinking the shape argument can tell us how many components to truncate at.

@twiecki what do you think the best way to use such a class in combination with an RV producing atoms would be to implement a Dirichlet process so that we can still record the values of the weights in the trace?

[souravsingh]

@twiecki I need some help with the PR.

[twiecki]

@AustinRochford I'm not sure I understand completely, but an easy solution seems to be to just set all atoms that are unused to 0 and always record the full shape in the trace?

[souravsingh]

@AustinRochford I will close the PR and make a new PR will updated changes again.