add multinomial probability distribution #38820

cxxly · 2022-01-09T03:56:36Z

PR types

New features

PR changes

APIs

Describe

add multinomial distribution with mean, variance,sample,entropy,prob,log_prob method.
update beta,dirichet,exponential family docs.
fix categorical entropy,sample bugs.

paddle-bot-old · 2022-01-09T03:56:41Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

paddle-bot-old · 2022-01-17T02:37:34Z

Sorry to inform you that 7e37d2b's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

iclementine · 2022-01-21T07:40:43Z

python/paddle/distribution/kl.py

@@ -68,6 +68,11 @@ def kl_divergence(p, q):
 def register_kl(cls_p, cls_q):
    """Decorator for register a KL divergence implemention function.

+    when call ``kl_divergence(p, q)`` , will search concrete implemention 


注意语法。

iclementine · 2022-01-21T10:14:39Z

python/paddle/distribution/beta.py

@@ -37,8 +44,15 @@ class Beta(ExponentialFamily):


    Args:
-        alpha (float|Tensor): alpha parameter of beta distribution, positive(>0).
+        alpha (float|Tensor): alpha parameter of beta distribution, 
+            positive(>0), support broadcast semantic. when the parameter is 


注意英文句法，首字母大写等。

iclementine · 2022-01-21T10:21:00Z

python/paddle/distribution/categorical.py

@@ -200,7 +208,8 @@ def kl_divergence(self, other):
        if not in_dygraph_mode():
            check_type(other, 'other', Categorical, 'kl_divergence')

-        logits = self.logits - nn.reduce_max(self.logits, dim=-1, keep_dim=True)
+        logits = self.logits - \
+            nn.reduce_max(self.logits, dim=-1, keep_dim=True)


这些推荐使用 paddle.max ，而不是用 nn 里面的函数。

这是之前同学遗留代码，我先更新下这部分代码，其余遗留在后续规划中，会统一更新

python/paddle/distribution/dirichlet.py

iclementine · 2022-01-21T10:23:08Z

python/paddle/distribution/dirichlet.py


    Args:
        concentration (Tensor): concentration parameter of dirichlet 
-            distribution
+            distribution, also called :math:`\alpha`. when concentration over 


注意句子首字母大写。英文句子用英文逗号，且后附空格。

python/paddle/distribution/distribution.py

iclementine · 2022-01-21T10:25:41Z

python/paddle/distribution/multinomial.py

+
+    Args:
+        total_count (int): Number of trials.
+        probs (Tensor): Probability of a trail falling into each category. Last 


注意拼写。 trial trail

iclementine · 2022-01-21T10:33:26Z

python/paddle/fluid/tests/unittests/distribution/test_distribution_multinomial.py

+        samples = self._dist.sample(sample_shape)
+        sample_mean = samples.mean(axis=0)
+        np.testing.assert_allclose(
+            sample_mean, self._dist.mean, atol=0, rtol=0.20)


这样子的 tolerance 可能会被 CI 系统认为不合理。是否写明一下这么做的原因。

iclementine · 2022-01-21T10:36:31Z

python/paddle/fluid/tests/unittests/distribution/test_distribution_multinomial.py

+        ('value-int', 10, np.array([0.2, 0.3, 0.5]), np.array([2, 3, 5])),
+        ('value-multi-dim', 10, np.array([[0.3, 0.7], [0.5, 0.5]]),
+         np.array([[4., 6], [8, 2]])),
+        # ('value-sum-non-n', 10, np.array([0.5, 0.2, 0.3]), np.array([4,5,2])),


是否添加采多个样的 case. 比如 Batch shape 是 (), 而 sample shape 是 (2,）类似这样的。

iclementine · 2022-01-21T10:43:37Z

python/paddle/fluid/tests/unittests/distribution/test_distribution_multinomial_static.py

+    def setUp(self):
+        self.prog = paddle.static.Program()
+        self.exe = paddle.static.Executor()
+        with paddle.static.program_guard(prog):


最好写两个 program, 虽然这里并不创建参数。

iclementine · 2022-01-27T12:04:08Z

python/paddle/distribution/kl.py

+    function registered by ``register_kl``, according to multi-dispatch pattern. 
+    If find the implemention function, it will return the result, or not will 
+    raise ``NotImplementError`` exception. User can register implemention 
+    funciton by the decorator. 


implemention functions (plural forum);

If an implemention function is found;

ortherwise, it will raise a NotImplementError exception;

Users

functions

iclementine · 2022-01-27T12:04:56Z

python/paddle/distribution/kl.py

@@ -167,7 +170,7 @@ def _kl_uniform_uniform(p, q):

 @register_kl(ExponentialFamily, ExponentialFamily)
 def _kl_expfamily_expfamily(p, q):
-    """compute kl-divergence using `Bregman divergences` 
+    """Compute kl-divergence using `Bregman divergences` 
    https://www.lix.polytechnique.fr/~nielsen/EntropyEF-ICIP2010.pdf


Use correct hyperlink format of rst.

iclementine · 2022-01-27T12:05:56Z

python/paddle/distribution/kl.py

@@ -205,5 +208,5 @@ def _kl_expfamily_expfamily(p, q):


 def _sum_rightmost(value, n):
-    """sum value along rightmost n dim"""
+    """Sum value along rightmost n dim"""


elements ...(plural form)
dimensions.

iclementine · 2022-01-27T12:08:03Z

python/paddle/distribution/beta.py

@@ -37,8 +44,14 @@ class Beta(ExponentialFamily):


    Args:
-        alpha (float|Tensor): alpha parameter of beta distribution, positive(>0).
-        beta (float|Tensor): beta parameter of beta distribution, positive(>0).
+        alpha (float|Tensor): Alpha parameter. It support broadcast semantic. 


It supports
Sementics.
is a tensor
represents
distributions
a

iclementine · 2022-01-27T12:13:23Z

python/paddle/distribution/dirichlet.py

-        concentration (Tensor): concentration parameter of dirichlet 
-            distribution
+        concentration (Tensor): "Concentration" parameter of dirichlet 
+            distribution, also called :math:`\alpha`. When It's over one 


iclementine · 2022-01-27T12:15:03Z

python/paddle/distribution/dirichlet.py

+        concentration (Tensor): "Concentration" parameter of dirichlet 
+            distribution, also called :math:`\alpha`. When It's over one 
+            dimension, the last axis is parameter of distribution,
+            ``event_shape=concentration.shape[-1:]`` , other axes is batch 


axes other than the last are condsiderd batch dimensions.

iclementine · 2022-01-27T12:18:36Z

python/paddle/distribution/multinomial.py

+    Args:
+        total_count (int): Number of trials.
+        probs (Tensor): Probability of a trial falling into each category. Last 
+            axis of probs indexes over categories, other axes index over batches.


The last axis

iclementine

LGTM

TODO: refine documentation in next PR!

…rror bug

jeff41404

approve

XiaoguangHu01

LGTM

cxxly force-pushed the distribution-multinomial branch from 7e37d2b to b4d9a37 Compare January 20, 2022 04:49

cxxly force-pushed the distribution-multinomial branch from b4d9a37 to 3baa872 Compare January 20, 2022 08:15

cxxly force-pushed the distribution-multinomial branch 2 times, most recently from 3f921de to 154a928 Compare January 20, 2022 10:51

cxxly force-pushed the distribution-multinomial branch from 154a928 to 4d2b27a Compare January 20, 2022 13:19

iclementine reviewed Jan 21, 2022

View reviewed changes

python/paddle/distribution/dirichlet.py Outdated Show resolved Hide resolved

iclementine reviewed Jan 21, 2022

View reviewed changes

python/paddle/distribution/distribution.py Show resolved Hide resolved

iclementine reviewed Jan 21, 2022

View reviewed changes

cxxly force-pushed the distribution-multinomial branch from 4d2b27a to 2a43a55 Compare January 21, 2022 11:54

cxxly force-pushed the distribution-multinomial branch 2 times, most recently from 1547cc0 to efdea2e Compare January 25, 2022 03:05

cxxly force-pushed the distribution-multinomial branch from efdea2e to 89ce615 Compare January 25, 2022 04:42

cxxly force-pushed the distribution-multinomial branch from 89ce615 to 138fcd0 Compare January 25, 2022 07:55

cxxly force-pushed the distribution-multinomial branch 4 times, most recently from 359cdd6 to 8654cf4 Compare January 27, 2022 05:58

iclementine reviewed Jan 27, 2022

View reviewed changes

iclementine previously approved these changes Jan 27, 2022

View reviewed changes

cxxly added 3 commits January 28, 2022 02:28

add multinomial probability distribution

28f6f8a

fix categorical sample bug when logits less than zero

63ac9d6

fix categorical sample can't pass hypothesis test and entropy shape e…

852300d

…rror bug

cxxly dismissed iclementine’s stale review via 852300d January 28, 2022 02:29

cxxly force-pushed the distribution-multinomial branch from 8654cf4 to 852300d Compare January 28, 2022 02:29

iclementine approved these changes Jan 28, 2022

View reviewed changes

dingjiaweiww approved these changes Jan 28, 2022

View reviewed changes

jeff41404 approved these changes Jan 28, 2022

View reviewed changes

XiaoguangHu01 approved these changes Jan 29, 2022

View reviewed changes

iclementine merged commit 01f606b into PaddlePaddle:develop Jan 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add multinomial probability distribution #38820

add multinomial probability distribution #38820

cxxly commented Jan 9, 2022 •

edited

Loading

paddle-bot-old bot commented Jan 9, 2022

paddle-bot-old bot commented Jan 17, 2022

iclementine Jan 21, 2022

cxxly Jan 21, 2022

iclementine Jan 21, 2022

cxxly Jan 21, 2022

iclementine Jan 21, 2022

cxxly Jan 21, 2022

iclementine Jan 21, 2022 •

edited

Loading

iclementine Jan 21, 2022 •

edited

Loading

cxxly Jan 21, 2022

iclementine Jan 21, 2022

cxxly Jan 21, 2022

iclementine Jan 21, 2022 •

edited

Loading

iclementine Jan 21, 2022

iclementine Jan 27, 2022

iclementine Jan 27, 2022

iclementine Jan 27, 2022

iclementine Jan 27, 2022

iclementine Jan 27, 2022

iclementine Jan 27, 2022

iclementine Jan 27, 2022

iclementine Jan 27, 2022

iclementine left a comment

jeff41404 left a comment

XiaoguangHu01 left a comment

add multinomial probability distribution #38820

add multinomial probability distribution #38820

Conversation

cxxly commented Jan 9, 2022 • edited Loading

PR types

PR changes

Describe

paddle-bot-old bot commented Jan 9, 2022

paddle-bot-old bot commented Jan 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iclementine Jan 21, 2022 • edited Loading

Choose a reason for hiding this comment

iclementine Jan 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iclementine Jan 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iclementine left a comment

Choose a reason for hiding this comment

jeff41404 left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

cxxly commented Jan 9, 2022 •

edited

Loading

iclementine Jan 21, 2022 •

edited

Loading

iclementine Jan 21, 2022 •

edited

Loading

iclementine Jan 21, 2022 •

edited

Loading