ENH: Multivariate t-distribution #12582

gwgundersen · 2020-07-19T15:59:16Z

Reference issue

Closes #10042. Duplicates #11413 because the old branch was out-of-date.

What does this implement/fix?

Adds support for the multivariate t-distribution, particularly pdf, logpdf, and rvs. See my blog post for a high-level discussion of my approach.

Additional information

Completed:

No licensing issues.
Unit tests comparing logpdf and pdf to MATLAB's mvtpdf function; unit tests for arguments, reproducibility (with random_state), and some testable mathematical properties.
All tests pass locally.
All public functions have docstrings and examples.
Code style follows existing content in _multivariate.py.
Docstrings with new functionality.
Added myself to THANKS.txt
Added a documentation page via the docstring.

Would appreciate feedback on:

How to unit test rvs?
Should we support cdf and logcdf given that the multivariate t-distribution does not have a closed-form solution for the CDF? While I have seen some implementations (see Consider support for multivariate student-t distribution? #10042 (comment) in particular), I don't yet fully understand how they work and would prefer adding these methods to be a separate PR.
Should we support entropy? I don't know anything about the differential entropy of the multivariate t-distribution, and would also prefer punting that to a future PR.

gwgundersen · 2020-07-19T16:38:34Z

My tests pass locally, but the CI tests fail for a reason I don't understand, e.g.:

loc = rng.random(3)
E   AttributeError: 'mtrand.RandomState' object has no attribute 'random'

mdhaber · 2020-08-03T08:54:28Z

@gwgundersen Thanks for submitting this. Other opinions may differ, but I suspect it's OK to start without cdf and entropy in this PR and add later, since not all multivariate distributions have all the methods we might like.

I'm not sure if this will help, but try git submodule update --init before make html. You can also see the documentation built by CI: click "Details".

It looks like it's not showing up in the documentation. I think you need to add your distribution to scipy/stats/__init__.py:

Multivariate distributions
==========================
.. autosummary::
   :toctree: generated/
   multivariate_normal   -- Multivariate normal distribution
   matrix_normal         -- Matrix normal distribution
   dirichlet             -- Dirichlet
   wishart               -- Wishart
   invwishart            -- Inverse Wishart
   multinomial           -- Multinomial distribution
   special_ortho_group   -- SO(N) group
   ortho_group           -- O(N) group
   unitary_group         -- U(N) group
   random_correlation    -- random correlation matrices

I'm not sure about the failing test or how to test the RVS; perhaps @WarrenWeckesser has thoughts?

gwgundersen · 2020-08-15T23:11:27Z

@mdhaber, thanks for that suggestion. I tried initializing and updating the git submodule, but it did not seem to work. Is it possible I need to rebuild Scipy from scratch to update the version number based on the commit? I've tried to do that and, oddly, am completely failing to do so—I was able to build Scipy previously, and I don't know if it's a change in the build procedure or something local to me. I wrote more details here.

mdhaber · 2020-08-16T03:59:03Z

@gwgundersen yes, it sounds to me like you need to build SciPy again, too. I'm not sure how to fix your build error, though. Sometimes my approach has been to nuke my environment and start from scratch, but I see that you've written up your issue, so hopefully you'll get a more helpful reply. I've also seen this issue for macs lately, but it looks different from yours.

THANKS.txt

scipy/stats/_multivariate.py

rlucas7 · 2020-09-14T01:28:37Z

@mdhaber and @gwgundersen I did some testing of this PR locally to see if I could get the docstrings to build.

I couldn't set my local changes to track this pr for some reason (not sure why) so I posted the changes here:

rlucas7@56c9d27

it is mostly spacing around sections of docstrings (e.g. before example line, etc).
The examples render locally for my html and latex doc builds.
There may be an extra newline character here or there but that should give you an idea of what's left for docstring changes.

Let me know if I can help further to get this PR to merging, seems close.

mdhaber · 2020-09-15T18:55:20Z

@rlucas7 Do you want me to add those to this PR?

gwgundersen · 2020-09-15T19:41:35Z

@mdhaber and @rlucas7, thanks for keeping this fire alive. I should have time to look at this again later in the week. Also, I have access to a new machine, which may help with my issues building the docs. I hope that building the docs correctly and pushing will fix some of the failing tests.

rlucas7 · 2020-09-16T00:56:20Z

@mdhaber wrote:

@rlucas7 Do you want me to add those to this PR?

Assuming @gwgundersen is ok with that then yes, the edits I made were all only newline characters so nothing beyond cosmetic changes to make sphinx happy.

Would be nice to get the doc build to green so we can focus on the scientific/probabilistic aspects of the PR and move this one forward.

rlucas7 · 2020-09-16T00:58:02Z

@mdhaber and @rlucas7, thanks for keeping this fire alive. I should have time to look at this again later in the week. Also, I have access to a new machine, which may help with my issues building the docs. I hope that building the docs correctly and pushing will fix some of the failing tests.

No worries @gwgundersen please let us know if you run into issues w/dev setup.

mdhaber · 2020-09-16T17:43:17Z

OK I went ahead and pushed @rlucas7's changes to @gwgundersen's branch so we can see how they affect the doc rendering.
@gwgundersen you'll need to pull the changes e.g. git pull origin/10042-multivariate-t-again before continuing work.

scipy/stats/_multivariate.py

scipy/stats/tests/test_multivariate.py

rlucas7

@mdhaber I left a couple suggested changes to get the rvs reproducibility test to pass for me locally. One had some weird issue trying to overwrite the suggested change you made, perhaps you can take a look, and if they look ok to you, commit and see if gets to green?

I also left several comments, mostly minor, 2 less minor ones though.

scipy/stats/_multivariate.py

scipy/stats/tests/test_multivariate.py

scipy/stats/_multivariate.py

scipy/stats/tests/test_multivariate.py

rlucas7

@gwgundersen hopefully my review/clarifying comments here help. Ping me in comments if still unclear. I think is mostly a matter of adding some branching for np.inf df case.

scipy/stats/_multivariate.py

scipy/stats/tests/test_multivariate.py

rlucas7

@gwgundersen apologies for my confusing review yesterday. For the df parameter the way things are currently written in this PR, everything should work correctly if df is e.g. 1.1 except for the check that you have for int. So replace that check with the suggested change and I think the PR is good from that side. I think my comments for when df==np.inf is/are still valid though, let me know if unclear.

scipy/stats/_multivariate.py

rlucas7

@gwgundersen left a couple more comments. I noticed something weird with the shapes, not sure the cause but behavior is inconsistent:

>>> import numpy as np
>>> import scipy.stats as ss
>>> loc = np.zeros(4)
>>> shape = np.eye(4)
>>> df = 1
>>> shape[2, 2] = 0
>>> ss.multivariate_t(loc, shape, df, allow_singular=True).pdf(np.zeros(4))
array([0.07599089])
>>> ss.multivariate_t(loc, shape, np.inf, allow_singular=True).pdf(np.zeros(4))
0.06349363593424098

Could you take a look and add a test for that one?

Left a couple suggestions for the last unit test that was previously puzzling me-now I think I understand that one better.

Most other comments are cosmetic or style changes.

scipy/stats/tests/test_multivariate.py

scipy/stats/_multivariate.py

scipy/stats/tests/test_multivariate.py

scipy/stats/_multivariate.py

gwgundersen · 2020-09-23T11:59:15Z

I noticed something weird with the shapes, not sure the cause but behavior is inconsistent:

>>> import numpy as np
>>> import scipy.stats as ss
>>> loc = np.zeros(4)
>>> shape = np.eye(4)
>>> df = 1
>>> shape[2, 2] = 0
>>> ss.multivariate_t(loc, shape, df, allow_singular=True).pdf(np.zeros(4))
array([0.07599089])
>>> ss.multivariate_t(loc, shape, np.inf, allow_singular=True).pdf(np.zeros(4))
0.06349363593424098

Ah, good catch. When using np.inf, we delegate to multivariate_normal, which uses _squeeze_output, converting values like [[2]] to 2. I've added _squeeze_output to multivariate_t's _logpdf function and rvs functions. I've added a new unit test test_shape_correctness:

def test_shape_correctness(self):
    # pdf and logpdf should return scalar when the 
    # number of samples in x is one.
    dim = 4
    loc = np.zeros(dim)
    shape = np.eye(dim)
    df = 1
    x = np.zeros(dim)
    res = multivariate_t(loc, shape, df).pdf(x)
    assert np.isscalar(res)
    res = multivariate_t(loc, shape, df).logpdf(x)
    assert np.isscalar(res)

    # pdf() and logpdf() should return probabilities of shape 
    # (n_samples,) when x has n_samples.
    n_samples = 7
    x = np.random.random((n_samples, dim))
    res = multivariate_t(loc, shape, df).pdf(x)
    assert (res.shape == (n_samples,))
    res = multivariate_t(loc, shape, df).logpdf(x)
    assert (res.shape == (n_samples,))

    # rvs() should return scalar unless a size argument is applied.
    res = multivariate_t(np.zeros(1), np.eye(1), 1).rvs()
    assert np.isscalar(res)

    # rvs() should return vector of shape (size,) if size argument 
    # is applied.
    size = 7
    res = multivariate_t(1, 1, 1).rvs(size=size)
    assert (res.shape == (size,))

scipy/stats/tests/test_multivariate.py

rlucas7 · 2020-09-24T00:54:40Z

scipy/stats/tests/test_multivariate.py

+        try:
+            multivariate_t(loc, shape, df, allow_singular=True)
+        except np.linalg.LinAlgError:
+            pytest.fail("Test failed by unexpectedly raising `LinAlgError`.")
+


see comment above-writing unit tests that happy code paths do not raise exceptions is unnecessary.

gwgundersen · 2020-10-05T21:14:58Z

@rlucas, what if I just do a git push --force with my rebased branch, despite these failing tests? I suspect the issue has to do with my machine or OS (Ubuntu 20.04.1), since:

The unit tests pass for you.
The code and tests appear to rely on LAPACK.
The same unit tests fail when using a newly cloned version of master.
The same unit tests fail when using my current branch (not rebased).

Alternatively, I could ask the developers at #11524 for help, since that PR introduced the unit tests that are failing for me.

gwgundersen · 2020-10-05T21:21:39Z

Also, the failing tests "smell" like they may be caused by differing mathematical or algorithmic conventions, e.g.:

>>> np.isclose(u1_2, u1)
array([[False,  True, False, ..., False, False, False],
       [False,  True, False, ..., False, False, False],
       [False,  True, False, ..., False, False, False],
       ...,
       [False, False, False, ..., False, False, False],
       [False,  True, False, ..., False, False, False],
       [False,  True, False, ..., False, False, False]])
>>> u1_2[0,0]
0.058479194
>>> u1[0,0]
-0.058478586
>>> u1_2[0,2]
0.040936727
>>> u1[0,2]
-0.040936757

rlucas7 · 2020-10-05T23:47:06Z

Try python runtests.py -v -s stats That will run only the stats tests. If that passes for you locally then yeah I think a force push is fine. Also, can you leave a comment on the other PR with the failing test stack trace? If you’re seeing that fail others are or will too. I run a Mac box so if this is specific to Ubuntu (Not sure if it is) I can’t help much.

…

-Lucas Roberts

On Oct 5, 2020, at 5:21 PM, Gregory Gundersen ***@***.***> wrote: Also, the failing tests "smell" like they may be caused by differing mathematical or algorithmic conventions, e.g.: >>> np.isclose(u1_2, u1) array([[False, True, False, ..., False, False, False], [False, True, False, ..., False, False, False], [False, True, False, ..., False, False, False], ..., [False, False, False, ..., False, False, False], [False, True, False, ..., False, False, False], [False, True, False, ..., False, False, False]]) >>> u1_2[0,0] 0.058479194 >>> u1[0,0] -0.058478586 >>> u1_2[0,2] 0.040936727 >>> u1[0,2] -0.040936757 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Fixed doc building. Removing unneeded newline formatting Added multivariate T to documentation. Removing thanks update test changes to multivariate-t docs

Co-authored-by: Lucas Roberts <rlucas7@users.noreply.github.com>

Fix rendering in docs. Co-authored-by: Matt Haberland <mhaberla@calpoly.edu> Added reference Co-authored-by: Lucas Roberts <rlucas7@users.noreply.github.com> Changed positive definite to positive semidefinite.

Fixes and tests for handling default arguments. Removed unneeded comment. Code and tests for defaulting to multivariate_normal when df=np.inf. Amended code to allow `df` to be non-integer, but must be greater than zero. Added unit tests. Amended documentation for df Call _squeeze_output on pdf result; unit test pdf shapes. Update scipy/stats/tests/test_multivariate.py Co-authored-by: Lucas Roberts <rlucas7@users.noreply.github.com> Update scipy/stats/_multivariate.py Co-authored-by: Lucas Roberts <rlucas7@users.noreply.github.com> Update scipy/stats/_multivariate.py Co-authored-by: Lucas Roberts <rlucas7@users.noreply.github.com> Added squeeze_output and tests for rvs.

Simplified tests using parameterize decorator; fixed names in pdf_correctness test. Use parameterize and assert_equal Use assert_raises alias rather than context manager pattern. Added newline to break up long line Removed test coverage simplified unit test Added numpy to imports Removed unneeded numpy import; made axes eual. Removed explicitly test not raising ValueError with df float; instead, use df float in another passing test.

ilayn · 2020-10-06T10:59:16Z

Just to make sure that this is not a sign problem (since the results are unique up to columns signs), could you please check if this turns out all true

>>> np.allclose(np.abs(u1_2), np.abs(u1), rtol=0., atol=np.finfo(np.float32).eps*100)

gwgundersen · 2020-10-06T12:00:33Z

@rlucas7, I rebased, confirmed the stats tests pass, and force pushed. The CI checks pass, and I didn't bring in any additional commits.

gwgundersen · 2020-10-06T12:04:54Z

@ilayn, that command returns False:

> /home/gwg/projects/scipy/build/testenv/lib/python3.7/site-packages/scipy/linalg/tests/test_decomp_cossin.py(151)test_cossin_separate()
-> assert_allclose(u1_2, u1, rtol=0., atol=10*np.finfo(dtype_).eps)
(Pdb) np.allclose(np.abs(u1_2), np.abs(u1), rtol=0., atol=np.finfo(np.float32).eps*100)
False

Also, despite my linalg tests failing locally, the CI tests all pass.

ilayn · 2020-10-06T13:10:18Z

That can happen also due to an outdated blas/lapack distro (mkl or openblas etc.) Because apparently not only signs are wrong but the results are also different so it is an actual miscalculation.

rlucas7 · 2020-10-07T00:47:48Z

That can happen also due to an outdated blas/lapack distro (mkl or openblas etc.) Because apparently not only signs are wrong but the results are also different so it is an actual miscalculation.

Thanks for commenting on this @ilayn I hadn't think about BLAS version issues.

So that we know in case of future questions, it would be great @gwgundersen if you could let us know which BLAS and version you have running.

rlucas7

making 2 minor (cosmetic) changes to docs and marking approved. I'll leave open until the upcoming weekend to give others some time to comment. If no further comments by Sunday October 11th, at 9am EST, I'll merge to master (sometime after 9am).

scipy/stats/_multivariate.py

gwgundersen · 2020-10-09T20:56:42Z

@ilayn, I'm not sure which BLAS and LAPACK versions are being used. This is what I tried when trying to find out:

>>> import numpy.distutils.system_info as sysinfo

>>> sysinfo.get_info('lapack')
{'libraries': ['lapack', 'lapack'],
 'library_dirs': ['/usr/lib/x86_64-linux-gnu'],
 'language': 'f77'}

>>> sysinfo.get_info('blas') 
{'libraries': ['blas', 'blas'],
 'library_dirs': ['/usr/lib/x86_64-linux-gnu'],
 'include_dirs': ['/usr/local/include',
  '/usr/include',
  '/home/gwg/miniconda3/envs/scipy/include'],
 'language': 'c',
 'define_macros': [('HAVE_CBLAS', None)]}

When I ls in those directories, I see multiple shared object files:

/usr/lib/x86_64-linux-gnu/lapack$ ls
liblapack.a  liblapack.so  liblapack.so.3  liblapack.so.3.9.0

/usr/lib/x86_64-linux-gnu/openblas-pthread$ ls
cmake       libblas.so.3  liblapack.so.3         libopenblasp-r0.3.8.so  pkgconfig
libblas.a   liblapack.a   libopenblas.a          libopenblas.so
libblas.so  liblapack.so  libopenblasp-r0.3.8.a  libopenblas.so.0

I also tried to re-build Scipy from scratch pointing to the latest object files:

export LAPACK=/usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
export BLAS=/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
python setup.py install

But the tests still fail. Happy to try something else if you have advice, but I can't find any info online about how to find the actual BLAS or LAPACK versions being used.

rlucas7 · 2020-10-11T17:26:05Z

Squashed and merged, nice work @gwgundersen

mdhaber · 2020-10-11T18:26:30Z

Thanks @gwgundersen, @rlucas!

powen-kao · 2020-10-11T21:41:37Z

Hello @gwgundersen ,
I find the code in the following line has a problem with Lint check and cause failure in continuous-integration/travis-ci/pr
I guess it's because of space before "="

scipy/scipy/stats/_multivariate.py

Line 4083 in 882849f

dev = x - loc

it produces the following error

36.65s$ pycodestyle scipy benchmarks/benchmarks
scipy/stats/_multivariate.py:4083:12: E221 multiple spaces before operator
1       E221 multiple spaces before operator
The command "pycodestyle scipy benchmarks/benchmarks" exited with 1.

rlucas7 · 2020-10-11T23:25:40Z

@hp5588 I'm seeing that now, I've got a PR to fix it.

ref:
#12938

gwgundersen mentioned this pull request Jul 19, 2020

ENH: Multivariate t-distribution (stale) #11413

Closed

8 tasks

pvanmulbregt added the scipy.stats label Jul 23, 2020

gwgundersen mentioned this pull request Aug 15, 2020

Consider support for multivariate student-t distribution? #10042

Closed

rlucas7 reviewed Aug 29, 2020

View reviewed changes

THANKS.txt Outdated Show resolved Hide resolved

mdhaber reviewed Sep 9, 2020

View reviewed changes

scipy/stats/_multivariate.py Show resolved Hide resolved

mdhaber reviewed Sep 9, 2020

View reviewed changes

scipy/stats/_multivariate.py Outdated Show resolved Hide resolved

mdhaber mentioned this pull request Sep 9, 2020

A Solid Foundation for Statistics in Python with SciPy mdhaber/scipy#26

Closed

rlucas7 added the enhancement A new feature or improvement label Sep 13, 2020

mdhaber reviewed Sep 16, 2020

View reviewed changes

scipy/stats/_multivariate.py Show resolved Hide resolved

rlucas7 reviewed Sep 17, 2020

View reviewed changes

scipy/stats/tests/test_multivariate.py Outdated Show resolved Hide resolved

mdhaber reviewed Sep 17, 2020

View reviewed changes

scipy/stats/tests/test_multivariate.py Outdated Show resolved Hide resolved

rlucas7 requested changes Sep 17, 2020

View reviewed changes

mdhaber reviewed Sep 17, 2020

View reviewed changes

scipy/stats/tests/test_multivariate.py Outdated Show resolved Hide resolved

rlucas7 requested changes Sep 19, 2020

View reviewed changes

rlucas7 reviewed Sep 20, 2020

View reviewed changes

scipy/stats/_multivariate.py Show resolved Hide resolved

scipy/stats/_multivariate.py Outdated Show resolved Hide resolved

rlucas7 requested changes Sep 22, 2020

View reviewed changes

rlucas7 reviewed Sep 23, 2020

View reviewed changes

scipy/stats/_multivariate.py Outdated Show resolved Hide resolved

rlucas7 reviewed Sep 23, 2020

View reviewed changes

scipy/stats/_multivariate.py Outdated Show resolved Hide resolved

rlucas7 requested changes Sep 24, 2020

View reviewed changes

gwgundersen and others added 7 commits October 5, 2020 22:43

Implemented the multivariate t-distribution with tests.

f885f2b

Fixed doc building. Removing unneeded newline formatting Added multivariate T to documentation. Removing thanks update test changes to multivariate-t docs

MAINT: stats: all RNGs have .uniform, not .random

5b389d1

MAINT: stats: replace random shape matrix with identity

38c1885

Co-authored-by: Lucas Roberts <rlucas7@users.noreply.github.com>

Made example contour plot window more complete.

5a401d8

Fix rendering in docs. Co-authored-by: Matt Haberland <mhaberla@calpoly.edu> Added reference Co-authored-by: Lucas Roberts <rlucas7@users.noreply.github.com> Changed positive definite to positive semidefinite.

Removed duplicate assert.

8205572

gwgundersen force-pushed the 10042-multivariate-t-again branch from c622659 to 4af54d7 Compare October 6, 2020 02:48

gwgundersen mentioned this pull request Oct 6, 2020

ENH: Cosine Sine Decomposition #11524

Merged

9 tasks

rlucas7 approved these changes Oct 7, 2020

View reviewed changes

scipy/stats/_multivariate.py Outdated Show resolved Hide resolved

scipy/stats/_multivariate.py Outdated Show resolved Hide resolved

rlucas7 added 2 commits October 6, 2020 21:13

Update scipy/stats/_multivariate.py

11364d1

Update scipy/stats/_multivariate.py

8895413

rlucas7 merged commit 882849f into scipy:master Oct 11, 2020

rlucas7 added this to the 1.6.0 milestone Oct 11, 2020

rlucas7 mentioned this pull request Oct 11, 2020

MAINT: fix linter on master #12938

Merged

mdhaber mentioned this pull request Sep 7, 2021

DOC: update detailed roadmap #14700

Merged

maw501 mentioned this pull request Oct 27, 2021

Add multivariate student-t distribution pyro-ppl/numpyro#1200

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Multivariate t-distribution #12582

ENH: Multivariate t-distribution #12582

gwgundersen commented Jul 19, 2020 •

edited

Loading

gwgundersen commented Jul 19, 2020

mdhaber commented Aug 3, 2020

gwgundersen commented Aug 15, 2020

mdhaber commented Aug 16, 2020

rlucas7 commented Sep 14, 2020 •

edited

Loading

mdhaber commented Sep 15, 2020

gwgundersen commented Sep 15, 2020

rlucas7 commented Sep 16, 2020 •

edited

Loading

rlucas7 commented Sep 16, 2020

mdhaber commented Sep 16, 2020

rlucas7 left a comment

rlucas7 left a comment

rlucas7 left a comment

rlucas7 left a comment

gwgundersen commented Sep 23, 2020 •

edited

Loading

rlucas7 Sep 24, 2020

gwgundersen commented Oct 5, 2020

gwgundersen commented Oct 5, 2020

rlucas7 commented Oct 5, 2020 via email

ilayn commented Oct 6, 2020

gwgundersen commented Oct 6, 2020 •

edited

Loading

gwgundersen commented Oct 6, 2020

ilayn commented Oct 6, 2020

rlucas7 commented Oct 7, 2020

rlucas7 left a comment

gwgundersen commented Oct 9, 2020 •

edited

Loading

rlucas7 commented Oct 11, 2020

mdhaber commented Oct 11, 2020

powen-kao commented Oct 11, 2020

rlucas7 commented Oct 11, 2020

ENH: Multivariate t-distribution #12582

ENH: Multivariate t-distribution #12582

Conversation

gwgundersen commented Jul 19, 2020 • edited Loading

Reference issue

What does this implement/fix?

Additional information

gwgundersen commented Jul 19, 2020

mdhaber commented Aug 3, 2020

gwgundersen commented Aug 15, 2020

mdhaber commented Aug 16, 2020

rlucas7 commented Sep 14, 2020 • edited Loading

mdhaber commented Sep 15, 2020

gwgundersen commented Sep 15, 2020

rlucas7 commented Sep 16, 2020 • edited Loading

rlucas7 commented Sep 16, 2020

mdhaber commented Sep 16, 2020

rlucas7 left a comment

Choose a reason for hiding this comment

rlucas7 left a comment

Choose a reason for hiding this comment

rlucas7 left a comment

Choose a reason for hiding this comment

rlucas7 left a comment

Choose a reason for hiding this comment

gwgundersen commented Sep 23, 2020 • edited Loading

rlucas7 Sep 24, 2020

Choose a reason for hiding this comment

gwgundersen commented Oct 5, 2020

gwgundersen commented Oct 5, 2020

rlucas7 commented Oct 5, 2020 via email

ilayn commented Oct 6, 2020

gwgundersen commented Oct 6, 2020 • edited Loading

gwgundersen commented Oct 6, 2020

ilayn commented Oct 6, 2020

rlucas7 commented Oct 7, 2020

rlucas7 left a comment

Choose a reason for hiding this comment

gwgundersen commented Oct 9, 2020 • edited Loading

rlucas7 commented Oct 11, 2020

mdhaber commented Oct 11, 2020

powen-kao commented Oct 11, 2020

rlucas7 commented Oct 11, 2020

gwgundersen commented Jul 19, 2020 •

edited

Loading

rlucas7 commented Sep 14, 2020 •

edited

Loading

rlucas7 commented Sep 16, 2020 •

edited

Loading

gwgundersen commented Sep 23, 2020 •

edited

Loading

gwgundersen commented Oct 6, 2020 •

edited

Loading

gwgundersen commented Oct 9, 2020 •

edited

Loading