Poincare ball model #45

ferrine · 2019-02-13T23:02:16Z

Huray, we are ready to start.

Image from here

Interesting reading I've done so far:

Hyperbolic Networks
Hyperbolic Entailment Cones for Learning Hierarchical Embeddings
Poincaré GloVe: Hyperbolic Word Embeddings

Some implementation takeouts (mostly from here):

Project results of all operations in the ball of radius 1 − eps, where eps = 10^{-5}
Numerical errors also appear when hyperbolic vectors get closer to 0, perturb with eps = 10^{-15}
Pass clipped to [-15, 15] input to tanh and clip tanh^{-1} to [-1+eps, 1-eps]

CC @leuchine!

leuchine · 2019-02-14T18:59:55Z

Thanks! @ferrine This is very insightful!

KhrulkovV · 2019-02-16T21:19:26Z

Very interesting, perhaps we can add more models (Lorentz hyperboloid model which is more stable for numerical optimization) later?

https://arxiv.org/abs/1806.03417

ferrine · 2019-02-16T23:16:00Z

@KhrulkovV Hi! Happy to see you there. I'll check that paper!

ferrine · 2019-02-17T01:48:48Z

I've tried to write some tests for parallel transport in Poincare ball model and found it a frustrating idea. Parallel transport is very numerically unstable and sometimes gives ridiculous errors. Sometimes passing tests and sometimes not. Trying to see what happens, I've plotted it in matplotlib and it was visually ok and perhaps makes little sense.

BTW, I found a way to add figures to docs, it will be neat

Updates: I found some bugs

Updates 1: bugs fixed!!!!

ferrine · 2019-02-19T00:46:35Z

I'm done. There are some docs+implementation. Waiting for tests pass now

ferrine · 2019-02-21T13:10:46Z

Tests with adam do not work if I use torch script with nontensor arguments, as:
pytorch/pytorch#14455

leuchine · 2019-02-25T21:51:34Z

Hi @ferrine . Thanks for taking your efforts to implement the Poincare model. Here is another paper on Lorentz model, which may be of interests to you. https://arxiv.org/pdf/1902.00913.pdf

Also, may I ask that do you mind sharing your scripts for drawing the tests for parallel transport in Poincare ball model? I plan to use it for testing my own parallel transport implementation. Thanks!

Best Regards,
Qi

ferrine · 2019-02-25T22:19:37Z

Sure, I have docs with plots also. Here is the requested source code https://github.com/ferrine/geoopt/blob/f7e842d559fc6e9de9b02bec31420ec1888df6b8/docs/plots/extended/poincare/parallel_transport.py And docs also https://geoopt.readthedocs.io/en/poincare/extended/poincare.html Docs will change soon. I think they worth to be polished

…

-- Max вт, 26 февр. 2019 г., 0:51 Qi Liu <notifications@github.com>:

Hi @ferrine <https://github.com/ferrine> . Thanks for taking your efforts to implement the Poincare model. Here is another paper on Lorentz model, which may be of interests to you. https://arxiv.org/pdf/1902.00913.pdf Also, may I ask that do you mind sharing your scripts for drawing the tests for parallel transport in Poincare ball model? I plan to use it for testing my own parallel transport implementation. Thanks! Best Regards, Qi — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#45 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALKb7hZwlhRD-pN_8CGj8KjQ9uzlyzd6ks5vRFrngaJpZM4a6dK4> .

leuchine · 2019-02-26T02:33:54Z

Thanks! @ferrine This is very helpful.

ferrine · 2019-03-07T21:38:24Z

Could anyone of you review docs here? Maybe @leuchine or @newkozlukov if you have free time?

The checklist is the following:

Docstring is rendered correctly
Docstring is easy to read
Plots are easy to read, they contain enough information to get a better intuition. (I'm fine to redo them)
There are no typos
There are no math typos
Name conventions are held, the function name is not misleading and its arguments are self explained

If Any of these points are violated, please write a short comment in the corresponding place. I believe this is the way to create better docs.

leuchine · 2019-03-07T21:43:10Z

Thanks! @ferrine I will try to read it this week.

leuchine · 2019-03-07T21:51:50Z

I just glanced through. The doc looks great. I will continue later.

A Typo: Exp, Log -> exp, log

BTW, the Lorentz model including exp, log and parallel transport has already been implemented here. @ferrine @KhrulkovV . https://github.com/facebookresearch/poincare-embeddings/blob/master/hype/lorentz.py
Also, as there are mapping functions from Poincare to Lorentz and vice versa, these code might be reused. Thanks!

KhrulkovV · 2019-03-07T21:54:03Z

There is a small typo in docs: distance on hyperboloid is arccosh(x, y), not arccos(x, y). Will read more carefully later.

ferrine · 2019-03-07T22:49:31Z

@leuchine

A Typo: Exp, Log -> exp, log

I used a capital letter to distinguish this operation from regular exp/log. Is it a convention to use lowercase?

The code from FB is great. I should take their best practice implementations using a Function class. It seems to help with numerical stability

leuchine · 2019-03-07T23:51:04Z

@ferrine It is not a big problem. Just a small convention in math publications I think. :) Low priority. Thanks!

ferrine · 2019-03-15T20:46:33Z

Any additional comments, guys? Eager to continue the PR

ferrine · 2019-03-17T18:37:15Z

I've observed weird behavior in torch.jit.trace (#57) rebased to follow master branch

rrkarim

I'm approving it for now with some uncertainty. More analysis is required for this branch.

ferrine · 2019-03-22T08:46:48Z

There are some fixes I have to do in docs...

…

On Fri, Mar 22, 2019, 10:10 Rasul Kerimov ***@***.*** wrote: ***@***.**** approved this pull request. I'm approving it for now with some uncertainty. More analysis is required for this branch. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#45 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALKb7vZESWBDHDwhq3gNU0CcIjifsJ_cks5vZIHhgaJpZM4a6dK4> .

ferrine · 2019-03-24T10:08:29Z

@leuchine, @KhrulkovV going to finish this in 5hrs today. If you have any comments, this is the best time to write them.

ferrine · 2019-03-24T10:15:46Z

todos for today:

$c$ counter-inuitively stands for **negative** curvature #59
proofread the docs
... coming soon

ferrine · 2019-03-24T15:17:38Z

I faced numerical issues implementing #59, I have concerns about this change

ferrine · 2019-03-25T22:47:02Z

@newkozlukov please approve that you allow my merging this PR without changing the parametrization as you proposed in #59. And you will pick my work up on doing this some time. I approve this change (#59) and if done before 1.0.0 can go without backward compatible code. Once you approve this, I finish this PR and switch to others.

SomeoneSerge

As this has been already too long and blocked many other activities, I'm approving this with assumption that #59 will be fixed in future PRs. I still insist that gyrovector formalism be either removed from the documents or introduced more gradually on per-example basis -- again, in future PRs.

ferrine · 2019-03-31T21:48:53Z

Thank you all!

* add base * add mobius add|sub * fix * missing formulas * remove unused import * add scalar mul, test props * unnessesary cons in project * no cover script functions * add distance * fix typo in comment * add geodesics * add expmap * add functions * add singlt apply * black * fix typos in docs * fix typos in docs * add parallel transport * add dist to a plane and parallel transport. Parallel transport is numerically unstable * fix math bugs * add cool plots * fix small things * add egrad2rgrad * add reference * docs * fix typos * finish Poincare ball implementation * fix small typo * add to inifinite and beyond test * add signed distance * infinity and beyond test * black * docfix * fix docs * fix doc * fix docs typos * add import * add dist0 * optim fails * fix numerics, do not repare broken test * black * some refactoring * fix typo * p.data -> p in optim * update docs a bit * split pr * remove torch script (it gave minor improvemets), delay to pytorch/pytorch#14455 resolution * fix coadd impl * coma typo in docs * nan police float32 * nan police! arcsinh * typo * nan police scripted!\nwratpping artanh in a script function results in umstable behavior * tests * fix typo * another test for parallel transport 0 * random doc fix to make typechecker happy * manifold->module migration fix * black * fix test for poincare (autocast double) * add float32 tests * fix typo * rename project->clip tangent * docs * fix side effect in tests * infinity anb beyond test was failing in torch==1.0.1 but not in torch_nightly, acceptable tolerance differs * add dim argument for poincare math * batched matvec * typo in dist formula * fix tracing issues and grad numerics for Arsinh,Artanh * _max_norm, specify device + dtype * clamp before save to backward in artanh * inplace ops in function impl * black * fix typo * fix spelling * some fixes to docs * euclidean -> Euclidean * black * math font for number * random travis fail? * pytorch future reminder

ferrine changed the title ~~add base~~ Poincare ball model Feb 13, 2019

ferrine force-pushed the poincare branch from f7e842d to 888b663 Compare February 26, 2019 21:27

ferrine force-pushed the master branch from bc1068f to e080c25 Compare March 2, 2019 11:29

ferrine force-pushed the poincare branch 5 times, most recently from 6ce5ed1 to 6025b5e Compare March 3, 2019 15:40

ferrine added 6 commits March 17, 2019 21:34

add base

237272a

add mobius add|sub

0025386

fix

3316cca

missing formulas

34b19ac

remove unused import

f0438c1

add scalar mul, test props

32a4f9e

ferrine added 8 commits March 17, 2019 21:35

add dim argument for poincare math

e70bdd5

batched matvec

2f89406

typo in dist formula

1ff3fe8

fix tracing issues and grad numerics for Arsinh,Artanh

56ba08c

_max_norm, specify device + dtype

e0a5916

clamp before save to backward in artanh

3e6fbfa

inplace ops in function impl

9f0df90

black

09a16cc

ferrine force-pushed the poincare branch from 6dbe25c to 09a16cc Compare March 17, 2019 18:36

rrkarim approved these changes Mar 22, 2019

View reviewed changes

fix typo

6a45012

ferrine added 2 commits March 24, 2019 13:41

fix spelling

de4c9b6

some fixes to docs

ffd1108

SomeoneSerge self-requested a review March 28, 2019 07:39

euclidean -> Euclidean

dc9aa0e

SomeoneSerge approved these changes Mar 31, 2019

View reviewed changes

ferrine added 4 commits March 31, 2019 22:38

black

92a8da3

math font for number

44238f1

random travis fail?

2d47aa1

pytorch future reminder

164a1d4

ferrine merged commit 9d5ff25 into master Mar 31, 2019

ferrine deleted the poincare branch March 31, 2019 21:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Poincare ball model #45

Poincare ball model #45

ferrine commented Feb 13, 2019 •

edited

leuchine commented Feb 14, 2019

KhrulkovV commented Feb 16, 2019 •

edited

ferrine commented Feb 16, 2019

ferrine commented Feb 17, 2019 •

edited

ferrine commented Feb 19, 2019

ferrine commented Feb 21, 2019 •

edited

leuchine commented Feb 25, 2019

ferrine commented Feb 25, 2019 via email

leuchine commented Feb 26, 2019

ferrine commented Mar 7, 2019

leuchine commented Mar 7, 2019

leuchine commented Mar 7, 2019

KhrulkovV commented Mar 7, 2019

ferrine commented Mar 7, 2019 •

edited

leuchine commented Mar 7, 2019

ferrine commented Mar 15, 2019

ferrine commented Mar 17, 2019

rrkarim left a comment

ferrine commented Mar 22, 2019 via email

ferrine commented Mar 24, 2019

ferrine commented Mar 24, 2019

ferrine commented Mar 24, 2019

ferrine commented Mar 25, 2019

SomeoneSerge left a comment

ferrine commented Mar 31, 2019

Poincare ball model #45

Poincare ball model #45

Conversation

ferrine commented Feb 13, 2019 • edited

leuchine commented Feb 14, 2019

KhrulkovV commented Feb 16, 2019 • edited

ferrine commented Feb 16, 2019

ferrine commented Feb 17, 2019 • edited

ferrine commented Feb 19, 2019

ferrine commented Feb 21, 2019 • edited

leuchine commented Feb 25, 2019

ferrine commented Feb 25, 2019 via email

leuchine commented Feb 26, 2019

ferrine commented Mar 7, 2019

leuchine commented Mar 7, 2019

leuchine commented Mar 7, 2019

KhrulkovV commented Mar 7, 2019

ferrine commented Mar 7, 2019 • edited

leuchine commented Mar 7, 2019

ferrine commented Mar 15, 2019

ferrine commented Mar 17, 2019

rrkarim left a comment

Choose a reason for hiding this comment

ferrine commented Mar 22, 2019 via email

ferrine commented Mar 24, 2019

ferrine commented Mar 24, 2019

ferrine commented Mar 24, 2019

ferrine commented Mar 25, 2019

SomeoneSerge left a comment

Choose a reason for hiding this comment

ferrine commented Mar 31, 2019

ferrine commented Feb 13, 2019 •

edited

KhrulkovV commented Feb 16, 2019 •

edited

ferrine commented Feb 17, 2019 •

edited

ferrine commented Feb 21, 2019 •

edited

ferrine commented Mar 7, 2019 •

edited