Document LDA-related models #2026

steremma · 2018-04-10T17:25:50Z

In this PR I will add documentation for the LDA model as well as all other related modules such as:

ldamodel.py
ldaseqmodel.py
ldamulticore.py
callbacks.py

…ater

…o document-lda

menshikh-iv · 2018-04-10T17:30:23Z

gensim/models/callbacks.py

@@ -18,63 +18,84 @@



missing module docstring with an example of callback usage

menshikh-iv · 2018-04-10T17:30:39Z

gensim/models/callbacks.py

+
+        Parameters
+        ----------
+        **parameters


type missing (for all kwargs)

I do this on purpose because the type can be anything in this function.

menshikh-iv · 2018-04-10T17:31:11Z

gensim/models/callbacks.py

+        viz_env : object, optional
+            Visdom environment to use for plotting the graph. Unused.
+        title : str, optional
+            Title of the graph plot in case `logger == 'visdom'`. Unused.


missing empty line at the end of docstring (here and everywhere)

menshikh-iv · 2018-04-10T17:31:42Z

gensim/models/callbacks.py

-        """
-        Args:
-            model : Trained topic model
+        """""Get the coherence score.


""""" - too much, should be 3

menshikh-iv · 2018-04-10T17:34:54Z

gensim/models/callbacks.py

@@ -354,7 +477,9 @@ class CallbackAny2Vec(object):
    ...         model.save(output_path)
    ...         self.epoch += 1
    ...
-    >>>
+
+    #. Create a callback to print logging information to the console:


better move this part to module docstring.

menshikh-iv · 2018-04-10T17:35:47Z

gensim/models/ldamodel.py

@@ -82,53 +98,70 @@ def update_dir_prior(prior, N, logphat, rho):


 class LdaState(utils.SaveLoad):
-    """
-    Encapsulate information for distributed computation of LdaModel objects.
+    """Encapsulate information for distributed computation of LdaModel objects.


missed :class:...LdaModel

menshikh-iv · 2018-04-10T17:41:30Z

gensim/models/ldamodel.py


    Objects of this class are sent over the network, so try to keep them lean to
    reduce traffic.

    """

    def __init__(self, eta, shape, dtype=np.float32):
+        """Initialize the state.


no need to write "Initialize the state." always, this is obvious (here and everywhere)

menshikh-iv · 2018-04-10T17:42:06Z

gensim/models/ldamodel.py


    >>> doc_lda = lda[doc_bow]

-    The model can be updated (trained) with new documents via
+    The model can be updated (trained) with new documents.

    >>> lda.update(other_corpus)


other_corpus not defined

menshikh-iv · 2018-04-10T17:42:32Z

gensim/models/ldamodel.py

+        ----------
+        prior : {str, list of float, np.ndarray of float, float}
+            A-priori belief on word probability. If `name` == 'eta' then the prior can be:
+


check please that this rendered exactly as you expect

…eed to be fixed.

…error when binded to the model so I removed it

…ded.

menshikh-iv · 2018-04-17T17:11:21Z

gensim/models/callbacks.py

+
+Examples
+--------
+To implement a Callback, inherit from this base class and override one or more of its methods.


Add example with LDA + perplexity

menshikh-iv · 2018-04-17T17:11:59Z

gensim/models/callbacks.py

        """
        for parameter, value in parameters.items():
            setattr(self, parameter, value)

    def get_value(self):
+        """TODO: According to my understanding this method MUST be override. I propose to make ti abstract, or at


yes, you are right

Warnings -------- you must override this method

menshikh-iv · 2018-04-17T17:15:02Z

gensim/models/callbacks.py

-    """
-    Metric class for coherence evaluation
-    """
+    """Metric class for coherence evaluation.s """


See also -> CoherenceModel

menshikh-iv · 2018-04-17T17:18:31Z

gensim/models/ldamodel.py

@@ -50,6 +50,7 @@

 logger = logging.getLogger('gensim.models.ldamodel')


Need to update module docstring (+ example, update current description)

menshikh-iv · 2018-04-17T17:19:37Z

gensim/models/ldamodel.py

+        The number of documents is strected in both state objects, so that they are of comparable magnitude.
+        This procedure corresponds to the stochastic gradient update from
+        `Hoffman et al. :"Online Learning for Latent Dirichlet Allocation"
+        <https://www.di.ens.fr/~fbach/mdhnips2010.pdf>`_.


algorithm number (eq?)

menshikh-iv · 2018-04-17T17:21:08Z

gensim/models/ldamodel.py

+        Examples
+        --------
+
+        #. Train an LDA model using a gensim corpus:


update example of class docstring, remove it from here

menshikh-iv · 2018-04-17T17:25:00Z

gensim/models/ldamulticore.py

@@ -62,21 +62,37 @@



update module docstring (here and everywhere)

menshikh-iv · 2018-04-17T17:27:35Z

gensim/models/ldaseqmodel.py

+        -------
+        list of float
+            Probabilities for each topic in the mixture. This is essentially a point in the `num_topics - 1` simplex.
+
        """


update other classes please

menshikh-iv · 2018-04-22T14:00:52Z

gensim/models/ldaseqmodel.py

+
+#. Persist model to disk:
+
+>>> ldaseq.save("ldaseq")


Please add also "how to get an vectors" to example, this is important always (for all examples)

You mean how to get the vector representation of a single document right? I will add an example.

menshikh-iv

Don't forget to resolve PEP8 issues (see TravisCI) and documentation build issues (see CircleCI)

menshikh-iv · 2018-04-25T06:46:39Z

gensim/models/ldamodel.py

+...     ['survey', 'response', 'eps'],
+...     ['human', 'system', 'computer']
+... ]
+>>> other_dictionary = Dictionary(other_texts)


This is a serious mistake, you should use only one dictionary for vectorization: here you should re-use dictionary that was used to create common_corpus or (better) create it yourself from start, convert common_texts, etc.

Same mistake in other files.

….py)

steremma and others added 5 commits April 4, 2018 21:04

Started documenting the LDA module. This is WIP, squash this commit l…

904ac18

…ater

More work on the LDA module

4746235

Started documenting the callbacks module

3cf885d

Merge branch 'document-lda' of https://github.com/steremma/gensim int…

355b56a

…o document-lda

Finished documenting the Callbacks module

761907c

menshikh-iv added the incubator project PR is RaRe incubator project label Apr 10, 2018

menshikh-iv suggested changes Apr 10, 2018

View reviewed changes

steremma added 2 commits April 11, 2018 15:30

Finished documenting LdaModel, example fix still pending

3dff0fb

Added documentation to the multicore version of LDA. Examples still n…

366bc08

…eed to be fixed.

menshikh-iv added this to To Do in Documentation via automation Apr 13, 2018

steremma and others added 7 commits April 13, 2018 15:05

applied code review corrections

10025de

temporary chunk of work on the sequential lda model

908064e

Fixed examples

59d471b

Fixed examples for multicore LDA

8fb1e51

Fixed examples in the callbacks module. EpochSaver raises a pickling …

80c5058

…error when binded to the model so I removed it

trivial fix

92e1b73

Added documentation for the DTM. Inspection and feedback probably nee…

ca8e749

…ded.

menshikh-iv suggested changes Apr 17, 2018

View reviewed changes

steremma added 3 commits April 18, 2018 17:19

Applied code review corrections

6277ae9

Documented part of the SSLM helper class.

b6e2a25

Finished documenting all helper classes in the sequential LDA module

51d5ad8

menshikh-iv reviewed Apr 22, 2018

View reviewed changes

Improved module level docstring example

95afc67

menshikh-iv suggested changes Apr 25, 2018

View reviewed changes

piskvorky mentioned this pull request Apr 30, 2018

Documentation fixes #2037

Open

steremma and others added 5 commits April 30, 2018 10:47

Fixed example

49d5b47

fix pep8

1fc53c9

formatting fixes in LDA based modules

29286b9

fix formatting in ldaseqmodel (module seems broken)

a4bb211

update lsimodel docs

af76ba6

menshikh-iv moved this from To Do to In Progress (common) in Documentation Jun 20, 2018

menshikh-iv changed the title ~~[DNM] Document LDA~~ Document LDA-related models Jun 20, 2018

menshikh-iv added 6 commits June 20, 2018 11:21

revert breaking code-changes

45464dc

fix PEP8 and documentation build

d39ce3a

cleanup callbacks

6f3ef20

cleanup ldaseqmodel

113b48b

cleanup ldamodel

c8b91c6

cleanup ldamulticore (+ add link to onlineldavb.py script to ldamodel…

0d8862c

….py)

menshikh-iv merged commit e455bc8 into piskvorky:develop Jun 20, 2018

Documentation automation moved this from In Progress (common) to Done Jun 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document LDA-related models #2026

Document LDA-related models #2026

steremma commented Apr 10, 2018

menshikh-iv Apr 10, 2018

menshikh-iv Apr 10, 2018

steremma Apr 13, 2018

menshikh-iv Apr 10, 2018

menshikh-iv Apr 10, 2018

menshikh-iv Apr 10, 2018

menshikh-iv Apr 10, 2018

menshikh-iv Apr 10, 2018

menshikh-iv Apr 10, 2018

menshikh-iv Apr 10, 2018

menshikh-iv Apr 17, 2018

menshikh-iv Apr 17, 2018

menshikh-iv Apr 17, 2018

menshikh-iv Apr 17, 2018

menshikh-iv Apr 17, 2018

menshikh-iv Apr 17, 2018

menshikh-iv Apr 17, 2018

menshikh-iv Apr 17, 2018

menshikh-iv Apr 17, 2018

menshikh-iv Apr 22, 2018

steremma Apr 23, 2018

menshikh-iv Apr 23, 2018

menshikh-iv left a comment

menshikh-iv Apr 25, 2018

		@@ -50,6 +50,7 @@

		logger = logging.getLogger('gensim.models.ldamodel')

Document LDA-related models #2026

Document LDA-related models #2026

Conversation

steremma commented Apr 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

menshikh-iv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment