Update ReadTheDocs dependencies through conda

.readthedocs.yaml

@@ -13,12 +13,13 @@ sphinx:
 formats: all
 
 # The conda env will bring along protobuf and python version
+# ALL PYTHON DEPS MUST BE SPECIFIED HERE
+# Look at help/write_docs.md for details
 conda:
   environment: docs/environment.yaml
 
 python:
   install:
-    - requirements: requirements.txt
     - method: pip
       path: .
 

docs/environment.yaml

@@ -29,23 +29,110 @@ dependencies:
   - xz=5.2.5=h516909a_0
   - zlib=1.2.11=h516909a_1006
   - pip:
+    - absl-py==0.9.0
     - alabaster==0.7.12
+    - attrs==19.3.0
+    - awscli==1.18.49
     - babel==2.8.0
+    - bcrypt==3.1.7
+    - blis==0.4.1
+    - bokeh==2.0.2
+    - boto==2.49.0
+    - boto3==1.12.49
+    - botocore==1.15.49
+    - cachetools==4.1.0
+    - catalogue==1.0.0
+    - cffi==1.14.0
     - chardet==3.0.4
+    - click==7.1.2
+    - cloudpickle==1.4.1
+    - colorama==0.4.3
     - commonmark==0.9.1
+    - conllu==2.3.2
+    - cryptography==2.9.2
+    - cycler==0.10.0
+    - cymem==2.0.3
+    - dask==2.15.0
+    - decorator==4.4.2
+    - distributed==2.15.1
     - docutils==0.16
+    - faiss-cpu==1.6.3
+    - filelock==3.0.12
+    - fire==0.3.1
+    - future==0.18.2
+    - gdown==3.10.3
+    - gensim==3.8.2
+    - google-api-core==1.17.0
+    - google-api-python-client==1.8.2
+    - google-auth==1.14.1
+    - google-auth-httplib2==0.0.3
+    - google-auth-oauthlib==0.4.1
+    - googleapis-common-protos==1.51.0
+    - grpcio==1.28.1
+    - h5py==2.10.0
+    - heapdict==1.0.1
+    - httplib2==0.17.3
     - idna==2.9
     - imagesize==1.2.0
     - jinja2==2.11.2
+    - jmespath==0.9.5
+    - joblib==0.14.1
+    - kiwisolver==1.2.0
+    - lxml==4.5.0
+    - markdown==3.2.1
     - markupsafe==1.1.1
+    - matplotlib==3.2.1
+    - more-itertools==8.2.0
+    - msgpack==1.0.0
+    - murmurhash==1.0.2
+    - networkx==2.4
+    - nltk==3.5
+    - nmslib==2.0.6
+    - numpy==1.18.3
+    - oauth2client==4.1.3
+    - oauthlib==3.1.0
     - packaging==20.3
+    - pandas==1.0.3
+    - paramiko==2.7.1
     - pbr==5.4.5
+    - pillow==7.1.2
+    - plac==1.1.3
+    - pluggy==0.13.1
+    - preshed==3.0.2
+    - psutil==5.7.0
+    - py==1.8.1
+    - pyasn1==0.4.8
+    - pyasn1-modules==0.2.8
+    - pybind11==2.5.0
+    - pycparser==2.20
     - pygments==2.6.1
+    - pygsheets==2.0.3.1
+    - pymongo==3.10.1
+    - pynacl==1.3.0
     - pyparsing==2.4.7
+    - pysbd==0.2.3
+    - pysocks==1.7.1
+    - pytest==5.4.1
+    - python-dateutil==2.8.1
+    - pytorch-lightning==0.7.5
     - pytz==2020.1
+    - pyyaml==5.3.1
     - recommonmark==0.6.0
+    - regex==2020.4.4
     - requests==2.23.0
+    - requests-oauthlib==1.3.0
+    - rsa==4.0
+    - s3transfer==0.3.3
+    - sacremoses==0.0.41
+    - scikit-learn==0.22.2.post1
+    - scipy==1.4.1
+    - scispacy==0.2.4
+    - sentencepiece==0.1.86
+    - sklearn==0.0
+    - smart-open==2.0.0
     - snowballstemmer==2.0.0
+    - sortedcontainers==2.1.0
+    - spacy==2.2.4
     - sphinx==3.0.3
     - sphinx-autodoc-typehints==1.10.3
     - sphinx-rtd-theme==0.4.3
@@ -56,6 +143,25 @@ dependencies:
     - sphinxcontrib-jsmath==1.0.1
     - sphinxcontrib-qthelp==1.0.3
     - sphinxcontrib-serializinghtml==1.1.4
+    - sqlitedict==1.6.0
+    - srsly==1.0.2
+    - tblib==1.6.0
+    - tensorboard==2.2.1
+    - tensorboard-plugin-wit==1.6.0.post3
+    - termcolor==1.1.0
+    - thinc==7.4.0
+    - tokenizers==0.5.2
+    - toolz==0.10.0
+    - torch==1.5.0
+    - tornado==6.0.4
+    - tqdm==4.45.0
+    - transformers==2.8.0
+    - typing-extensions==3.7.4.2
+    - uritemplate==3.0.1
     - urllib3==1.25.9
+    - wasabi==0.6.0
+    - wcwidth==0.1.9
+    - werkzeug==1.0.1
+    - zict==2.0.0
 prefix: /home/jsybran/anaconda3/envs/latest
 

docs/help/write_docs.md

@@ -103,37 +103,26 @@ will take a while, but the result should appear online after a few minutes.
 The hardest part about ReadTheDocs is getting the remote server to properly
 install all dependencies needed to install the docs and build agatha. If the
 build fails remotely, but works locally (by running `make html` in the `docs`
-directory) then this is almost certainly the issue. Take a look at
-`.readthedocs.yaml` to see how dependencies are loaded.
-
-Pretty much, there are different types of dependencies that are specified in
-various confutation and requirements files. As always 'requirements.txt' in the
-root directory contains all the modules needed to run Agatha. However, some data
-dependencies are large, and we don't actually need them to build the docs. For
-example, spaCy models and BERT transformers. These now live in
-`data_requirements.txt` and will not be loaded by the remote document server.
-
-Lastly, there's a bunch of dependencies that aren't necessary for Agatha
-generally, but are necessary if you're going to build the docs. For instance,
-the particular version of [Sphinx][1] or the couple of extensions we use. These
-dependencies. We've had some issues making sure these dependencies are loaded
-properly through requirements files, so we turn to [conda].
+directory) then this is almost certainly the issue. In order to configure
+dependencies for ReadTheDocs, you must specify a conda environment in
+`docs/environment.yaml`.  We use conda for this process, as opposed to pip,
+because conda allows for some non-python imports, such as `protobuf`. However,
+keep in mind that as developers, we will want to keep `requirements.txt` updated
+whenever possible.
 
 ReadTheDocs loads conda first and loads whatever we specify in our
 `docs/environment.yaml` file. Take a look at [these docs][9] for how conda loads
 from yaml files. Conda also allows us to specify non-python dependencies, such
-as [protobuf][10] that is necessary to build agatha.
-
-To update the conda environment used by ReadTheDocs, you need to [create a conda
-environment yaml file][8]. However, as developers, we often will just want a
-`requirements.txt` file to add to an already-existing environment. Here's the
-best way to add a documentation dependency, or a dependency that cannot be
-installed via `pip`.
+as [protobuf][10] that is necessary to build agatha.  To update the conda
+environment used by ReadTheDocs, you need to [create a conda environment yaml
+file][8].  Note that ReadTheDocs does _NOT_ load the `requirements.txt` file.
+Here's the best way to add a documentation dependency, or a dependency that
+cannot be installed via `pip`.
 
  1. Run `conda env create -n <unique_name> -f docs/environment.yaml` to create a
     new environment with all of the previously nessesary requirements.
  2. If the new dependency can be installed via pip, add it to the
-    `docs/requirements.txt` file. Then run `pip install -r docs/requirements.txt`.
+    `requirements.txt` file. Then run `pip install -r requirements.txt`.
  3. If the new dependency is only available through conda, simply run `conda
     install <new_dep>`
  4. Run `conda env export > docs/environment.yaml` to update the stored conda

requirements.txt

@@ -7,17 +7,30 @@
 # Required to Construct the Semantic Network And Train Models ##################
 ################################################################################
 
+cloudpickle
+dask
+distributed
 faiss-cpu
 fsspec
+gdown
 gensim
+h5py
 lxml
 networkx
 nltk
+numpy
 pandas
+protobuf
 pygsheets
 pymongo
+pytorch-lightning
 scispacy
+sentencepiece
+sklearn
 spacy
+sqlitedict
+torch
+tqdm
 transformers
 
 
@@ -43,3 +56,29 @@ matplotlib
 
 # Used to run the semmeddb tool
 fire
+
+################################################################################
+# Needed to build the docs #####################################################
+################################################################################
+
+# WARNING: This file will NOT be loaded by ReadTheDocs.org. Instead, this file
+# exists as an easy way for developers to add the documentation dependencies to
+# an existing environment.
+
+# Read help/write_docs.md for more information about how to update the
+# ReadTheDocs dependencies. But tl;dr: update environment.yaml
+
+# Generates Documentation
+sphinx>=3.0.0
+
+# Allows for markdown docs
+recommonmark
+
+# Allow sphinx to see type hints
+sphinx-autodoc-typehints
+
+# Theme
+sphinx-rtd-theme
+
+# Run ApiDoc on Build
+sphinxcontrib-apidoc

setup.py

@@ -111,8 +111,4 @@ def run(self):
       "torch",
       "tqdm",
     ],
-    dependency_links=[
-      # This replaces the pytorch-lightning until their next version
-      'http://github.com/PyTorchLightning/pytorch-lightning/master'
-    ]
 )