Build documentation automatically with master merge and update website content #379

mathfac · 2019-12-13T00:19:45Z

Motivation and Context

Before the proposed change we have to build and update documentation manually. As results Docs are quite outdated on our website: https://aihabitat.org/docs/. With current change documentation automatically will be updated with new commits in master powered by CircleCI.

How Has This Been Tested

Run CI and check produced commits for website content.

A commit to master branch of habitat-website:

A commit to gh-pages branch of habitat-website:

Types of changes

Docs change / refactoring / dependency upgrade

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

erikwijmans · 2019-12-13T02:12:46Z

.circleci/config.yml

+              cd ../..
+
+              # Update website
+              git clone git@github.com:facebookmicrosites/habitat-website.git


This actual publish step should be run only once the PR is merged into master

Yes, that's for testing purposes. I added it as incomplete steps in the PR description.

codecov-io · 2019-12-13T08:45:37Z

Codecov Report

Merging #379 into master will decrease coverage by 11.42%.
The diff coverage is 87.36%.

@@             Coverage Diff             @@
##           master     #379       +/-   ##
===========================================
- Coverage   64.49%   53.06%   -11.43%     
===========================================
  Files          64      151       +87     
  Lines        2929     6479     +3550     
  Branches       84       84               
===========================================
+ Hits         1889     3438     +1549     
- Misses       1040     3041     +2001

Flag	Coverage Δ
#CPP	`45.83% <83.5%> (?)`
#JavaScript	`10% <90.62%> (ø)`	⬆️
#Python	`77.42% <100%> (-1.41%)`	⬇️

Impacted Files	Coverage Δ
src/esp/assets/MeshData.h	`100% <ø> (ø)`
src/esp/scene/SemanticScene.h	`26.92% <ø> (ø)`
src/esp/gfx/Simulator.h	`100% <ø> (ø)`
src/esp/nav/PathFinder.h	`100% <ø> (ø)`
src/esp/assets/MeshMetaData.h	`100% <ø> (ø)`
habitat_sim/nav/__init__.py	`100% <ø> (ø)`	⬆️
src/esp/assets/GenericInstanceMeshData.h	`0% <ø> (ø)`
src/esp/geo/geo.cpp	`34.28% <0%> (ø)`
src/esp/gfx/PTexMeshShader.cpp	`0% <0%> (ø)`
src/esp/physics/PhysicsManager.cpp	`45.04% <0%> (ø)`
... and 115 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae63bad...214c85a. Read the comment docs.

apsdehal

LGTM. Left general comments. Thanks for working on this.

apsdehal · 2019-12-16T17:17:59Z

.circleci/config.yml

+                  rm -rf published/docs/${dir}
+                  cp -r ../habitat-sim/build/docs-public/${dir} published/docs/.
+              done
+              git checkout published/docs/habitat-sim/index.html


index.html will never be updated?

It will be update only manually as we have hardcoded links to tutorials. cc @mosra if I understood incorrectly.

Huh, no, it shouldn't behave like that, at all. I wasn't aware of https://github.com/facebookmicrosites/habitat-website/commit/f16a3d1c932a8b9ed1f6f48ef99a9f269ce0a909 -- it's behaving correctly on my side, generating links as intended (testing with current master). I bet it's due to some unfortunate minor difference in docutils on the CI?

In any case, the following patch should make it work correctly, as it explicitly uses the :ref: role instead of relying on the overriden default role:

diff --git a/docs/pages/index.rst b/docs/pages/index.rst index b2fe3e8..c1907aa 100644 --- a/docs/pages/index.rst +++ b/docs/pages/index.rst @@ -14,9 +14,9 @@ over 10,000 FPS multi-process on a single GPU! .. TODO: this is waiting on m.css to propagate page titles to links -- `New Actions <std:doc:new-actions>` -- `Notebooks <std:doc:notebooks>` -- `Creating a stereo agent <std:doc:stereo-agent>` +- :ref:`New Actions <std:doc:new-actions>` +- :ref:`Notebooks <std:doc:notebooks>` +- :ref:`Creating a stereo agent <std:doc:stereo-agent>` .. note-warning::

@mathfac can you confirm the above makes it work correctly, without the git checkout needed?

.circleci/config.yml

apsdehal · 2019-12-16T17:22:57Z

.circleci/config.yml

+              git commit -m "Build ${NOW}"
+              git status
+              git stash
+              git push origin master


It might make sense to push both master/gh-pages together once all of the required copy/pasting is finished. If there is some issue in the below commands, master/gh-pages won't go out of sync.

.circleci/config.yml

mosra

Thank you for this!

Apart from this one thing, I have basically the same comments as @apsdehal.

mosra · 2019-12-17T19:45:09Z

.circleci/config.yml

+              conda install -y -c conda-forge pelican
+              mkdir -p ../build/docs
+              git submodule update --init
+              doxygen


Since doxygen is run by the shell script below again, this isn't necessary, only makes the build longer.

erikwijmans · 2020-01-18T19:44:44Z

.circleci/config.yml

+            - install_and_test_ubuntu
+          filters:
+            branches:
+              only: autodoc


Suggested change

only: autodoc

only: master

?

Yes, planned to change to master, still need for testing.

erikwijmans · 2020-01-18T19:45:35Z

.circleci/config.yml

+              rm -rf __pycache__
+              git add .
+              git commit -m "Build ${NOW}"
+              # git push origin gh-pages


Suggested change

# git push origin gh-pages

git push origin gh-pages

erikwijmans · 2020-01-18T19:52:28Z

.circleci/config.yml

+      - run:
+          name: Update public documentation
+          command: |
+              # Update website


Add

mkdir -p ~/.ssh touch ~/.ssh/known_hosts echo "github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> ~/.ssh/known_hosts

To deal with the fingerprint issue.

Thank you for the snipet! For know add checkout step at the beginning. It should take care of that and may be useful for other manipulation with docs.

mathfac · 2020-01-19T04:03:11Z

Successful doc updated run.
Changed branch and git push, ready for merge.

mathfac · 2020-01-20T18:50:36Z

The documentation update step succeeded on master branch. New "loadGibsonHouse" function is available in the public doc.
Now with every merge into master the public documentation will be updated automatically.

* Add support for eval on all episodes * Add None correctly

…e content (facebookresearch#379) Before the proposed change we have to build and update documentation manually. As results Docs are quite outdated on our website: https://aihabitat.org/docs/. With current change documentation automatically will be updated with new commits in master powered by CircleCI. Install latex, pelican, doxygen and other dependencies needed. Setup securely ssh key for updating website repository. Rebuild Habitat-SIM with wide options enabled for documentation update. Build docs with Doxygen. Build docs with Mcss documentation. Update habitat-website master branch with changed parts of documentation. Update habitat-website gh-pages branch (website update) with changed parts of documentation. Apply master only branch filter. Run build as separate CircleCI job. Add links update for tutorials.

Added autoupdate of documentation on website for Habitat-SIM

35bbee2

mathfac added this to the Better eng milestone Dec 13, 2019

mathfac requested review from mosra, apsdehal, erikwijmans and msbaines December 13, 2019 00:19

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Dec 13, 2019

mathfac force-pushed the autodoc branch from 6b2733a to 257bd04 Compare December 13, 2019 00:22

erikwijmans reviewed Dec 13, 2019

View reviewed changes

mathfac force-pushed the autodoc branch from 257bd04 to a6878d4 Compare December 13, 2019 08:08

Moved upper doc build for testing

c9164f7

mathfac force-pushed the autodoc branch from a6878d4 to c9164f7 Compare December 13, 2019 08:39

apsdehal reviewed Dec 16, 2019

View reviewed changes

mosra reviewed Dec 17, 2019

View reviewed changes

mathfac self-assigned this Dec 28, 2019

mathfac force-pushed the autodoc branch from d2f04bc to 10a71ef Compare January 17, 2020 11:13

Addressed comments, separate master push separately.

ae63bad

mathfac force-pushed the autodoc branch 5 times, most recently from 622a391 to d0f0a7d Compare January 17, 2020 11:54

Merge branch 'master' into autodoc

6f28629

mathfac force-pushed the autodoc branch from d0f0a7d to 6f28629 Compare January 17, 2020 12:07

erikwijmans reviewed Jan 18, 2020

View reviewed changes

mathfac force-pushed the autodoc branch from 7cc9c3e to 012bc91 Compare January 19, 2020 00:31

Added checkout step to avoid github known host issues

3a70456

mathfac force-pushed the autodoc branch from 012bc91 to 3a70456 Compare January 19, 2020 02:55

mathfac requested review from mosra and erikwijmans January 19, 2020 06:56

erikwijmans approved these changes Jan 19, 2020

View reviewed changes

Swapped to master branch, ready to merge.

214c85a

mathfac force-pushed the autodoc branch from 179bbf9 to 214c85a Compare January 19, 2020 17:57

mathfac merged commit 5fa5789 into master Jan 19, 2020

mathfac deleted the autodoc branch January 19, 2020 23:31

eundersander pushed a commit to eundersander/habitat-sim that referenced this pull request Aug 6, 2020

Add support for eval on all episodes (facebookresearch#379)

ec1d508

* Add support for eval on all episodes * Add None correctly

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build documentation automatically with master merge and update website content #379

Build documentation automatically with master merge and update website content #379

mathfac commented Dec 13, 2019 •

edited

Loading

erikwijmans Dec 13, 2019

mathfac Dec 13, 2019

codecov-io commented Dec 13, 2019 •

edited

Loading

apsdehal left a comment

apsdehal Dec 16, 2019

mathfac Dec 18, 2019

mosra Dec 18, 2019

apsdehal Dec 16, 2019

mosra left a comment

mosra Dec 17, 2019

erikwijmans Jan 18, 2020

mathfac Jan 18, 2020

erikwijmans Jan 18, 2020

erikwijmans Jan 18, 2020

mathfac Jan 18, 2020 •

edited

Loading

mathfac commented Jan 19, 2020

mathfac commented Jan 20, 2020

Build documentation automatically with master merge and update website content #379

Build documentation automatically with master merge and update website content #379

Conversation

mathfac commented Dec 13, 2019 • edited Loading

Motivation and Context

How Has This Been Tested

Types of changes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Dec 13, 2019 • edited Loading

Codecov Report

apsdehal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mosra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mathfac Jan 18, 2020 • edited Loading

Choose a reason for hiding this comment

mathfac commented Jan 19, 2020

mathfac commented Jan 20, 2020

mathfac commented Dec 13, 2019 •

edited

Loading

codecov-io commented Dec 13, 2019 •

edited

Loading

mathfac Jan 18, 2020 •

edited

Loading