Nightly build for documentation deployment #615

dhruvbatra · 2020-05-02T21:32:13Z

Motivation and Context

Our CI runs on every commit, but that means if a commit doesn't land in habitat-sim, our documentations (e.g. changes to habitat-api docs) don't update on aihabitat.org.

For instance, facebookresearch/habitat-lab#351 fixed an issue with our documentation and while it has landed into H-API master, aihabitat.org is still out of date.

This PR follows the nightly build example here:
https://circleci.com/docs/2.0/workflows/#nightly-example

to run the update_docs jobs in our CI.

This PR also (I believe, not sure) addresses the issue discussed here:
https://github.com/facebookmicrosites/habitat-website/pull/64

How Has This Been Tested

How does one test the testing script? Who watches the watchmen? 🤷
(Edit: Apparently CI will complain if you break dependency in the CI config, so we have an answer)

Types of changes

-- Added nightly workflow to update_docs job, following example here: https://circleci.com/docs/2.0/workflows/#nightly-example
-- Looks like a valid schedule requires a cron key and a filters key. So the master branch instructions are repeated.

-- Added nightly workflow to `update_docs` job, following example here: https://circleci.com/docs/2.0/workflows/#nightly-example -- Looks like a valid schedule requires a cron key and a filters key. So the master branch instructions are repeated.

-- Fixing CI error.

erikwijmans · 2020-05-02T22:17:41Z

I think the issue here isn't that the doc builder isn't running on habitat-API commits, it's that it doesn't build the docs for habitat-API at all: https://github.com/facebookresearch/habitat-sim/blob/master/.circleci/config.yml#L437

codecov · 2020-05-02T22:27:46Z

Codecov Report

Merging #615 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #615   +/-   ##
=======================================
  Coverage   58.69%   58.69%           
=======================================
  Files         136      136           
  Lines        6162     6162           
  Branches       84       84           
=======================================
  Hits         3617     3617           
  Misses       2545     2545

Flag	Coverage Δ
#CPP	`53.40% <ø> (ø)`
#JavaScript	`10.00% <ø> (ø)`
#Python	`77.24% <ø> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c8497c...b2fdce8. Read the comment docs.

dhruvbatra · 2020-05-03T06:24:01Z

@erikwijmans -- good catch!

I did some more digging, looks like habitat-api docs haven't been built/published for over 7 months (cc: @abhiskk: see pic below for how outdated our api docs are).

I added a job for building them. This should be sufficient, but I'm stuck on this error:
facebookresearch/habitat-lab#385

facebookresearch/habitat-lab#385

erikwijmans · 2020-05-10T18:32:28Z

.circleci/config.yml

@@ -422,6 +422,32 @@ jobs:
          background: true
          paths:
            - ./habitat-api
+      - run:
+          name: Build api documentation


We also need to add this (minus the cache saving) to habitat-api to make sure PRs there don't break the doc built for habitat-api, which would then break habitat-sim's CI.

facebookresearch/habitat-lab#393

erikwijmans · 2020-05-10T18:32:53Z

.circleci/config.yml

+              conda install -y -c conda-forge pelican
+              mkdir -p ../build/docs
+              ./build-public.sh
+      - save_cache:


The save_cache block under "Build sim documentation" can be removed.

Really? Don't we need it for the Update public documentation job?

That save is now being done as part of this block (the Build api documentation block).

mosra · 2020-05-11T15:53:09Z

.circleci/config.yml

+              conda create -y -n habitat-py38 python=3.8
+              . activate habitat-py38; #cd habitat-api
+              conda install -q -y -c conda-forge ninja numpy pytest pytest-cov ccache hypothesis
+              pip install -r requirements.txt --progress-bar off


Shouldn't be needed anymore as facebookresearch/habitat-lab#385 is fixed by #627.

mosra · 2020-05-11T15:53:53Z

.circleci/config.yml

+              cd docs
+              conda install -y -c conda-forge doxygen==1.8.16
+              conda install -y  jinja2 pygments docutils
+              conda install -y -c conda-forge pelican


Pelican is needed for the website only, not for the docs. (So to be also removed in the "Build sim documentation" section where this got copied from.)

…t-sim/pull/615/files/ac3d4afe333ee89469e1dbcb2cebffd5124c9de5#r423141857

erikwijmans · 2020-05-12T03:29:30Z

Probably needs to be merged with master to get the latest m.css changes

mathfac · 2020-05-12T03:38:58Z

.circleci/config.yml

@@ -475,6 +486,16 @@ workflows:
      - cpp_lint
      - js_lint
      - install_and_test_ubuntu
+  nightly:


Dhruv, to test the setup you can send commit with commented

# filters: # branches: # only: master

Then it will run on your current commit and start updating the doc.

mathfac

Dhruv thank you for finalizing this! Great CI skills. As I understand logic will as follow both HSIM & HAPI will have their own build doc step on each PR. And nightly docs will be uploaded to the website from HSIM CI.

CI test need to pass successful and most probably merged with master + try testing without master filter to make sure it works E2E.

mathfac · 2020-05-12T03:55:32Z

Dhruv, you can ssh to CI machine and run experiment commands there, for this you need to enable Github integration:

Then on failed or successful job of your interest run job with ssh:

* master: Interactive RigidObject python tutorial page (#611) Update README.md Separate Scene And Object functionality into different classes (#628) Updated m.css submodule. (#627) Update agent state set state to be less confusing (#614) Direct Navmesh/Meshdata handling; PluginManager instance variable (#623) Documentation generator updates (#624) Update corrade submodule. (#618) Physics primitives (non-colliding) (#622) save_nav_mesh python bindings (#619) --Renamed GltfMeshData -> GenericMeshData; (#617) Update README.md Update README.md

dhruvbatra · 2020-05-13T06:31:56Z

Looked into the failing CI. It looks like the same error as previously observed & fixed on h-api. Merging into master to confirm the hypothesis. If tests pass, I am ready to merge this in.

dhruvbatra · 2020-05-13T22:11:03Z

All CI tests are passing. Merging this in. Thanks @mathfac @erikwijmans @mosra

mathfac · 2020-05-14T18:26:39Z

Thank you! It runs daily now:

dhruvbatra · 2020-05-14T18:35:35Z

Awesome! Next step: nightly conda builds 😄

…tra-patch-2 Nightly build for documentation deployment

* Update preemption to work with automatic requeueing * Preemption updates * Undo * Remove cast

Update config.yml

89f4d64

-- Added nightly workflow to `update_docs` job, following example here: https://circleci.com/docs/2.0/workflows/#nightly-example -- Looks like a valid schedule requires a cron key and a filters key. So the master branch instructions are repeated.

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label May 2, 2020

dhruvbatra requested review from mathfac, mosra and erikwijmans May 2, 2020 21:32

Update config.yml

f0fa22b

-- Fixing CI error.

Building habitat-api docs as well

af39ded

dhruvbatra and others added 2 commits May 9, 2020 14:52

Adding temporary workaround: build h-api doc with py3.8

70e5392

facebookresearch/habitat-lab#385

fixing CI error

133bf9a

erikwijmans reviewed May 10, 2020

View reviewed changes

dhruvbatra mentioned this pull request May 11, 2020

Build api doc in CI facebookresearch/habitat-lab#393

Merged

removing h-sim cache block

ac3d4af

mosra reviewed May 11, 2020

View reviewed changes

Dhruv Batra added 2 commits May 11, 2020 18:13

Dropping pelican following https://github.com/facebookresearch/habita…

64b5f9e

…t-sim/pull/615/files/ac3d4afe333ee89469e1dbcb2cebffd5124c9de5#r423141857

Dropping py3.8

8a5ea94

mathfac reviewed May 12, 2020

View reviewed changes

dhruvbatra merged commit b97cbd9 into master May 13, 2020

dhruvbatra deleted the dhruvbatra-patch-2 branch May 13, 2020 22:12

Ram81 pushed a commit to Ram81/habitat-web-sim that referenced this pull request Dec 10, 2020

Merge pull request facebookresearch#615 from facebookresearch/dhruvba…

3a00972

…tra-patch-2 Nightly build for documentation deployment

luoj1 pushed a commit to luoj1/habitat-sim that referenced this pull request Aug 16, 2022

[Baselines] Preemption update (facebookresearch#615)

b6865eb

* Update preemption to work with automatic requeueing * Preemption updates * Undo * Remove cast

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nightly build for documentation deployment #615

Nightly build for documentation deployment #615

dhruvbatra commented May 2, 2020 •

edited

Loading

erikwijmans commented May 2, 2020

codecov bot commented May 2, 2020 •

edited

Loading

dhruvbatra commented May 3, 2020 •

edited

Loading

erikwijmans May 10, 2020

dhruvbatra May 11, 2020

erikwijmans May 10, 2020

dhruvbatra May 11, 2020

erikwijmans May 11, 2020

dhruvbatra May 11, 2020

mosra May 11, 2020

dhruvbatra May 12, 2020

mosra May 11, 2020

dhruvbatra May 12, 2020

erikwijmans commented May 12, 2020

mathfac May 12, 2020

mathfac left a comment •

edited

Loading

mathfac commented May 12, 2020

dhruvbatra commented May 13, 2020

dhruvbatra commented May 13, 2020

mathfac commented May 14, 2020

dhruvbatra commented May 14, 2020

Nightly build for documentation deployment #615

Nightly build for documentation deployment #615

Conversation

dhruvbatra commented May 2, 2020 • edited Loading

Motivation and Context

How Has This Been Tested

Types of changes

erikwijmans commented May 2, 2020

codecov bot commented May 2, 2020 • edited Loading

Codecov Report

dhruvbatra commented May 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikwijmans commented May 12, 2020

Choose a reason for hiding this comment

mathfac left a comment • edited Loading

Choose a reason for hiding this comment

mathfac commented May 12, 2020

dhruvbatra commented May 13, 2020

dhruvbatra commented May 13, 2020

mathfac commented May 14, 2020

dhruvbatra commented May 14, 2020

dhruvbatra commented May 2, 2020 •

edited

Loading

codecov bot commented May 2, 2020 •

edited

Loading

dhruvbatra commented May 3, 2020 •

edited

Loading

mathfac left a comment •

edited

Loading