Deploy development documentation from Travis [not ready to merge]

[jenshnielsen]

This is as much to get some input and see if this makes any sense as a final pull request.

The idea is to upload the documentation build by Travis on the master branch to a git repository on the gh-pages branch. For security this is done in a separate repository from the main code. Travis stores an encrypted private deployment key which can only be used to upload to this repository. According to the travis docs this key can only be decrypted on a main branch and not a PR to protect against leaks.

See http://docs.travis-ci.com/user/deployment/custom/ for the Travis documentation

I imaging that this should be linked from the main list of version on matplotlib.org For this to work as intended it will also require modification of versions.html in the main repository to use absolute rather than relative links and point to this version.

I originally tried using a submodule but that will require modifying the main repo when a new branch is pushed
to link it to the tip of the dev repository.

The one that I am currently building can be seen at http://jenshnielsen.github.io/mplddocs/index.html (Builds every time that I push against the master branch of my mpl fork)

And my fork of the main documentation at http://jenshnielsen.github.io/matplotlibdocs/ with links to the development docs

What do you think? Does it make any sense to upload the development documentation?

[jenshnielsen]

I guess this fails because the key is encrypted for my Travis repository and not the main Matplotlib one.

[mdboom]

I think if you do git checkout --orphan gh-pages than this will create an orphaned commit, meaning it has no parent. That way, this repository won't grow to an an unwieldy size. I don't think there's any need to keep history here.

[mdboom]

This is a great idea. Thanks for working on it.

A couple of things: It seems like the versions.html section isn't being included in either http://jenshnielsen.github.io/mplddocs/index.html or http://jenshnielsen.github.io/matplotlibdocs/. (I didn't investigate further).

The little blurb that says "This is the documentation for 1.5.x" maybe should say "git master" or something?

Other than that, I think this is great. I think it's an important feature that the docs are in a separate repository from either matplotlib itself and matplotlib.github.com to avoid bloating either of those repositories.

[jenshnielsen]

@mdboom
I made the changes to use orphan, thats a great idea.

I was trying to deploy the pages within a subpage of my github repository.
This should fix the issues with broken links. Can you try the version up at http://jenshnielsen.github.io/ ?

Still needs to be done:

• Replace my deployment key with one that matplotlib can decrypt on Travis.
• Create development repository such as matplotlib/mplddocs and replace reference to git@github.com:jenshnielsen/mplddocs.git with this one. [needs owners permission to matplotlib]
• Add link to main page in matplotlib.github.com

[tacaswell]

version.html and one or two other files are not part of main repo (ie, not part of the sphinx build) and only live in the doc repo (I think there are some scripts in tools folder for talking to SF to get the download lists). @jenshnielsen Could you look into finding what those are and getting them tracked in the main repo?

[jenshnielsen]

Sure, I thought that versions.html was excluded on purpose since all versions of the documentation should ajax load the one from the webroot to have consistent links across versions.

Anyway the following files and dirs are in the root of matplotlib.github.com and not in the docs build

update_downloads.py
downloads.txt
downloads.tpl.html

which together total are used to generate:

downloads.html

And

CNAME
.nojekyll
sample_doc/ (from sample_doc repo)
versions.html
win32_static.tar.gz (delete?)
win32_static_vs.tar.gz (delete?)
xkcd/ (special doc build with xkcd)

[mdboom]

http://jenshnielsen.github.io/ looks good now. Thanks! Let us know what you need from us in terms of keys etc.

[jenshnielsen]

@mdboom The secrets to decrypt the keys are now stored in the Matplotlib Travis instance (I apparently had sufficient permissions to do that)

The only thing needed now is to create a repository within the matplotlib organisation to upload the devel documentation (mirroring jenshnielsen/mplddocs but should probably be called something better)

Then we need to add the following public key as a deploy key for this repository:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2Sh92ev1lL0u0LyiHFQ2gk1WKlN7gVk3r1Rs7cLGxYeVUJyHzOlHHwnwaozBAQGsU0mrutX4PDudJkoy5pWUdeXzeBdJwSyvjnvyOofMMjSWgxSN+dAEADvagnv+jrS7GVr6uNhTGGfzZ9JxuOL+XKpd0JFO3QF6WmgThk6v2qLFV6jDwkLw38C+DVZF/umjGlnyZs8RvjHkUvauLh5QUALQiJ6sKqL2ibBxkIb/z+w+xj46DGNfAHTcv8eTgORhq61BpJUuJO6npyioVqr8VPXQPsCFtfuwwJwM3HzbHs5qDbYnmkAM1X9HBUwTgYI4MpFKgoTuX9h9RMcYgHIV9 matplotlibDocsDeploy

When that repository is created I can modify .travis.yml to point to that repository.

As a last thing I will open a pr against the main documentation repository to add a link to the development documentation.

[tacaswell]

@jenshnielsen I have created https://github.com/matplotlib/devdocs for this purpose.

[jenshnielsen]

Thanks, I changed .travis.yml to point to that repository. Provided that the public key above has been added as a deploy key to that repository this should be ready. Unfortunately there is no real way to test this in full before merged to master since it only runs on the master branch. But it does work (with modifications) on my master branch

[tacaswell]

What's the worst that can happen...

[jenshnielsen]

Since everything is in a after_success block I don't think that it can fail the build

[jenshnielsen]

Still doesn't work i.e https://travis-ci.org/matplotlib/matplotlib/builds/46665810 fails to upload. The key is correctly decrypted so the matplotlib travis user has the correct secrets but it appears not to have the correct permissions to push to the repository.

The specific error is:

ERROR: Permission to matplotlib/devdocs.git denied to jenshnielsen/mplddocs.

Which is rather odd since I have no idea where jenshnielsen/mplddocs is coming from now. @tacaswell is the public key above added to deployment keys for the devdocs repository? If so I guess that I should try generating a new key which has not been associated with my repository.

[tacaswell]

Part of it was probably permission errors on the gh side. The developer group did not have access to the devdoc repo

[tacaswell]

@jenshnielsen Can you use that public key to push to devdocs from your local machine?

[jenshnielsen]

Nope looks like I get the same error. Did you add it as a deployment key? I.e. Go to the devdocs / settings of the repository / Deploy keys / and add it there?

I created this key specifically for this purpose and it should only have rights to push to that repository to limit the scope of what can be done automatically.

[mdboom]

@jenshnielsen: I went to the matplotlib/devdocs settings, went to deploy keys, and none were listed. So I tried to add the key above and it says "Key is already in use". So I'm a bit lost at this point. I added @jenshnielsen as an "owner" to the repository since maybe you can more efficiently debug this yourself as you know all of the moving pieces.

[jenshnielsen]

I guess I missed the fact that a deployment key can only be used for one repository, which I guess makes sense from a security point of view, I deleted the key from jenshnielsen/mplddocs and added it here. That seemed to be allowed. I have now restated the latest docs build of master on travis. Lets see if that works. (here https://travis-ci.org/matplotlib/matplotlib/jobs/46724819)

[jenshnielsen]

Looks like it worked up here at http://matplotlib.org/devdocs/ and https://github.com/matplotlib/devdocs/tree/gh-pages

Thanks for the help @mdboom and @tacaswell