Deployement of wheels and website: Remaing tasks #405

jcfr · 2018-07-24T05:19:27Z

amueller · 2018-07-24T14:21:29Z

alright, I'll try to do that today. Thanks so much for your help so far!

Btw, this is somewhat unrelated, but maybe interesting for scikit-ci: I also want to auto-build the docs and upload to github-pages and that's a common task that I don't think is very automated yet.

jcfr · 2018-07-24T14:35:49Z

I also want to auto-build the docs and upload to github-pages and that's a common task that I don't think is very automated yet.

This will happen later tonight

Few options:

(1) add a simple script doc/deploy-gh-page.sh that would be called each time master is updated. Similar to this one. We will most likely add that script to scikit-ci-addons
or
(2) add a travis step

amueller · 2018-07-24T14:42:14Z

Thanks. That doesn't have the sphinx build, though, right? That's an important part, and with sphinx-gallery (which is getting more and more popular) the build might take quite a while if there's examples to run. Which makes caching more important.

amueller · 2018-07-24T14:43:23Z

Here's what scikit-learn does:
https://github.com/scikit-learn/scikit-learn/blob/master/build_tools/circle/build_doc.sh
https://github.com/scikit-learn/scikit-learn/blob/master/build_tools/circle/push_doc.sh

amueller · 2018-07-24T14:48:18Z

The solution you posted pushes the whole website every time, no matter the amount of changes, right? Or is the cloning part just not part of that comment? I'm not that familiar with worktree.

The scikit-learn website is 650mb. The wordcloud website is not as big, but if you want to put a solution into scikit-ci, I would favor a scalable one.

Doing just git clone on the repo will definitely not finish before CI timeout, --depth 1 is critical.

amueller · 2018-07-24T14:50:14Z

(and "of course" building the docs requires a whole different conda environment, and system package like texlive installed...)

jcfr · 2018-07-24T15:03:45Z

That doesn't have the sphinx build, though, right?

The process to build the website would be the following:

cd doc
pip install -r requirements-doc.txt
make html
And finally publish the content of `_build/html` directory to gh-page using the worktree approach referenced above.

I suggest we move forward with such an approach. To support this, would it be possible to add the user @scikit-build-bot as a collaborator to the project with write access. We would then create a personal token with that bot user and use that to publish the site.

scalable approach to documentation site building and publishing

scikit-ci, I would favor a scalable one

Make sense. To address this, we usually decouple the documentation building by having a "service" listening a webhook associated with the repo. Based on the payload, a circleci build on a different project is triggered and will take care of updating the different website and also posting an update on the original PR or commit.

For reference, and for a different project, here are the building block allowing to have doxygen documentation automatically deployed when a master is updated, a release is created or a PR is pushed:

From the contributor, the following check is posted on the PR:

And when master is updated, similar build is also triggered:

amueller · 2018-07-24T22:14:24Z

wow that's pretty cool!

amueller · 2018-07-24T22:15:07Z

can we get that on scikit-learn? The doc build there works but you need a browser plugin to view it directly.

amueller · 2018-07-24T22:20:31Z

alright, tried to set the password and push a tag

jcfr · 2018-07-24T22:28:19Z

alright, tried to set the password and push a tag

For the deployment step to work, the tag has to match the following regex:

word_cloud/.circleci/config.yml

Lines 168 to 170 in 88d6ab4

    
           filters: 
        
             tags: 
        
               only: /^[0-9]+(\.[0-9]+)*(\.post[0-9]+)?$/

I suggest to delete the test-tag, update the version and push something like 1.4.2. If everything looks on test.pypi.org, we can then remove the reference to --repository-url https://test.pypi.org/legacy/ and either delete and re-push the tag 1.4.2 or push 1.4.3.

Later, will also add How to make a release instruction

amueller · 2018-07-24T22:29:21Z

oh lol.
Do I need to create an account on test.pypi.org? My login from the main site doesn't seem to work and there doesn't seem to be a wordcloud package.

jcfr · 2018-07-24T22:48:57Z

Do I need to create an account on test.pypi.org?

Yeap. And adding me as a maintainer too.

Look like there is a package now 😄 https://test.pypi.org/project/wordcloud/

On windows it failed because two tags were associated with the same commit and appveyor was confused. I just deleted the test-tag

On travis, it looks the secure variable are incorrect ...

jcfr · 2018-07-24T22:51:22Z

On travis, it looks the secure variable are incorrect

I just added the env. variable directly in the travis setting and will remove them from the configuration file.

amueller · 2018-07-24T22:51:50Z

ah, sorry, didn't realize I had to change those as well...

jcfr · 2018-07-24T22:53:20Z

I used mine and they should have work. Fixing things now

See #405

amueller · 2018-07-24T22:54:08Z

you're the maintainer on test.pypi.org btw ;)

jcfr · 2018-07-24T23:04:34Z

Version bumped in aa54ef8

And has just been tag re-pushed. I expect the wheels to all be available on test.pypi.org .. if that looks good, I will remove reference to --repository-url https://test.pypi.org and re-push the tag.

Later tonight I should also have the website updated.

Then, we could do 1.5 instead ?

amueller · 2018-07-24T23:05:28Z

Sounds great. You're the best! Sent from phone. Please excuse spelling and brevity.

…

On Tue, Jul 24, 2018, 19:04 Jean-Christophe Fillion-Robin < ***@***.***> wrote: Version bumped in aa54ef8 <aa54ef8> And has just been tag re-pushed. I expect the wheels to all be available on test.pypi.org .. if that looks good, I will remove reference to --repository-url https://test.pypi.org and re-push the tag. Later tonight I should also have the website updated. Then, we could do 1.5 instead ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#405 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbcFr1Fjo8HqeZlmkBi0Yn8sZnVTez5ks5uJ6gDgaJpZM4VcGRf> .

jcfr · 2018-07-24T23:10:01Z

Travis wheel are now uploaded. Waiting for windows to show up.

That said, it look like there is a "random" error showing up on linux python 3.5 ... will see if I can reproduce locally

test_small_canvas 

wordcloud/wordcloud.py:431: in generate_from_frequencies
    / (sizes[0] + sizes[1]))
E   IndexError: list index out of range

During handling of the above exception, another exception occurred:
wordcloud/wordcloud.py:436: in generate_from_frequencies
    font_size = sizes[0]
E   IndexError: list index out of range

During handling of the above exception, another exception occurred:
test/test_wordcloud.py:324: in test_small_canvas
    WordCloud(max_words=50, width=20, height=20).generate(THIS)
wordcloud/wordcloud.py:605: in generate
    return self.generate_from_text(text)
wordcloud/wordcloud.py:587: in generate_from_text
    self.generate_from_frequencies(words)
wordcloud/wordcloud.py:439: in generate_from_frequencies
    "Couldn't find space to draw. Either the Canvas size"
E   ValueError: Couldn't find space to draw. Either the Canvas size is too small or too much of the image is masked out.

jcfr · 2018-07-24T23:44:23Z

@amueller upload on Appveyor failed because you haven't verified your PyPi account on the test server. Assuming you used the same password as the production server, this shouldn't prevent from moving forward with the release.

HTTPError: 400 Client Error: User '*****' does not have a verified primary email address. Please add a verified primary email before attempting to upload to PyPI. See https://test.pypi.org/help/#verified-email for more 
information.for more information. for url: https://test.pypi.org/legacy

amueller · 2018-07-24T23:52:26Z

Yes they are the same and just verified the email. Sent from phone. Please excuse spelling and brevity.

…

On Tue, Jul 24, 2018, 19:44 Jean-Christophe Fillion-Robin < ***@***.***> wrote: @amueller <https://github.com/amueller> upload on Appveyor failed because you haven't verified your PyPi account on the test server. Assuming you used the same password as the production server, this shouldn't prevent from moving forward with the release. HTTPError: 400 Client Error: User '*****' does not have a verified primary email address. Please add a verified primary email before attempting to upload to PyPI. See https://test.pypi.org/help/#verified-email for more information.for more information. for url: https://test.pypi.org/legacy — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#405 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbcFq6RI8fN6qjRGuz8Mv0CSJWdwXA1ks5uJ7FXgaJpZM4VcGRf> .

jcfr · 2018-07-25T08:24:22Z

Closing. All sub-tasks have been completed.

amueller · 2018-07-25T15:07:06Z

omg this is amazing! the website builds of master, not tags, right?

jcfr · 2018-07-25T17:11:03Z

the website builds of master, not tags, right?

yes

When is the website deployed ?

Each time a PR is done (or master updated), the website always build. This allows to know if something is broken during the generation process.

It is then only deployed when the branch is merged to master (or master updated)

word_cloud/.circleci/config.yml

Lines 95 to 111 in 107a5db

    
             deploy-master: 
        
               docker: 
        
                 - image: circleci/python:3.7.0-stretch 
        
               steps: 
        
                 - checkout 
        
                 - attach_workspace: 
        
                     at: ./ 
        
                 - run: 
        
                     name: Deploy master 
        
                     command: | 
        
                       echo "Deploy master (not implemented)" 
        
                 - run: 
        
                     name: Deploy website 
        
                     command: | 
        
                       source_sha=$(cat doc/_build/html/.buildinfo | grep sha | cut -d: -f2) 
        
                       repo_slug=$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME 
        
                       doc/deploy-website.sh $repo_slug $source_sha --html-dir doc/_build/html

That said we could easily decide to:

deploy only when a tag is pushed
or deploy to a "preview website"

(also agreed .. this may be overkill for this particular project, this is just to show case what is possible ..)

amueller · 2018-07-25T17:37:57Z

no this is great and exactly what I want :)

jcfr added the enhancement label Jul 24, 2018

jcfr self-assigned this Jul 24, 2018

jcfr added a commit that referenced this issue Jul 24, 2018

travis: Remove secure variable from configuration, add them to the UI

366e59b

See #405

jcfr closed this as completed Jul 25, 2018

jcfr modified the milestone: 1.5.0 Jul 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deployement of wheels and website: Remaing tasks #405

Deployement of wheels and website: Remaing tasks #405

jcfr commented Jul 24, 2018 •

edited

Loading

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018 •

edited

Loading

amueller commented Jul 24, 2018

amueller commented Jul 24, 2018

amueller commented Jul 24, 2018

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018 •

edited

Loading

amueller commented Jul 24, 2018

amueller commented Jul 24, 2018

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018

jcfr commented Jul 24, 2018

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018

amueller commented Jul 24, 2018 via email

jcfr commented Jul 24, 2018 •

edited

Loading

jcfr commented Jul 24, 2018

amueller commented Jul 24, 2018 via email

jcfr commented Jul 25, 2018

amueller commented Jul 25, 2018

jcfr commented Jul 25, 2018

amueller commented Jul 25, 2018

Deployement of wheels and website: Remaing tasks #405

Deployement of wheels and website: Remaing tasks #405

Comments

jcfr commented Jul 24, 2018 • edited Loading

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018 • edited Loading

amueller commented Jul 24, 2018

amueller commented Jul 24, 2018

amueller commented Jul 24, 2018

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018 • edited Loading

scalable approach to documentation site building and publishing

amueller commented Jul 24, 2018

amueller commented Jul 24, 2018

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018

jcfr commented Jul 24, 2018

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018

amueller commented Jul 24, 2018

jcfr commented Jul 24, 2018

amueller commented Jul 24, 2018 via email

jcfr commented Jul 24, 2018 • edited Loading

jcfr commented Jul 24, 2018

amueller commented Jul 24, 2018 via email

jcfr commented Jul 25, 2018

amueller commented Jul 25, 2018

jcfr commented Jul 25, 2018

When is the website deployed ?

amueller commented Jul 25, 2018

jcfr commented Jul 24, 2018 •

edited

Loading

jcfr commented Jul 24, 2018 •

edited

Loading

jcfr commented Jul 24, 2018 •

edited

Loading

jcfr commented Jul 24, 2018 •

edited

Loading