Use npms.io to fetch data + use Github to store list through Travis-CI

[zckrs]

This PR groups the previously commit about use npms.io to fetch data (c928917) and adds a feature to store list of generators as JSON on Github to avoid dependency with a tiers server (9a08537).

Changes

yeoman-generator-list

Each build on Travis-CI for yeoman-generator-list will execute query on npms.io to store some informations about the generators.
If build succeed (and others conditions) we push a commit with the JSON list on a orphan branch of yeoman-generator-list (named cache-generators-list)

yeoman.github.io (PR)

We would like get the latest version of list (example).

To accomplished:

• we use Github API to get the latest SHA commit
• to avoid the rate limit by Github (60req/hour) we store in localstorage the SHA with a updatedDate
• we use RawGit with this SHA to serve file through MaxCDN and set proper headers

How to update the list

• Each succeed build on master branch will create and push a new fresh list
• We can contact Travis-CI to enable and configure a daily Cron Job
• In complementary we can trigger a new build by Travis-CI API curl -s -X POST -H "Travis-API-Version: 3" -H "Authorization: token secret_token_here" -H "Content-Type: application/json" -d '{"request":{"branch":"master"}}' https://api.travis-ci.org/repo/yeoman%2Fyeoman-generator-list/requests (rate limit: 10req/hour)

---

• Example of Travis build
• Example of orphan branch
• Final url of list

[eddiemonge]

I am not ok with using travis to host the list as its not a true cdn and doesn't give all the benefits cloud storage does. the rest looks ok. working on moving the "build" system somewhere else so it can deploy easier

[zckrs]

Travis don't host the list who use on site.
He just create and push the list.

At end Github host the list. See https://github.com/zckrs/yeoman-generator-list/commits/cache-generators-list

[eddiemonge]

Right, we have had problems hosting on github. its not as fast, reliable or good as gcdn

[zckrs]

Totally true and that why I use http://rawgit.com

RawGit serves raw files directly from GitHub with proper Content-Type headers.
No traffic limits or throttling. Files are served via MaxCDN's super fast global CDN.

[SBoudrias]

I think generating the list from Travis is good idea. We can talk about where we want to push the json list itself further.

@zckrs can you setup the new process to run on a test branch with hourly build from Travis. We could leave it there for a week to make sure it runs fine and does correctly update the list every hour. I think that'd give us more confidence in relying on this new flow to generate the json list.

[zckrs]

Done.

You can view commits here https://github.com/yeoman/yeoman-generator-list/commits/cache-generators-list
pushed by Travis here https://travis-ci.org/yeoman/yeoman-generator-list/builds

Do you want I run the gh-pages of my fork of yeoman.github.io?

[mischah]

Hej @zckrs,

your solution seems to work like a charm 💖

I just wonder if its possible to make less noise?

In the Gitter room:

An on my Github homepage:

Yours, Michael

[zckrs]

Right I use my personal access token to push commit.

I would like find a solution to avoid this noise.

Maybe we should use a specifc user (https://github.com/yeoman-bot) but external to organization.

Or create a new repo to store data

[diegohaz]

Is this accurate? One of my generators isn't in the list anymore (https://github.com/diegohaz/generator-rest) and another is shown with 0 stars even though the repository has 300+ (https://github.com/diegohaz/arc, the generator code is in another branch).

[zckrs]

@diegohaz

0 stars because npms.io not return a github object https://api.npms.io/v2/package/generator-arc

You can compare your generator-arc with https://github.com/kriasoft/react-static-boilerplate who have a Yeoman generator in another branch.

[zckrs]

FYI about accuracy npms.io

Whenever a new version is published it gets analyzed instantly, otherwise its each 15 days

[diegohaz]

I see. generator-react-static has the github object; generator-arc and generator-react-fullstack, also from kriasoft, hasn't. But I don't understand why.

I was thinking that's because generator-arc isn't fully released (it doesn't work properly yet) and npms.io might test it somehow. But this is also happening to react-fullstack...

[diegohaz]

Maybe @satazor can help with this?

[zckrs]

@diegohaz Better open issue on npms repo

[SBoudrias]

@zckrs a git user is only an email and a name. It doesn't need to have an account on github. For yeoman.io we set it up as such https://github.com/yeoman/yeoman.github.io/blob/source/.travis.yml#L33

[zckrs]

True but to have a personal access token we need a account registered currently is mine https://github.com/yeoman/yeoman-generator-list/pull/52/files#diff-354f30a63fb0907d4ad57269548329e3R11

[SBoudrias]

@zckrs yes, that's required, but it doesn't link commits to your account.

[zckrs]

Any commits are linked to my account but activities yes. This is the problem of @mischah

[mischah]

@SBoudrias How did you set up the token for this:
https://github.com/yeoman/generator/blob/master/.travis.yml#L19

[SBoudrias]

You need to create an oauth token https://github.com/settings/developers

[mischah]

Like I’ve described over here:
https://github.com/yeoman/generator/pull/971/files#diff-354f30a63fb0907d4ad57269548329e3R31

[SBoudrias]

Exactly like that

[satazor]

Regarding the issue about these not having a github object, I've answered here: npms-io/npms-analyzer#112

[SBoudrias]

@zckrs I think this setup looks good for now. Can you merge on master and set it up so it'll work from there?

[zckrs]

Done.

---

@mischah about your issue on the spam in Github activites. Can you open a new issue to describe problem and discuss on a possible solution ?