New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merge doxygen docs to neovim.org repo #48

Closed
justinmk opened this Issue Jun 7, 2014 · 55 comments

Comments

Projects
None yet
4 participants
@justinmk
Copy link
Member

justinmk commented Jun 7, 2014

Assuming @stefan991 has interest in this, I think we should consider either linking to the doxygen docs at http://neovim.menkar.uberspace.de/devdoc or actually having his script push the latest build to this repo and hosting them on neovim.github.io.

It should be possible to modify the doxygen CSS to match the neovim branding, but it might be a pain. I know it's important to maintain a consistent look, but it's also important to provide a central channel for users in search of project materials. At the moment we're a bit scattered.

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jun 7, 2014

@justinmk Im not sure about pushing the builded docs to Github. If we do this this should get its own repo which can be hosted as a subdir on neovim.github.io (e.g. neovim.github.io/devdoc).

The easiest solution would be to point a subdomain (e.g. devdoc.neovim.org) to my server.

Updating the doxygen css is definitely on my todo list :)

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jun 10, 2014

neovim/neovim#824 looks fabulous! We need to link to it as soon as we figure out the layout adjustments for neovim.org.

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 10, 2014

So I did a bit of sleuthing and we could actually automate the entire thing and still host the dev-docs using GitHub Pages. Here are two interesting links:

And unless I'm reading it incorrectly, we should be able to generate the documentation on Travis and then we could push it to a gh-pages branch. Everything would be automated and yet not rely on any other servers than what we already have.

Does that look do-able? Would we want to do that in this case?

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jun 10, 2014

I'm not sure but I'm guessing @stefan991 is hesitant to push mega-uploads to the main site repo in case it causes some issue on neovim.org?

Note also we have clang analysis report now: neovim/neovim#167 (comment)

It's ~90 MB, we have 1 GB on github.io.

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 10, 2014

Ah, gotcha. Okay, that's totally fine in that regard. How do we want to add them to the site then?

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jun 11, 2014

I'm not sure but I'm guessing @stefan991 is hesitant to push mega-uploads to the main site repo in case it causes some issue on neovim.org?

At least it would clutter the history of the main site.
If we put the builded docs into its own repo we could delete the old history once 1GB gets filled up.

Does that look do-able? Would we want to do that in this case?

This should be do-able. The linked posts describe pretty well what we need to do.
We could create a github bot account like the one from rust (bors) which could do the pushes from travis to a new documentation repo. @jdavis could ask the community to come up with a good name :)

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 11, 2014

Haha, k. I'll ask.

Also, I think a separate bot account + a different repo is preferable like you mentioned. It keeps the history of the docs and website separate.

Edit: Actually, I'll ask if @justinmk thinks it would be a good idea to separate it into its own repo + a bot GitHub account.

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jun 11, 2014

If we put the builded docs into its own repo we could delete the old history once 1GB gets filled up.

Yeah, or we could just git commit --amend && git push -f every time.

I'll ask if @justinmk thinks it would be a good idea to separate it into its own repo + a bot GitHub account.

👍

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 11, 2014

That's a great idea.

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 17, 2014

@stefan991 Since Justin approves, should I ask on Twitter about a name for the Neovim GitHub bot? I imagine it could be useful in multiple ways beyond just this development documentation.

Also, have you made any progress on it?

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jun 17, 2014

Since Justin approves, should I ask on Twitter about a name for the Neovim GitHub bot?

👍

Also, have you made any progress on it?

No I haven't started yet, but it's the next item on the todo list :)

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 17, 2014

K cool.

I asked here, I have a feeling we'll have some interesting responses: https://twitter.com/Neovim/status/478947139473911808

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jun 17, 2014

@jdavis travis documentation generation is working:

commit on my branch: stefan991/neovim@1c7ac92
output repo: https://github.com/stefan991/documentation_test
hosted documentation on github: http://stefan991.github.io/documentation_test/index.html

This is based on neovim/neovim#824. After this PR gets merged I can adjust this to work with the bot/repo on the Neovim organization.

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 17, 2014

Awesome. That looks great.

Now to wait on neovim/neovim#824 like you said.

Did you see any names that you liked for the bot? I thought Marvim was the best and cleverest so far, haha.

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jun 18, 2014

@jdavis I like Marvim as well.

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 18, 2014

Great. Do you want to go with Marvim then? It looks like the name is available too: https://github.com/marvim

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jun 18, 2014

Great. Do you want to go with Marvim then?

Yea 👍
@tarruda @justinmk what do you mean?

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jun 18, 2014

Next steps (corrections welcome):

  • Merge neovim/neovim#824 after suggested changes
  • Create a repo for the doxygen output in the neovim organization
  • Create bot user
  • Give bot user commit rights to doxygen repo
  • Fork neovim main repo with bot
  • Bot will merge the fork daily or weekly (?). This triggers a travis build, which automatically pushes to the doxygen repo.

I think stefan991/neovim@1c7ac92 should be merged to neovim main repo, but I don't think we want to build it more than once a day or maybe once per week. The bot should have its own travis account (?) because we don't want to slow down the builds for the main repo.

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 19, 2014

That looks like a great plan, @justinmk. Let me know if you need any help with things, @stefan991.

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jun 24, 2014

Let me know if you need any help with things, @stefan991.

@jdavis: as I'm not a member of the Neovim organization it would be nice if you could do the following tasks:

  • Create a repo for the doxygen output in the neovim organization
  • Create bot user
  • Give bot user commit rights to doxygen repo

_

I think stefan991/neovim@1c7ac92 should be merged to neovim main repo, but I don't think we want to build it more than once a day or maybe once per week. The bot should have its own travis account (?) because we don't want to slow down the builds for the main repo.

@justinmk If the commit gets merged into the main repo, the task in the build matrix will show up and that causes a delay, too. I'm also not sure how we should add the travis task to the fork on the bot account without adding it to the main repo.

A solution could be to combine it with the coverty build matrix task. The coverty part of the task would only run on the coverty branch and doxygen would only be run on the master branch. Doxygen does only take about 30s on travis, so I don't think it would cause that much of a delay. (see https://travis-ci.org/stefan991/neovim/builds/27942277)

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 24, 2014

I only am part of the Neovim org because I am added to the team for this repo; I don't have permissions to create a repo or add a bot to a repo =[

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jun 24, 2014

I'll look into this. The license issue has priority for a few days though.

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 25, 2014

Yup, that's understandable for sure.

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 26, 2014

The bot has been created, @justinmk: Meet @marvim

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jun 26, 2014

@stefan991 @jdavis Added us all (including the bot) to the 'bot' team[1] which has commit rights to https://github.com/neovim/doc .

[1] Unless I'm missing something obvious, Github only allows teams to be assigned to repos--can't assign an individual user to a repo.

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 29, 2014

How's this coming, @stefan991?

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jun 29, 2014

@jdavis having final examinations this week, will continue here after them :)

@fwalch

This comment has been minimized.

Copy link
Member

fwalch commented Jun 29, 2014

@elmart in #780 (comment):

It might be nice to have some kind of statistics for each language (i.e. how many untranslated strings there are) that indicate where translation contributions are needed.

A new target could be added to the build invoking the script mentioned above. Or, if simple enough, it could be inlined within the Makefile instead of having a separate script.

We could run such a target periodically and export results to some public place. But, as I said before, I wouldn't include it on every build, cause most builds (development ones) don't touch localization and that would just add more burden.

Such a translation overview (and also the report of neovim/neovim#167, I guess) could also use the bot?

One thing that's not clear to me: how is the bot itself triggered (i.e. when does it merge upstream into its fork)?

@justinmk If the commit gets merged into the main repo, the task in the build matrix will show up and that causes a delay, too. I'm also not sure how we should add the travis task to the fork on the bot account without adding it to the main repo.

Couldn't $TRAVIS_REPO_SLUG help here? Since the builds get triggered from the bot's fork, Doxygen/other reports could only be built if this is equal to marvim/neovim.

@jdavis

This comment has been minimized.

Copy link
Contributor

jdavis commented Jun 29, 2014

@stefan991 Ahhh, by all means, take your time. I didn't realize school was still going on over there. Good luck =]

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jul 9, 2014

We could use something like Travis Cron to trigger periodic builds on a bot repository. This repository would not be a fork of the main repo, but basically an empty repo with only a .travis.yml

@fwalch thanks for the mention of Travis Cron.

Alternatively, the travis.yml could directly clone the main repo neovim/neovim, execute a "generate report" script (or just the Travis build script of neovim/neovim) and then push to generated-docs.

I think this is the better alternative. Not sure though if we should put this cron repo on the marvin account or the neovim organization. I would choose the later, so there is only one place to manage the permissions. (@jdavis , @justinmk )

@fwalch

This comment has been minimized.

Copy link
Member

fwalch commented Jul 10, 2014

We could use something like Travis Cron to trigger periodic builds on a bot repository. This repository would not be a fork of the main repo, but basically an empty repo with only a .travis.yml

@fwalch thanks for the mention of Travis Cron.

Just found Tron CI (code), but didn't try it yet.

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jul 10, 2014

Current status: I initialized the generated-docs repo and added a first manual doxygen build to it:

Next step: create a new repository on the neovim organization named 'marvim-travis' and enable TravisCI on this repo. @justinmk Would you mind doing this as I don't have the permissions to?

I have an updated version of the documentation publish script locally that I could push to this new repo.

After the first build succeeds, we need to enable this on Travis Cron.

@fwalch Tron CI seems to be an alternative, but as Travis Cron seems to be more widely used I would prefer it. Switching afterwards shouldn't be a problem, though.

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jul 10, 2014

Thanks @stefan991! I can't figure out how to view the generated-docs repo as a website.

https://help.github.com/articles/user-organization-and-project-pages seems to indicate that only neovim.github.io repo can be a gh-pages site. If I'm not mistaken this means we must push the generated docs to neovim.github.io repo or we must put the generated-docs repo under marvim's account as a marvim.github.io repo. Anyone know differently?

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jul 10, 2014

@justinmk here is the link: http://neovim.org/generated-docs/doxygen/
The master of neovim.github.io will be a github site. Other repos with a gh-pages branch will be accessible on a subfolder of the neovim.github.io site (or neovim.org as we have an DNS record that points to neovim.github.io)

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jul 10, 2014

Perfect!

Since we are phasing out the existing docs repo I think we should publish doxygen to that, in a "dev" directory so the URL looks like neovim.org/docs/dev. Then later on we can publish the user docs to docs/user/ #55.

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jul 10, 2014

Since we are phasing out the existing docs repo I think we should publish doxygen to that, in a "dev" directory so the URL looks like neovim.org/docs/dev. Then later on we can publish the user docs to docs/user/ #55.

Sounds good 👍

Next step: create a new repository on the neovim organization named 'marvim-travis' and enable TravisCI on this repo. @justinmk Would you mind doing this as I don't have the permissions to?

Did you miss this?

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jul 10, 2014

@stefan991 Can we put the repo under marvim's account or does it need to be under the neovim organization?

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jul 10, 2014

@justinmk It doesn't need to be on the Neovim organization. I would choose to host this on the Neovim organization so there is only one place to manage permissions for repositories related to Neovim.

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jul 10, 2014

@stefan991 Oh, now I see what you meant before, sorry. Created https://github.com/neovim/bot-ci

As for enabling travis, I will have to look at that later tonight, but you have access to that repo if you want to try it.

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jul 10, 2014

@justinmk enabling travis needs admin rights on a repo. I pushed the modified publish doc script to the repo. Should be ready for the first build :)

@fwalch

This comment has been minimized.

Copy link
Member

fwalch commented Jul 10, 2014

@stefan991 Branch in the doc script is gh-pages-test, is this intentional?

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jul 10, 2014

@fwalch yes, if something goes wrong on the first builds this doesn't "damage" the gh-pages branch. If the script is tested this will be changed back to gh-pages.

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jul 11, 2014

here is the link: http://neovim.org/generated-docs/doxygen/

@stefan991 I renamed the generated-docs repo to doc (singular--don't confuse it with the plural docs repo which is going to be deleted in the future). But http://neovim.org/doc/doxygen/ isn't working, so I guess there's some setup needed?

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jul 11, 2014

@justinmk Not really a setup, but it seems that the page only gets regenerated on a push to gh-pages.
I did this and renamed the doxygen folder to dev. link: http://neovim.org/doc/dev/

I updated the bot-ci script as well to reflect these changes.

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jul 11, 2014

Thanks!

@fwalch

This comment has been minimized.

Copy link
Member

fwalch commented Jul 15, 2014

@justinmk When you enable Travis for neovim/bot-ci, do not turn on building of pull requests, otherwise every submitted PR will directly change neovim/doc. I'd suggest leaving "build pushes" switched on to regenerate new/modified reports as quickly as possible.

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jul 15, 2014

@fwalch Thanks for noting that. I enabled travis for bot-ci.

@fwalch

This comment has been minimized.

Copy link
Member

fwalch commented Jul 15, 2014

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jul 15, 2014

Yay! Thank you @fwalch and @stefan991. I guess next step is setting up travis-cron?

@fwalch

This comment has been minimized.

Copy link
Member

fwalch commented Jul 15, 2014

Yup. You need to "apply" at https://traviscron.pythonanywhere.com/

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jul 15, 2014

@fwalch Done.

@stefan991

This comment has been minimized.

Copy link

stefan991 commented Jul 15, 2014

@justinmk I saw that you added a linkt to the gh-pages branch from the readme of the master branch. You can set the default branch wich gets displayed on the web in the project settings of the repository which would save one click.

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jul 31, 2014

FYI, traviscron doesn't seem to work: FiloSottile/travis-cron#4

I'll keep http://tron-ci.herokuapp.com/ in mind although if I get tired of pushing the "build" button every couple days.

Other than that, I think this issue is done. Nice work everyone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment