added .travis.yml

[junghans]

No description provided.

[mgorny]

Whaaaat?!

[junghans]

This is just convenience as we have to modify stuff in these files/directories below, we could also prefix all commands with sudo though.

[mgorny]

I know that but 777?! How about a+rwX instead? Won't make stuff executable.

[mgorny]

But in general, a great idea. Full repoman scan is something that should be done often before committing, but it's not because it's too heavy on resources. If we can outsource that to travis, I'd open pull requests to myself just to see the result :).

[junghans]

That is what I am doing for the science overlay ;-)

[mgorny]

Hmm, does travis re-do all that preparation (portage install etc.) for every task or just once for host doing the work? I wonder if the performance could be improved if we passed a few categories at a time. Maybe even used some globbing to reduce the list :).

[prometheanfire]

[a-de][dg-z]* type of thing :D

[mgorny]

Oh, and i should have some doubt about using -d. It will kinda add errors for which most of developers don't check, so users will see earlier developer mistakes…

[mgorny]

I'm going to fork this and experiment a bit.

[mrueg]

May I suggest using http://docs.travis-ci.com/user/pull-requests/ check for the changed directories in the pull request and just run repoman on these dirs? Then we should get quick results for pull-requests, for the main tree there's already autorepoman that does a great job.

[mgorny]

@junghans, I'm going back to your initial approach. Having too many categories in one job severely increases the risk of timeout :).

@mrueg, Kinda the point is to check the whole tree. AutoRepoMan is post-factum. Like, I want to remove package X. I push it here so that travis checks for me if I break something :P.

[mrueg]

Well travis has a maximum run time of 1h, autorepoman (which is parallelized) checks take around 135min iirc. I'm not sure if you can run repoman on the whole tree in under 1h.
Additionally travis times out after 10 minutes with no output, that's what https://github.com/mrueg/repoman-travis/blob/master/spinner.sh is used for.

[junghans]

@mgorny, I also liked the per-category approach better as it allows to find error easier.

@mrueg, I worked around the no-output issue by adding -v to the repoman call.

[mgorny]

Ok, I did some hacking and testing and here's my ultra-optimal version: https://github.com/mgorny/gentoo-portage-rsync-mirror/blob/travis/.travis.yml :P

Highlights:

• uses pypy to be a little faster.
• Runs 16 concurrent jobs, splitting categories from profiles/categories. This seems like a nice number with reasonable runtime per category and doesn't cause travis to get laggy because of number of tasks.
• Uses git portage so we don't have to update the version in the future.

[mgorny]

The test run is here; https://travis-ci.org/mgorny/gentoo-portage-rsync-mirror

[junghans]

Looks good, I took it!

[junghans]

I am not 100% if these two lines are actually necessary!

[junghans]

portage: 'portage' user or group missing.
         For the defaults, line 1 goes into passwd, and 2 into group.
         portage:x:250:250:portage:/var/tmp/portage:/bin/false
         portage::250:portage
*** WARNING ***  For security reasons, only system administrators should be
*** WARNING ***  allowed in the portage group.  Untrusted users or processes
*** WARNING ***  can potentially exploit the portage group for attacks such as
*** WARNING ***  local privilege escalation.

[mgorny]

So necessary, right? :P

[junghans]

Getting the warning multiple times would be a bit annoying!

[junghans]

Just FYI, packages which make travis fail:

sys-apps/kudzu: SRC_URI.mirror
x11-libs/guile-gtk: SRC_URI.mirror
media-sound/muine: SRC_URI.mirror

Not bad at all!

[mgorny]

SRC_URI.mirror should not be fatal. I'm going to fix this in a few minutes :).

But @junghans, do you consider this ready to merge? It's kinda a goal to avoid changing much in the git ;). In fact, we could commit .travis.yml to CVS instead… it is suitable for use with CVS checkout after all, isn't it?

[junghans]

I think so! I fixed media-sound/muine and x11-libs/guile-gtk. And sys-apps/kudzu will be removed in bug #537910.

[mrueg]

If you want to add something to the top-level in cvs, please discuss this on the mailing lists first.

[mgorny]

I know, @mrueg. The lovely mailing list which will love the idea. Their precious inodes!

[mgorny]

@swegener, can we get this cherry-picked into the git, please? :)

[junghans]

I did another rebase, now dev-vcs/git-cola has a bad dependency, because some old versions of dev-python/jsonpickle got dropped by @idella.

So I guess this testing is really useful ;-)

[junghans]

BTW, did somebody enable travis for this repo?

[mgorny]

Ok, that's going to be harder. I need someone from masters to give me admin access to the repo since I revoked it from all developers :D.

[mgorny]

Enabled now.

[mgorny]

Ok, we have a problem. Travis doesn't handle the load. I've just noticed there were 8 tests queued, and I cancelled them to get fresh output :). However, it looks like we would use a way to avoid testing every push :).

[junghans]

@mgorny, is guess you could increase the number of concurrent builds in the travis settings.

@henrikhodne, is there a way to only have one build at the time and then skip to the latest commit?

[mgorny]

That settings is already set to 0 = unlimited (I asked them). Looks like things are even worse today. I just canceled a lot of builds since it got stuck since ~12. Either travis is overloaded or they lowered out priority :).

[mgorny]

I disabled checking pushes and left travis for pull requests. We can re-enable them for a while from time to time to get tree-wide check updated. However, we need to look for a more permanent solution.

For a start, Patrick has suggested that pcheck is doing tree-wide dependency checks well. We may look into using it. Maybe tree-wide pcheck + repoman full in changed directories.