New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

basic toolset for fedora packaging CI #578

Closed
wants to merge 9 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@praiskup
Member

praiskup commented Jun 1, 2016

Hi, I was about to discuss this as new PR. There is still at least one question -- how to
provide credentials for copr build. But, if the fedora's image will work on ubuntu machine,
we should be able to hook this into travis:

./packaging/rpm_ci 'copr-credentials-file'

What do you think?

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 1, 2016

@codecov-io

This comment has been minimized.

codecov-io commented Jun 1, 2016

Current coverage is 58.07%

Merging #578 into master will decrease coverage by 12.43%

  1. 2 files (not in diff) in ...a/org/postgresql/ssl were modified. more
    • Misses -3
  2. 6 files (not in diff) in .../org/postgresql/jdbc were modified. more
    • Hits -1
  3. 2 files (not in diff) in .../org/postgresql/core were modified. more
@@             master       #578   diff @@
==========================================
  Files           147        147           
  Lines         15505      15505           
  Methods           0          0           
  Messages          0          0           
  Branches       3064       3064           
==========================================
- Hits          10932       9004   -1928   
- Misses         4573       5284    +711   
- Partials          0       1217   +1217   

Powered by Codecov. Last updated by bbb0d35...51698d9

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 1, 2016

@praiskup , Here's how Travis encryption works: https://docs.travis-ci.com/user/encryption-keys/
We use that to encrypt "maven central deployment keys": https://github.com/pgjdbc/pgjdbc/blob/master/.travis.yml#L14

PS. For faster Travis turnaround, you can comment / cut non-relevant travis jobs (right from .travis.yml) while you are working on copr PR, so Travis would check just copr stuff.

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 1, 2016

Hm.. However, encrypted keys are disabled for PR-testing, so Travis encryption is not that suitable for copr-PR testing.

However, it could still be used for running copr against "master branch builds"

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 2, 2016

Thanks for those links.

However, it could still be used for running copr against "master branch builds"

That means for each "pushed" change? So it means we can immediately see which
commit caused the issue?

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 5, 2016

That means for each "pushed" change? So it means we can immediately see which
commit caused the issue?

Exactly.
I think it should be good enough, and it would still provide us with copr failures if any.

@praiskup praiskup force-pushed the praiskup:fedora-ci-brainstorm branch 3 times, most recently from 3f74702 to 4a083b4 Jun 6, 2016

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 7, 2016

Please have a look, in case of pull request, the build simply passes, otherwise Fedora
docker image is pulled from dockerhub and the build is submited to copr.

Copr is at the moment overloaded, but I've already reported this issue to copr upstream;
turns out it is better to not have this task hooked into pull requests check anyway :)

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 7, 2016

I've pushed that as a fedora-ci-brainstorm branch. Hopefully that would trigger copr stuff: https://travis-ci.org/pgjdbc/pgjdbc/builds/135822378

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 7, 2016

Uploading package postgresql-jdbc-9.5.git-1.20160607_094414.src.rpm
Build was added to pgjdbc-travis.
Created builds: 332242
Watching build(s): (this may be safely interrupted)
  09:44:15 Build 332242: importing
  09:44:47 Build 332242: pending
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
@praiskup

This comment has been minimized.

Member

praiskup commented Jun 7, 2016

Ah, yes - we need to have higher timeout probably, if that is not problem from your POV.

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 7, 2016

@praiskup do you know if copr-cli can be configured to output status once a minute? (so it won't get killed by Travis' 10m timeout)

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 7, 2016

Ah, that timeout-ed on the travis side :(, let me check.

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 7, 2016

I've added a dummy while true ... sleep loop.
Will see how it goes: https://travis-ci.org/pgjdbc/pgjdbc/builds/135831173

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 7, 2016

Nice, thanks.
Anyways, the build will fail like this, most probably -- is there some known issue to you in
the testsuite failure?
https://copr-be.cloud.fedoraproject.org/results/@pgjdbc/pgjdbc-travis/fedora-rawhide-x86_64/00332242-postgresql-jdbc/build.log.gz

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 7, 2016

@praiskup , it might be due to the fact that port 5432 is hard-coded into the test.
Can you please try using TestUtil.getPort() to check if that would help?

HostSpec hostSpec = new HostSpec(TestUtil.getServer(), 5432);

Re, "copr failed",

  1. Can you add some kind of cat the build log so we see failure reason right in Travis?
  2. Can you add print copr url action, so one can navigate from Travis log to the relevant Copr build?
@praiskup

This comment has been minimized.

Member

praiskup commented Jun 7, 2016

No problem, but that sounds like request for new PR?

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 7, 2016

Well, I've fixed 5432 -> TestUtil.getPort() in master. I'm not sure if that would solve copr issue though.

Here's updated travis-corp: https://travis-ci.org/pgjdbc/pgjdbc/builds/135842986

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 7, 2016

Huh, I'm not able to find the maven (plain text, not xml) logs to 'cat' them. Pointers?

@praiskup praiskup force-pushed the praiskup:fedora-ci-brainstorm branch 2 times, most recently from 9b15e85 to 7b5273c Jun 7, 2016

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 7, 2016

Can you predict https://copr-be.cloud.fedoraproject.org/results/@pgjdbc/pgjdbc-travis/fedora-rawhide-x86_64/00332242-postgresql-jdbc/build.log.gz kind of URL and just wget or curl it?

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 7, 2016

The unfortunate thing is that copr does not implement "live" logs, so it is not that much
useful link. We still need to wait until the build is finished.

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 7, 2016

But we can predict at least the link to concrete build? That requires some parsing of
copr-cli output.

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 7, 2016

so it is not that much useful link. We still need to wait until the build is finished.

Even if "live logs" are missing, it would be great if "full log" did appear at Travis, so it would be much easier to analyze the fail.

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 7, 2016

Makes sense to show it after the build finished .. agreed.

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 7, 2016

In the mean time, copr-cli managed to succeed somehow: https://travis-ci.org/pgjdbc/pgjdbc/builds/135842986

Does it keep the built package for inspection somewhere?
Are there steps to "publish the snapshot rpm"?

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 7, 2016

I'm not sure whether that ^^ makes sense?

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 11, 2016

@praiskup thanks, I've fired yet another travis job: https://travis-ci.org/pgjdbc/pgjdbc/builds/136914308

Just one more thing: is log cleanup at copr side automatic?

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 13, 2016

Yes, there is some policy older builds are removed (including logs). No need to remove
those manually.

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 13, 2016

In the last commit, I've just simplified the log output.

FTR, the timeout we faced before was some temporary issue in copr build system,
it should not happen regularly (the build was successful in the end).

@praiskup praiskup force-pushed the praiskup:fedora-ci-brainstorm branch from b58b222 to 7c77093 Jun 20, 2016

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 20, 2016

git-rebase against actual HEAD, RPMs generated in Copr now contain
git revision hash to make clear where the RPM comes from..

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 20, 2016

@praiskup, I've tried to run copr recently, and it fails: https://travis-ci.org/pgjdbc/pgjdbc/jobs/138634964

[ERROR]     Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:2.5.3 or one of its dependencies could not be resolved: Cannot access central (https://repo.maven.apache.org/maven2) in offline mode and the artifact org.apache.felix:maven-bundle-plugin:jar:2.5.3 has not been downloaded from it before. -> [Help 2]
[ERROR]     Unknown packaging: bundle @ line 11, column 14
@praiskup

This comment has been minimized.

Member

praiskup commented Jun 20, 2016

I've fixed this too (commit 1ca233a). I'm not sure whether pgjdbc has new requirements, or whether fedora's dependencies changed.

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 20, 2016

maven-bundle-plugin was there for quite some time: https://github.com/pgjdbc/pgjdbc-parent-poms/blob/master/pgjdbc-core-parent/pom.xml#L212

+docker run -e HOME=/git -u 1001 -ti --rm -v /home/travis/build/pgjdbc/pgjdbc/packaging/..:/git praiskup/copr-and-jdbc-ci copr-ci-git /git/packaging/rpm postgresql-jdbc 9.5.git
Unable to find image 'praiskup/copr-and-jdbc-ci:latest' locally
latest: Pulling from praiskup/copr-and-jdbc-ci
Status: Downloaded newer image for praiskup/copr-and-jdbc-ci:latest
+ export TZ=UTC
+ TZ=UTC
+ copr_fe_url=https://copr.fedorainfracloud.org/coprs/g/pgjdbc/pgjdbc-travis/build
+ copr_be_link=https://copr-be.cloud.fedoraproject.org/results/@pgjdbc/pgjdbc-travis/fedora-rawhide-x86_64/
+ status_file=copr_build_id
+ set -o pipefail
+ cd /git/packaging/rpm
++ git rev-parse --short=7 HEAD
+ git_rev=b58b222
++ date +%Y%m%d_%H%M%S
+ date_rev=20160620_064524
+ release=20160620_064524.gitb58b222
+ sed 's!^Release:.*$!Release: 1.20160620_064524.gitb58b222%{?dist}!' postgresql-jdbc.spec.tpl
sed: can't read postgresql-jdbc.spec.tpl: No such file or directory

It looks like "version parsing" has issues: https://travis-ci.org/pgjdbc/pgjdbc/builds/138829929

I thought a bit about it, and I think the most important part would be to automatically use the proper parent pom version. hack-parents.patch looks a bit like a hack.

On the other hand, we do not update it often, so I think we can live with that for a while.

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 20, 2016

That build failure is weird. Sed is unable to find 'postgrseql-jdbc.spec.tpl' file, which is in git,
however. The 'git hash' does not match the one in this PR -> have you merged the changes
completely?

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 20, 2016

Regarding the hack-parents.patch, I agree - it is a hack. That hack however allows us to
maintain only one package for Fedora.

%global upstreamrel git
%global source_path pgjdbc/src/main/java/org/postgresql
%global parent_ver 1.0.8
%global parent_poms_builddir ./pgjdbc-parent-poms-REL%parent_ver

This comment has been minimized.

@vlsi

vlsi Jun 20, 2016

Member

Is -REL%parent_ver required here?
It looks like if we use just ./pgjdbc-parent-poms, then hack-parents.patch could be altered to a simple sed replace <relativePath /> with <relativePath>... kind of command, so it would not be required to update the patch as parent version gets updated.

This comment has been minimized.

@praiskup

praiskup Jun 20, 2016

Member

Still we need to download the tarball (generated by github?) and that tarball contains
the release-version of pgjdbc-parent-poms. So we would have to move the
pgjdbc-parent-poms-RELXXX to pgjdbc-parent-poms first.

This comment has been minimized.

@praiskup

praiskup Jun 20, 2016

Member

Which is, however, better than edit the patch every time... I'll fix that.

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 20, 2016

Done. The macro %parent_ver still needs to be updated.

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 20, 2016

Here's my attempt to autoupdate %parent_ver: 51698d9

If no objections will integrate that.

@praiskup

This comment has been minimized.

Member

praiskup commented Jun 20, 2016

I'm fine with that ;) that encapsulated mvn command was missed in my know-how to
do it myself.

@vlsi vlsi closed this in 1eb4085 Jun 20, 2016

@vlsi

This comment has been minimized.

Member

vlsi commented Jun 20, 2016

@praiskup , thanks for moving this forward

@vlsi vlsi reopened this Jun 20, 2016

@vlsi vlsi closed this Jun 20, 2016

@vlsi vlsi added this to the 9.4.1209 milestone Jul 11, 2016

zemian pushed a commit to zemian/pgjdbc that referenced this pull request Oct 6, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment