Build bdcs-cli and run tests inside Docker #21
Conversation
Makefile
Outdated
| [ -f "metadata.db" ] || curl https://s3.amazonaws.com/weldr/metadata.db > metadata.db | ||
| sudo docker ps | grep api || sudo docker run -d --rm --name api -p 4000:4000 -v `pwd`:/mddb --security-opt label=disable --network welder welder/bdcs-api-rs:latest | ||
|
|
||
| sudo docker build -t $(ORG_NAME)/bdcs-cli:latest -f $< --cache-from $(ORG_NAME)/bdcs-cli:latest . |
There was a problem hiding this comment.
I think we need to figure out how to make the --cache-from conditional. We should be able to run these on our local dev systems with the Fedora docker packages.
There was a problem hiding this comment.
Agree. How about I deal with this separately b/c the --cache-from option is used at several different places in the welder repos ?
There was a problem hiding this comment.
PS: I can try aliasing docker build to something which passes the --cache-from option if we are in CI but it's not going to be very pretty.
There was a problem hiding this comment.
Yeah, a separate set of patches is fine.
Dockerfile.build
Outdated
|
|
||
| # install the build dependencies first so we can cache this layer | ||
| COPY BDCSCli.cabal . | ||
| RUN cabal update && cabal install --dependencies-only --enable-tests --force-reinstall |
There was a problem hiding this comment.
Is the install really needed? It may mask problems with the ghc rpm packages.
I think we want to switch to depending on just the rpms so that we can be sure everything is packaged correctly.
There was a problem hiding this comment.
Hmmm, do we know if we have all necessary requirements as RPMs ? Should I just figure out their names and dnf install together with the other packages in this Dockerfile ?
There was a problem hiding this comment.
cabal: Encountered missing dependencies:
aeson -any,
aeson-pretty -any,
cond -any,
gi-ggit -any,
gi-gio -any,
gi-glib -any,
hspec -any,
htoml -any,
lens -any,
lens-aeson -any,
monad-loops -any,
semver -any,
split -any,
temporary -any,
unordered-containers -any,
wreq -any
This is after I install ghc-aeson and pretty much everything else. My command was for i in cabal install --dependencies-only --dry-run | grep - | grep -v "In order" |
grep -v basement | grep -v overloading | sed -r 's|-[0-9]+(.[0-9]+)*||'; do \ echo ghc-$i; done | xargs dnf -y install && dnf clean all
any ideas ?
There was a problem hiding this comment.
Here's what I came up with for my docker development image (I currently can't get my F26 desktop to work because some of these conflict with desktop packages I have installed) - https://github.com/bcl/docker-bdcs-hs
There was a problem hiding this comment.
OK, that worked. I needed the -devel counterpars of everything plus --enable-testing. Not sure if I need --enable-coverage with --dry-run though.
atodorov
force-pushed
the
images_testing
branch
from
19169fc
to
55be922
Compare
|
Just FYI in |
|
Update: looks like the API container has been started and reachable via http://api:4000/ but for some reason I don't receive a package list but nothing. Can't really figure out what is going on wrong. |
|
Running it locally I'm seeing several things: I'm hitting the OOM killer here when using the database from s3. If I use the one from Jenkins I don't have memory problems. |
|
OK, I think we do have a weird packaging/dependency problem. CC'ing @dashea The bdcs-deps repository provides RPM packages for ghc-memory and ghc-cryptonite but their names are suffixed with -weldr. As a result they are installed in the first step and then when trying to install all cabal dependencies we also get the RPMs from the Fedora repos and end up with: Here's what As you can see the required library versions are with different numbers than what we provide. @dashea what should we do about ^^^^ ? Aren't you guys seeing the same thing when trying to build locally ? |
|
For the record: the metadata.db files from both Jenkins and S3 appear to be the same currently. |
|
@atodorov We could do the same with cryptohash as with the cryptonite: build a -weldr version (which would use the -weldr versions of cryptonite and memory), and then rebuild packages against ghc-cryptohash-weldr. It looks like wreq is the only thing we use that requires it. The trickier part with cryptohash is that the version in Fedora is already the highest version available, so we need something other than version numbers during the build of wreq to ensure that the -weldr version of cryptohash is used. |
Sorry, I meant the metadata.db produced by the demo, not the centos7 one - http://jenkins.install.bos.redhat.com/view/Composer/job/welder-deployment/lastSuccessfulBuild/ |
|
@bcl these two are different but the OOM issue should be the one triggered when there are multiple versions of the same package. I checked Jenkins and unfortunately right as I submitted this PR it imported some newly updated RPMs into the DB which started causing the OOM. Will rebuild metadata.db and try again. On a side note: we should resolve the ghc-memory and ghc-cryptonite version mismatch but I think this doesn't affect this particular PR. |
|
After newly generated metadata.db The tests are now failing with:
I've checked the CentOS repo and the import log and also inspected metadata.db and sure there is the jboss-servlet-3.0-api package. The log says:
I'm in favor of merging this PR because its taken too long and then trying to figure out what's wrong with the jboss recipe. Thoughts ? |
atodorov
force-pushed
the
images_testing
branch
2 times, most recently
from
843dfb6
to
c5154e4
Compare
atodorov
force-pushed
the
images_testing
branch
from
503dee0
to
3749490
Compare
|
Changes Unknown when pulling 3749490 on atodorov:images_testing into ** on weldr:master**. |
No description provided.