Create an intermediate Docker container for building wheels, also add requests to final image by rodrigc · Pull Request #4041 · buildbot/buildbot

rodrigc · 2018-04-11T17:45:02Z

Contributor Checklist:

I have updated the unit tests
I have created a file in the master/buildbot/newsfragment directory (and read the README.txt in that directory)
I have updated the appropriate documentation

rodrigc · 2018-04-11T17:49:13Z

@kallisti5 I have tried to fix your problem with requests in the Docker image in a slightly different way.

Can you try out my branch, build the Docker image, and report back?

Do this:

git clone -b docker https://github.com/rodrigc/buildbot
cd buildbot
make docker-buildbot-master

codecov · 2018-04-11T18:04:58Z

Codecov Report

Merging #4041 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4041   +/-   ##
=======================================
  Coverage   88.55%   88.55%           
=======================================
  Files         326      326           
  Lines       34330    34330           
=======================================
  Hits        30402    30402           
  Misses       3928     3928

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aeba4a2...2c40ded. Read the comment docs.

kallisti5 · 2018-04-11T18:36:26Z

$ make docker-buildbot-master

docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy -t buildbot-build:latest -f master/Dockerfile.build master
Sending build context to Docker daemon 11.86 MB
Step 1/6 : FROM alpine:3.7 AS buildbot-build
Error parsing reference: "alpine:3.7 AS buildbot-build" is not a valid repository/tag: invalid reference format
make: *** [Makefile:67: docker-buildbot-build] Error 1

kallisti5 · 2018-04-11T19:40:59Z

nevermind, upgraded from stock Fedora docker to Docker CE. Testing...

kallisti5 · 2018-04-11T20:37:54Z

Seems to work locally (at least I don't get missing imports now). I can't test much further on the production server since FROM * AS requires a later version of Docker than 1.13.1. Unable to test the actual oauth to 127.0.0.1 since everything rightfully freaks out.

2018-04-11 20:34:05+0000 [-] BuildMaster is running
2018-04-11 20:34:21+0000 [-] /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:858: urllib3.exceptions.InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2018-04-11 20:34:21+0000 [-] /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:858: urllib3.exceptions.InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2018-04-11 20:34:21+0000 [-] While rendering resource:
	Traceback (most recent call last):
	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
	    _inlineCallbacks(r, g, deferred)
	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks
	    deferred.errback()
	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 500, in errback
	    self._startRunCallbacks(fail)
	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
	    self._runCallbacks()
	--- <exception caught here> ---
	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/usr/lib/python2.7/site-packages/buildbot/www/resource.py", line 91, in failHttpError
	    f.trap(Error)
	  File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 359, in trap
	    self.raiseException()
	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/usr/lib/python2.7/site-packages/buildbot/www/resource.py", line 84, in failHttpRedirect
	    f.trap(Redirect)
	  File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 359, in trap
	    self.raiseException()
	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 63, in renderLogin
	    details = yield self.auth.verifyCode(code)
	  File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
	    result = inContext.theWork()
	  File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
	    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
	  File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 122, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 85, in callWithContext
	    return func(*args,**kw)
	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 173, in thd
	    return self.getUserInfoFromOAuthClient(session)
	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 276, in getUserInfoFromOAuthClient
	    return self.getUserInfoFromOAuthClient_v4(c)
	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 340, in getUserInfoFromOAuthClient_v4
	    teams.add('%s/%s' % (orgs_name_slug_mapping[org], node['node']['name']))
	exceptions.TypeError: 'NoneType' object has no attribute '__getitem__'

rodrigc · 2018-04-12T06:28:48Z

@kallisti5 can you:

build the Docker image locally
use docker save to save the image to a tar file
copy the tar file to your prod environment
use docker load to load the image from the tar file on your prod environment

I'd like to get additional feedback, to see if there are other fixes we can do to make this image more usable out of the box.

tardyp · 2018-04-12T13:21:10Z

We have to make this work on docker-hub, as this is our way to build official docker image.

Can you make sure of that?

rodrigc · 2018-04-12T17:23:44Z

Dockerhub can only accept a single Dockerfile as input.
I modified my patch, so that instead of having a separate Dockerfile.build file,
I have a single consolidated Dockerfile which uses a multi-stage build. During the multi-stage build, a temporary "buildbot-build" container is created, which is then used during the build of the final "buildbot" container.

I got it to work on Dockerhub:

https://hub.docker.com/r/rodrigc6/buildbot/builds/bf2tmscmxx5tgohcgm5gedx/

rodrigc · 2018-04-13T13:07:03Z

@kallisti5 See: https://hub.docker.com/r/rodrigc6/buildbot/ ,

If you have time, please do: docker pull rodrigc6/buildbot,
and give that a try and provide feedback.
If there are any problems with that, we can fix them.

kallisti5 · 2018-04-13T14:22:50Z

Thanks! I tried it locally and it doesn't seem to internally expose 80 anymore for the web interface?

Buildbot:

  buildbot:
#    image: buildbot/buildbot-master:v1.1.0
    image: rodrigc6/buildbot
    hostname: buildbot
    restart: on-failure
    volumes:
      - buildbot_config:/var/lib/buildbot:z
      - ./data/buildbot/master.cfg:/var/lib/buildbot/master.cfg:z
    ports:
      - '0.0.0.0:9989:9989'

Pootle (another service from our docker-compose)

  pootle:
    image: nielx/pootle:2.7
    restart: on-failure
    volumes:
      - pootle_data:/var/pootle:z
    depends_on:
      - postgres
      - redis
      - smtp

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                        PORTS                                      NAMES
577d2a6a1628        nielx/pootle:2.7    "/app/pootle-entry..."   42 hours ago        Up 42 hours                   80/tcp                                     infrastructure_pootle_1
f713099e64c5        rodrigc6/buildbot   "dumb-init /usr/sr..."   8 seconds ago       Up 5 seconds                  0.0.0.0:9989->9989/tcp                     infrastructure_buildbot_1

kallisti5 · 2018-04-13T14:24:45Z

aka:
https://github.com/haiku/infrastructure/blob/master/docker/pootle/Dockerfile#L34

kallisti5 · 2018-04-13T14:29:25Z

actually, it should always be exposed internally... maybe something else is going on. sec.

kallisti5 · 2018-04-13T14:37:57Z

Yeah. It seems to be working after I tore down all our containers and restarted them... however there also seems to be a bug in the GitHub auth plugin.

buildbot_1     | 2018-04-13 14:31:49+0000 [-] While rendering resource:
buildbot_1     | 	Traceback (most recent call last):
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
buildbot_1     | 	    _inlineCallbacks(r, g, deferred)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks
buildbot_1     | 	    deferred.errback()
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 500, in errback
buildbot_1     | 	    self._startRunCallbacks(fail)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
buildbot_1     | 	    self._runCallbacks()
buildbot_1     | 	--- <exception caught here> ---
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
buildbot_1     | 	    current.result = callback(current.result, *args, **kw)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/resource.py", line 91, in failHttpError
buildbot_1     | 	    f.trap(Error)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 359, in trap
buildbot_1     | 	    self.raiseException()
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
buildbot_1     | 	    current.result = callback(current.result, *args, **kw)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/resource.py", line 84, in failHttpRedirect
buildbot_1     | 	    f.trap(Redirect)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 359, in trap
buildbot_1     | 	    self.raiseException()
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
buildbot_1     | 	    result = result.throwExceptionIntoGenerator(g)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
buildbot_1     | 	    return g.throw(self.type, self.value, self.tb)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 63, in renderLogin
buildbot_1     | 	    details = yield self.auth.verifyCode(code)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
buildbot_1     | 	    result = inContext.theWork()
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
buildbot_1     | 	    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 122, in callWithContext
buildbot_1     | 	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 85, in callWithContext
buildbot_1     | 	    return func(*args,**kw)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 172, in thd
buildbot_1     | 	    session = self.createSessionFromToken(content)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 127, in createSessionFromToken
buildbot_1     | 	    s.params = {'access_token': token['access_token']}
buildbot_1     | 	exceptions.KeyError: 'access_token'
buildbot_1     |

I'm pretty sure something else is going on here. I don't see any more references to missing python modules however. (which means this change worked!) :-D

rodrigc · 2018-04-13T14:43:48Z

Great! Thanks for the feedback. If you have the time, please debug a little bit regarding Github auth and provide feedback. There is a working instance of Github auth at https://nine.buildbot.net , so it would be good to narrow down whether there is a problem with the Docker image, or your config/setup.

Thanks!!

kallisti5 · 2018-04-13T14:45:44Z

We have a pretty huge master.cfg that has been maintained since the early days of buildbot.
Here is our auth stuff:

https://github.com/haiku/infrastructure/blob/master/data/buildbot/master.cfg#L498

Maybe it is the Github API v4 stuff? (testing now)

kallisti5 · 2018-04-13T14:53:09Z

Ah. Nope. Didn't like my util.Secrets.

I paste in the auth string directly and get another error:

buildbot_1     | 2018-04-13 14:50:42+0000 [-] While rendering resource:
buildbot_1     | 	Traceback (most recent call last):
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
buildbot_1     | 	    _inlineCallbacks(r, g, deferred)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks
buildbot_1     | 	    deferred.errback()
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 500, in errback
buildbot_1     | 	    self._startRunCallbacks(fail)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
buildbot_1     | 	    self._runCallbacks()
buildbot_1     | 	--- <exception caught here> ---
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
buildbot_1     | 	    current.result = callback(current.result, *args, **kw)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/resource.py", line 91, in failHttpError
buildbot_1     | 	    f.trap(Error)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 359, in trap
buildbot_1     | 	    self.raiseException()
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
buildbot_1     | 	    current.result = callback(current.result, *args, **kw)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/resource.py", line 84, in failHttpRedirect
buildbot_1     | 	    f.trap(Redirect)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 359, in trap
buildbot_1     | 	    self.raiseException()
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
buildbot_1     | 	    result = result.throwExceptionIntoGenerator(g)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
buildbot_1     | 	    return g.throw(self.type, self.value, self.tb)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 63, in renderLogin
buildbot_1     | 	    details = yield self.auth.verifyCode(code)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
buildbot_1     | 	    result = inContext.theWork()
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
buildbot_1     | 	    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 122, in callWithContext
buildbot_1     | 	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 85, in callWithContext
buildbot_1     | 	    return func(*args,**kw)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 173, in thd
buildbot_1     | 	    return self.getUserInfoFromOAuthClient(session)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 276, in getUserInfoFromOAuthClient
buildbot_1     | 	    return self.getUserInfoFromOAuthClient_v4(c)
buildbot_1     | 	  File "/usr/lib/python2.7/site-packages/buildbot/www/oauth2.py", line 340, in getUserInfoFromOAuthClient_v4
buildbot_1     | 	    teams.add('%s/%s' % (orgs_name_slug_mapping[org], node['node']['name']))
buildbot_1     | 	exceptions.TypeError: 'NoneType' object has no attribute '__getitem__'
buildbot_1     |

Guessing that is related to API v4

kallisti5 · 2018-04-13T20:26:46Z

anyway I'm getting off topic.
This change works and solves the missing requests module.

The container provided works as expected:
https://build.haiku-os.org/buildbot

I'll look into the remaining github auth issues and open a bug report with details :-)

rodrigc · 2018-04-14T03:59:29Z

@kallisti5 Great! Can you just confirm that you solved the GitHub oauth problem independently of the Docker image that I provided you, and the only that I provided was the requests module in the image?

Glad to see Buildbot used by the Haiku project!

…ker image

…ot-build" container. We then copy the binary wheels into the final buildbot container. This allows us to avoid installing build tools like gcc in the final container.

rodrigc · 2018-04-14T21:07:04Z

@tardyp I confirmed this works on Dockerhub: https://hub.docker.com/r/rodrigc6/buildbot/
and @kallisti5 got this working for the Haiku project. Is this OK to merge?

kallisti5 · 2018-06-07T02:52:56Z

Just a small follow up. The 1.1.2 docker image's github plug is loading just fine as expected. :-)

rodrigc added 3 commits April 14, 2018 13:38

Add a requirements file for installing extra Python packages into doc…

159338b

…ker image

Use a multi-stage build so that we can build binary wheels in "buildb…

f88a1ae

…ot-build" container. We then copy the binary wheels into the final buildbot container. This allows us to avoid installing build tools like gcc in the final container.

Add newsfragment

2c40ded

tardyp merged commit 20a9635 into buildbot:master Apr 18, 2018

rodrigc deleted the docker branch April 18, 2018 08:31

lorenjanwilson mentioned this pull request Apr 23, 2018

buildbot-master docker container doesn't contain psycopg2 module #4057

Closed

rodrigc mentioned this pull request May 5, 2018

master/Dockerfile: Add py-requests to solve #4039 #4040

Closed

Conversation

rodrigc commented Apr 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contributor Checklist:

Uh oh!

rodrigc commented Apr 11, 2018

Uh oh!

codecov bot commented Apr 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kallisti5 commented Apr 11, 2018

Uh oh!

kallisti5 commented Apr 11, 2018

Uh oh!

kallisti5 commented Apr 11, 2018

Uh oh!

rodrigc commented Apr 12, 2018

Uh oh!

tardyp commented Apr 12, 2018

Uh oh!

rodrigc commented Apr 12, 2018

Uh oh!

rodrigc commented Apr 13, 2018

Uh oh!

kallisti5 commented Apr 13, 2018

Uh oh!

kallisti5 commented Apr 13, 2018

Uh oh!

kallisti5 commented Apr 13, 2018

Uh oh!

kallisti5 commented Apr 13, 2018

Uh oh!

rodrigc commented Apr 13, 2018

Uh oh!

kallisti5 commented Apr 13, 2018

Uh oh!

kallisti5 commented Apr 13, 2018

Uh oh!

kallisti5 commented Apr 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rodrigc commented Apr 14, 2018

Uh oh!

rodrigc commented Apr 14, 2018

Uh oh!

kallisti5 commented Jun 7, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rodrigc commented Apr 11, 2018 •

edited

Loading

codecov bot commented Apr 11, 2018 •

edited

Loading

kallisti5 commented Apr 13, 2018 •

edited

Loading