build refactor #80

ytjohn · 2016-03-29T04:53:01Z

Performing a build refactor similar to on-http#181. This brings both on-http and on-taskgraph onto the same pattern. It also ensures that people building their own packages can use the same process that travis uses.

yyscamper · 2016-03-29T06:10:38Z

@RackHD/corecommitters @RackHD/rackhd_dev @heckj please review

ytjohn · 2016-03-29T21:17:31Z

.dockerignore updated

…build-refactor

benbp · 2016-04-18T18:42:04Z

👍

anhou · 2016-04-19T02:13:17Z

👍

anhou · 2016-04-19T02:13:33Z

test this please

JenkinsRHD · 2016-04-19T02:38:19Z

*** BUILD #475 ***
Test Name: test_node_catalogs
Error Details: Node catalog is empty!
Stack Trace: Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/var/lib/jenkins/jobs/on-taskgraph/workspace/RackHD/test/tests/api/v1_1/nodes_tests.py", line 302, in test_node_catalogs
    assert_not_equal(0, len(resp), message='Node catalog is empty!')
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/asserts.py", line 151, in assert_not_equal
    raise ASSERTION_ERROR(message)
AssertionError: Node catalog is empty!


Test Name: test_node_catalogs
Error Details: Node catalog is empty!
Stack Trace: Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/var/lib/jenkins/jobs/on-taskgraph/workspace/RackHD/test/tests/api/v2_0/nodes_tests.py", line 266, in test_node_catalogs
    assert_not_equal(0, len(resp), message='Node catalog is empty!')
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/asserts.py", line 151, in assert_not_equal
    raise ASSERTION_ERROR(message)
AssertionError: Node catalog is empty!


Test Name: test_node_workflows_post
Error Details: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Content-Length': '128', 'X-Powered-By': 'Express', 'Connection': 'keep-alive', 'ETag': 'W/"80-7OjUGBy4o6RiaRnKGDICnw"', 'Date': 'Tue, 19 Apr 2016 02:33:43 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json; charset=utf-8'})
HTTP response body: {"message":"BadRequestError: required option user for task Task.Obm.Ipmi.CreateSettings in graph Graph.Obm.Ipmi.CreateSettings"}

-------------------- >> begin captured logging << --------------------
tests.api.v2_0.workflows_tests: INFO: starting amqp listener for node 571598fa2a9756ec086b5250
amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL.  See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
amqp: DEBUG: Open OK!
--------------------- >> end captured logging << ---------------------
Stack Trace:   File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/var/lib/jenkins/jobs/on-taskgraph/workspace/RackHD/test/tests/api/v2_0/workflows_tests.py", line 189, in test_node_workflows_post
    Api().nodes_post_workflow_by_id(id, name='Graph.noop-example', body={})
  File "/tmp/.venv/local/lib/python2.7/site-packages/on_http_api2_0/apis/api_api.py", line 2594, in nodes_post_workflow_by_id
    callback=params.get('callback'))
  File "/tmp/.venv/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 322, in call_api
    response_type, auth_settings, callback)
  File "/tmp/.venv/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 149, in __call_api
    post_params=post_params, body=body)
  File "/tmp/.venv/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 358, in request
    body=body)
  File "/tmp/.venv/local/lib/python2.7/site-packages/on_http_api2_0/rest.py", line 208, in POST
    body=body)
  File "/tmp/.venv/local/lib/python2.7/site-packages/on_http_api2_0/rest.py", line 177, in request
    raise ApiException(http_resp=r)
'(400)\nReason: Bad Request\nHTTP response headers: HTTPHeaderDict({\'Content-Length\': \'128\', \'X-Powered-By\': \'Express\', \'Connection\': \'keep-alive\', \'ETag\': \'W/"80-7OjUGBy4o6RiaRnKGDICnw"\', \'Date\': \'Tue, 19 Apr 2016 02:33:43 GMT\', \'Access-Control-Allow-Origin\': \'*\', \'Content-Type\': \'application/json; charset=utf-8\'})\nHTTP response body: {"message":"BadRequestError: required option user for task Task.Obm.Ipmi.CreateSettings in graph Graph.Obm.Ipmi.CreateSettings"}\n\n-------------------- >> begin captured logging << --------------------\ntests.api.v2_0.workflows_tests: INFO: starting amqp listener for node 571598fa2a9756ec086b5250\namqp: DEBUG: Start from server, version: 0.9, properties: {u\'information\': u\'Licensed under the MPL.  See http://www.rabbitmq.com/\', u\'product\': u\'RabbitMQ\', u\'copyright\': u\'Copyright (C) 2007-2013 GoPivotal, Inc.\', u\'capabilities\': {u\'exchange_exchange_bindings\': True, u\'connection.blocked\': True, u\'authentication_failure_close\': True, u\'basic.nack\': True, u\'consumer_priorities\': True, u\'consumer_cancel_notify\': True, u\'publisher_confirms\': True}, u\'platform\': u\'Erlang/OTP\', u\'version\': u\'3.2.4\'}, mechanisms: [u\'AMQPLAIN\', u\'PLAIN\'], locales: [u\'en_US\']\namqp: DEBUG: Open OK!\n--------------------- >> end captured logging << ---------------------'

Test Name: test_tag_create
Error Details: list index out of range
Stack Trace: Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/var/lib/jenkins/jobs/on-taskgraph/workspace/RackHD/test/tests/api/v2_0/tags_tests.py", line 35, in test_tag_create
    "rules":[{"equals": updated_catalog[0]["data"]["System Information"]["Manufacturer"],
IndexError: list index out of range

WangWinson · 2016-04-19T03:02:35Z

extra/make-cicd.sh

+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+cd $SCRIPT_DIR/..
+
+export DEBEMAIL="hwimo robots <hwimo@hwimo.lab.emc.com>"


Minor : Shall we use some public email address instead of internal here ?

Yes, that needs to be changed. And the FULLNAME as well, I think.

I made these to mimic HWIMO-BUILD. Hopefully you'll be able to have jenkins
run make-cicd instead of HWIMO-BUILD as well (and deprecate HWIMO-BUILD).

If you want a different email address, let me know and I can add it to the
PR. However, a couple similar PRs have been accepted with the lab email, so
we would probably want to do it across the collection of projects.

On Tue, Apr 19, 2016 at 8:13 AM, david d zuhn notifications@github.com
wrote:

In extra/make-cicd.sh
#80 (comment):

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+# This creates a package in a manner similar to
+# HWIMO-BUILD: datestring package version, apidoc
+# no coveralls. But it also utilizes build-package.bash
+# similar to how travis-ci does.
+
+# Ensure we're always in the right directory.
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+cd $SCRIPT_DIR/..
+
+export DEBEMAIL="hwimo robots hwimo@hwimo.lab.emc.com"

Yes, that needs to be changed.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/RackHD/on-taskgraph/pull/80/files/80418370126621d13da1d7652881234c1d75e132#r60219016

yyscamper · 2016-04-27T07:48:06Z

jenkins: test this please

davidzuhn · 2016-04-27T13:24:31Z

extra/make-deb.sh

+  rm -rf packagebuild
+fi
+
+git clone . packagebuild


It would be good, for the manifest based builds, to include the recent modification from HWIMO-BUILD to copy the local package.json file from . rather than from the git clone. The manifest build process goes through the package.json dependencies and changes the references from within RackHD git repos to the versions specified in the manifest. That info is kept in $PWD/package.json, but not checked in via git.

I'm going go through the open and recent build refactor PRs and change git clone to an rsync. We found that the builds from travis are doing some form of shallow clone, and the git clone . doesn't seem to work anymore.

ytjohn · 2016-04-27T14:11:11Z

Added fixes the travis-ci shallow clone related issues we saw in RackHD/on-dhcp-proxy#35 and RackHD/on-dhcp-proxy#34

JenkinsRHD · 2016-04-27T14:36:56Z

*** BUILD #514 ***
Test Name: test_node_workflows_del_active
Error Details: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Content-Length': '55', 'X-Powered-By': 'Express', 'Connection': 'keep-alive', 'ETag': 'W/"37-0X6VYPGMDPPXpP9BP8JfJg"', 'Date': 'Wed, 27 Apr 2016 14:32:27 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json; charset=utf-8'})
HTTP response body: {"message":"Failed to cancel 5720cd780d91afd12cda07a8"}

-------------------- >> begin captured logging << --------------------
urllib3.connectionpool: WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /api/1.1/nodes
urllib3.connectionpool: INFO: Starting new HTTP connection (11): localhost
--------------------- >> end captured logging << ---------------------
Stack Trace:   File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/var/lib/jenkins/jobs/on-taskgraph/workspace/RackHD/test/tests/api/v1_1/nodes_tests.py", line 376, in test_node_workflows_del_active
    raise e
'(500)\nReason: Internal Server Error\nHTTP response headers: HTTPHeaderDict({\'Content-Length\': \'55\', \'X-Powered-By\': \'Express\', \'Connection\': \'keep-alive\', \'ETag\': \'W/"37-0X6VYPGMDPPXpP9BP8JfJg"\', \'Date\': \'Wed, 27 Apr 2016 14:32:27 GMT\', \'Access-Control-Allow-Origin\': \'*\', \'Content-Type\': \'application/json; charset=utf-8\'})\nHTTP response body: {"message":"Failed to cancel 5720cd780d91afd12cda07a8"}\n\n-------------------- >> begin captured logging << --------------------\nurllib3.connectionpool: WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by \'ProtocolError(\'Connection aborted.\', BadStatusLine("\'\'",))\': /api/1.1/nodes\nurllib3.connectionpool: INFO: Starting new HTTP connection (11): localhost\n--------------------- >> end captured logging << ---------------------'

Test Name: teardown
Error Details: 'X-Auth-Token'
-------------------- >> begin captured logging << --------------------
modules.redfish_auth: INFO: auth already disabled.
--------------------- >> end captured logging << ---------------------
Stack Trace:   File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/var/lib/jenkins/jobs/on-taskgraph/workspace/RackHD/test/tests/api/redfish_1_0/session_service_tests.py", line 38, in teardown
    Auth.disable()
  File "/var/lib/jenkins/jobs/on-taskgraph/workspace/RackHD/test/modules/redfish_auth.py", line 67, in disable
    del config.api_client.default_headers['X-Auth-Token']
"'X-Auth-Token'\n-------------------- >> begin captured logging << --------------------\nmodules.redfish_auth: INFO: auth already disabled.\n--------------------- >> end captured logging << ---------------------"

Test Name: pollers_data_get
Error Details: Condition was True.
Stack Trace: Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/var/lib/jenkins/jobs/on-taskgraph/workspace/RackHD/test/tests/api/v2_0/pollers_tests.py", line 157, in pollers_data_get
    assert_false(poller.get('lastFinished', False))
  File "/tmp/.venv/local/lib/python2.7/site-packages/proboscis/asserts.py", line 67, in assert_false
    raise ASSERTION_ERROR(message)
AssertionError: Condition was True.

Modified lsscsi command parser to combine phrases correctly

build refactor

8041837

ytjohn force-pushed the build-refactor branch from 3e8a39a to 8041837 Compare March 29, 2016 21:17

Merge branch 'master' of https://github.com/RackHD/on-taskgraph into …

c8114e4

…build-refactor

WangWinson reviewed Apr 19, 2016
View reviewed changes

davidzuhn reviewed Apr 27, 2016
View reviewed changes

fix for travis shallow clone

c903be4

anhou merged commit 87ed18f into RackHD:master May 3, 2016

sushilrai pushed a commit to sushilrai/on-taskgraph that referenced this pull request Dec 5, 2017

Merge pull request RackHD#80 from iceiilin/lsscsi-parser

adf6bf1

Modified lsscsi command parser to combine phrases correctly

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build refactor #80

build refactor #80

ytjohn commented Mar 29, 2016

yyscamper commented Mar 29, 2016

ytjohn commented Mar 29, 2016

benbp commented Apr 18, 2016

anhou commented Apr 19, 2016

anhou commented Apr 19, 2016

JenkinsRHD commented Apr 19, 2016

WangWinson Apr 19, 2016

davidzuhn Apr 19, 2016 •

edited

Loading

ytjohn Apr 19, 2016

yyscamper commented Apr 27, 2016 •

edited by jlongever

Loading

davidzuhn Apr 27, 2016

ytjohn Apr 27, 2016

ytjohn commented Apr 27, 2016

JenkinsRHD commented Apr 27, 2016

build refactor #80

build refactor #80

Conversation

ytjohn commented Mar 29, 2016

yyscamper commented Mar 29, 2016

ytjohn commented Mar 29, 2016

benbp commented Apr 18, 2016

anhou commented Apr 19, 2016

anhou commented Apr 19, 2016

JenkinsRHD commented Apr 19, 2016

WangWinson Apr 19, 2016

Choose a reason for hiding this comment

davidzuhn Apr 19, 2016 • edited Loading

Choose a reason for hiding this comment

ytjohn Apr 19, 2016

Choose a reason for hiding this comment

yyscamper commented Apr 27, 2016 • edited by jlongever Loading

davidzuhn Apr 27, 2016

Choose a reason for hiding this comment

ytjohn Apr 27, 2016

Choose a reason for hiding this comment

ytjohn commented Apr 27, 2016

JenkinsRHD commented Apr 27, 2016

davidzuhn Apr 19, 2016 •

edited

Loading

yyscamper commented Apr 27, 2016 •

edited by jlongever

Loading