Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

workflow level progress #214

Merged
merged 4 commits into from
Feb 21, 2017
Merged

workflow level progress #214

merged 4 commits into from
Feb 21, 2017

Conversation

lanchongyizu
Copy link
Member

Implement workflow level progress indicator after task level progress.
https://rackhd.atlassian.net/browse/RAC-3923

@RackHD/corecommitters @pengz1 @iceiilin

houndci-bot
houndci-bot reviewed Feb 10, 2017
View reviewed changes
lib/task-scheduler.js Outdated
error: error
});
});
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing semicolon.

houndci-bot
houndci-bot reviewed Feb 10, 2017
View reviewed changes
lib/task-scheduler.js Outdated
return progressData;
})
.then(function(progressData) {
return taskMessenger.publishProgressEvent(graph.instanceId, progressData)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing semicolon.

pengz1
pengz1 reviewed Feb 10, 2017
View reviewed changes
Copy link
Member

@pengz1 pengz1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 after update graph completion description.

lib/task-scheduler.js Outdated
assert.uuid(task.taskId, 'task.taskId');
})
.then(function() {
return store.getTaskById(task);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a database operation, if possible you should avoid introducing more database operation. Suggestion is to update task related attribute in taskMessenger.publishProgressEvent.

lib/task-scheduler.js Outdated
@@ -210,6 +211,7 @@ function taskSchedulerFactory(
.tap(self.handleStreamDebug.bind(self, 'Received evaluate task event'))
.map(self.updateTaskDependencies.bind(self))
.mergeLossy(self.concurrencyMaximums.updateTaskDependencies)
.tap(self.publishTaskFinishedProgressEvent.bind(self))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A question here: can we move taskStarted progress to task-scheduler as well? It is better to keep taskStarted and taskFinished events in one file.

lib/task-scheduler.js Outdated
progress: {
maximum: null,
value: 0,
description: 'Graph "' + graph.definition.friendlyName + '" finished'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be difference between graph finished, failed and cancelled on description.

lib/task-scheduler.js Outdated
return Rx.Observable.just(data.graphId)
.flatMap(store.getActiveGraphById)
.filter(function(graph) {return !_.isEmpty(graph);})
.tap(self.publishGraphFinishedProgressEvent.bind(self))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't put this event at the end of the Promise chain just before catch?

@JenkinsRHD
Copy link
Contributor

BUILD on-taskgraph #2058 : FAILURE

@lanchongyizu
Copy link
Member Author

test this please.

@JenkinsRHD
Copy link
Contributor

BUILD on-taskgraph #2067 : FAILURE

@lanchongyizu
Copy link
Member Author

test this please

@JenkinsRHD
Copy link
Contributor

BUILD on-taskgraph #2068 : FAILURE

@lanchongyizu
Copy link
Member Author

test this please

yyscamper
yyscamper reviewed Feb 13, 2017
View reviewed changes
lib/task-scheduler.js Outdated
'graph.definition.friendlyName');
var progressData = {
progress: {
maximum: null,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the graph finishes, but the progress.maximum is null! Isn't set the progress.value to progress.maximum?

yyscamper
yyscamper reviewed Feb 13, 2017
View reviewed changes
lib/task-runner.js Outdated
@@ -58,7 +58,7 @@ function taskRunnerFactory(
this.activeTasks = {};
this.lostBeatLimit = options.lostBeatLimit || 3;
this.domain = options.domain || Constants.Task.DefaultDomain;
this.maxTaskProgress = options.totalSteps || 100;
this.maxTaskProgress = 100;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove the options.totalSteps? Suppose the totalSteps is 4, during the task running, the progress is updated as 1/4, 2/4, ..., in your design, when task finishes, the progress will jump to 100/100, but use 4/4 here will be more reasonable.

yyscamper
yyscamper reviewed Feb 13, 2017
View reviewed changes
lib/task-scheduler.js
@@ -210,6 +211,9 @@ function taskSchedulerFactory(
.tap(self.handleStreamDebug.bind(self, 'Received evaluate task event'))
.map(self.updateTaskDependencies.bind(self))
.mergeLossy(self.concurrencyMaximums.updateTaskDependencies)
.tap(function(task){
return self.publishTaskFinishedProgressEvent(task);
})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A little suggestion, just like publishGraphProgress, you could change the tap to:

tap(self.publishTaskFinishedProgressEvent.bind(self))

yyscamper
yyscamper reviewed Feb 13, 2017
View reviewed changes
lib/task-scheduler.js Outdated
var progressData = {
progress: {
value: null,
maximum: null,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a task is finished, why the graph progress value is still kept in null state?

yyscamper
yyscamper reviewed Feb 13, 2017
View reviewed changes
spec/lib/task-scheduler-spec.js Outdated
.to.have.been.calledWith(graph.instanceId, progressData);
});
});

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a special feature for publish task/graph progress event is that, it will swallow the error if the publishing itself has some error, so that the event publishing will not impact the existing logic. So here need a test case to check the error is really being swallowed.

houndci-bot
houndci-bot reviewed Feb 16, 2017
View reviewed changes
spec/lib/task-scheduler-spec.js Outdated
name: 'test graph name',
tasks: {}
};
graph.tasks['taskId'] = {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

['taskId'] is better written in dot notation.

lib/task-scheduler.js Outdated
* @memberOf TaskScheduler
*/
TaskScheduler.prototype.publishTaskFinishedProgressEvent = function(task) {
var self = this;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'self' is defined but never used.

lib/task-runner.js Outdated
var progress = new Progress(graph);
progress.setGraphDescription(graphDescription);
progress.calculateGraphProgress();
progress.setTaskProgress(taskId, taskProgress)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing semicolon.

@lanchongyizu
Copy link
Member Author

Jenkins: depends on RackHD/on-tasks#404 RackHD/on-http#590 RackHD/on-core#256

@JenkinsRHD
Copy link
Contributor

BUILD on-taskgraph #2088 : ABORTED

BUILD unit-tests #11751 Error Logs ▼Test Name: Task Scheduler "before all" hook Error Details: No provider for Progress! (TaskGraph.TaskScheduler -> Progress) Stack Trace: Error: No provider for Progress! (TaskGraph.TaskScheduler -> Progress) at Injector.token.get (node_modules/on-core/node_modules/di/dist/cjs/injector.js:160:15) at node_modules/on-core/node_modules/di/dist/cjs/injector.js:176:19 at Array.map (native) at Injector.token.get (node_modules/on-core/node_modules/di/dist/cjs/injector.js:172:32) at Context.<anonymous> (spec/lib/task-scheduler-spec.js:78:41)

Test Name: Task Runner "before all" hook
Error Details: No provider for Progress! (TaskGraph.TaskRunner -> Progress)
Stack Trace: Error: No provider for Progress! (TaskGraph.TaskRunner -> Progress)
at Injector.token.get (node_modules/on-core/node_modules/di/dist/cjs/injector.js:160:15)
at node_modules/on-core/node_modules/di/dist/cjs/injector.js:176:19
at Array.map (native)
at Injector.token.get (node_modules/on-core/node_modules/di/dist/cjs/injector.js:172:32)
at Context.<anonymous> (spec/lib/task-runner-spec.js:71:38)

BUILD Install-Ubuntu-14-04 #411 Error Logs ▼Test Name: test_nodes_discovery Error Details: timeout waiting for task discovery -------------------- >> begin captured logging << -------------------- tests.api.v1_1.nodes_tests: INFO: Wait start time: 2017-02-16 02:13:10.614936 amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] amqp: DEBUG: Open OK! kombu: INFO: Starting AMQP worker -> graph.finished.*> amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] amqp: DEBUG: Open OK! kombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091// amqp: DEBUG: using channel_id: 1 amqp: DEBUG: Channel open modules.worker: ERROR: subtask timeout after 1200 seconds, (id=discovery), stopping.. kombu: INFO: Stopping AMQP worker -> graph.finished.*> modules.worker: INFO: stopping subtask for discovery amqp: DEBUG: Closed channel #1 --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run testMethod() File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest self._testFunc() File "/home/jenkins/workspace/Templates/Install-Ubuntu-14-04/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func compatability.capture_type_error(s_func) File "/home/jenkins/workspace/Templates/Install-Ubuntu-14-04/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error func() File "/home/jenkins/workspace/Templates/Install-Ubuntu-14-04/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func func(test_case.state.get_state()) File "/home/jenkins/workspace/Templates/Install-Ubuntu-14-04/RackHD/test/tests/api/v1_1/nodes_tests.py", line 123, in test_nodes_discovery message='timeout waiting for task {0}'.format(self.__task.id)) File "/home/jenkins/workspace/Templates/Install-Ubuntu-14-04/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/asserts.py", line 67, in assert_false raise ASSERTION_ERROR(message) "timeout waiting for task discovery\n-------------------- >> begin captured logging << --------------------\ntests.api.v1_1.nodes_tests: INFO: Wait start time: 2017-02-16 02:13:10.614936\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu: INFO: Starting AMQP worker -> graph.finished.*>\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091//\namqp: DEBUG: using channel_id: 1\namqp: DEBUG: Channel open\nmodules.worker: ERROR: subtask timeout after 1200 seconds, (id=discovery), stopping..\nkombu: INFO: Stopping AMQP worker -> graph.finished.*>\nmodules.worker: INFO: stopping subtask for discovery\namqp: DEBUG: Closed channel #1\n--------------------- >> end captured logging << ---------------------"
BUILD SmokeTest-CIT #642 Error Logs ▼Test Name: test_nodes_discovery Error Details: timeout waiting for task discovery -------------------- >> begin captured logging << -------------------- tests.api.v1_1.nodes_tests: INFO: Wait start time: 2017-02-16 02:14:47.959678 amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] amqp: DEBUG: Open OK! kombu: INFO: Starting AMQP worker -> graph.finished.*> amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] amqp: DEBUG: Open OK! kombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091// amqp: DEBUG: using channel_id: 1 amqp: DEBUG: Channel open modules.worker: ERROR: subtask timeout after 1200 seconds, (id=discovery), stopping.. kombu: INFO: Stopping AMQP worker -> graph.finished.*> modules.worker: INFO: stopping subtask for discovery amqp: DEBUG: Closed channel #1 --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run testMethod() File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest self._testFunc() File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func compatability.capture_type_error(s_func) File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error func() File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func func(test_case.state.get_state()) File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v1_1/nodes_tests.py", line 123, in test_nodes_discovery message='timeout waiting for task {0}'.format(self.__task.id)) File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/asserts.py", line 67, in assert_false raise ASSERTION_ERROR(message) "timeout waiting for task discovery\n-------------------- >> begin captured logging << --------------------\ntests.api.v1_1.nodes_tests: INFO: Wait start time: 2017-02-16 02:14:47.959678\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu: INFO: Starting AMQP worker -> graph.finished.*>\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091//\namqp: DEBUG: using channel_id: 1\namqp: DEBUG: Channel open\nmodules.worker: ERROR: subtask timeout after 1200 seconds, (id=discovery), stopping..\nkombu: INFO: Stopping AMQP worker -> graph.finished.*>\nmodules.worker: INFO: stopping subtask for discovery\namqp: DEBUG: Closed channel #1\n--------------------- >> end captured logging << ---------------------"

Test Name: teardown
Error Details: 'X-Auth-Token'
-------------------- >> begin captured logging << --------------------
modules.redfish_auth: INFO: auth already disabled.
--------------------- >> end captured logging << ---------------------
Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/redfish_1_0/session_service_tests.py", line 38, in teardown
Auth.disable()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/modules/redfish_auth.py", line 67, in disable
del config.api_client.default_headers['X-Auth-Token']
"'X-Auth-Token'\n-------------------- >> begin captured logging << --------------------\nmodules.redfish_auth: INFO: auth already disabled.\n--------------------- >> end captured logging << ---------------------"

Test Name: teardown
Error Details: 'X-Auth-Token'
-------------------- >> begin captured logging << --------------------
modules.redfish_auth: INFO: auth already disabled.
--------------------- >> end captured logging << ---------------------
Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/redfish_1_0/account_service_tests.py", line 38, in teardown
Auth.disable()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/modules/redfish_auth.py", line 67, in disable
del config.api_client.default_headers['X-Auth-Token']
"'X-Auth-Token'\n-------------------- >> begin captured logging << --------------------\nmodules.redfish_auth: INFO: auth already disabled.\n--------------------- >> end captured logging << ---------------------"

Test Name: test_node_workflows_post
Error Details: Failure running Graph.noop-example
-------------------- >> begin captured logging << --------------------
tests.api.v1_1.workflows_tests: INFO: Starting AMQP listener for node 58a5517c7d21322c08c2e7cb
amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
amqp: DEBUG: Open OK!
urllib3.connectionpool: WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /api/1.1/nodes/58a5517c7d21322c08c2e7cb/workflows/active
tests.api.v1_1.workflows_tests: INFO: Starting AMQP listener for node 58a5517c7d21322c08c2e7cc
amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
amqp: DEBUG: Open OK!
urllib3.connectionpool: WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /api/1.1/nodes/58a5517c7d21322c08c2e7cc/workflows/active
tests.api.v1_1.workflows_tests: INFO: Starting AMQP listener for node 58a551857d21322c08c2e7cd
amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
amqp: DEBUG: Open OK!
urllib3.connectionpool: WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /api/1.1/nodes/58a551857d21322c08c2e7cd/workflows/active
kombu: INFO: Starting AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>
kombu: INFO: Starting AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>
kombu: INFO: Starting AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>
amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
amqp: DEBUG: Open OK!
kombu.mixins: INFO: Connected to amqp://guest:@127.0.0.1:9091//
amqp: DEBUG: using channel_id: 1
amqp: DEBUG: Open OK!
kombu.mixins: INFO: Connected to amqp://guest:@127.0.0.1:9091//
amqp: DEBUG: using channel_id: 1
amqp: DEBUG: Channel open
amqp: DEBUG: Open OK!
kombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091//
amqp: DEBUG: using channel_id: 1
amqp: DEBUG: Channel open
amqp: DEBUG: Channel open
modules.worker: ERROR: subtask timeout after 300 seconds, (id=58a5517c7d21322c08c2e7cb), stopping..
kombu: INFO: Stopping AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>
modules.worker: INFO: stopping subtask for 58a5517c7d21322c08c2e7cb
amqp: DEBUG: Closed channel #1
modules.worker: ERROR: subtask timeout after 301 seconds, (id=58a5517c7d21322c08c2e7cc), stopping..
kombu: INFO: Stopping AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>
modules.worker: INFO: stopping subtask for 58a5517c7d21322c08c2e7cc
amqp: DEBUG: Closed channel #1
modules.worker: ERROR: subtask timeout after 302 seconds, (id=58a551857d21322c08c2e7cd), stopping..
kombu: INFO: Stopping AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>
modules.worker: INFO: stopping subtask for 58a551857d21322c08c2e7cd
amqp: DEBUG: Closed channel #1
tests.api.v1_1.workflows_tests: ERROR: Timeout for Graph.noop-example, node 58a5517c7d21322c08c2e7cb
tests.api.v1_1.workflows_tests: ERROR: Timeout for Graph.noop-example, node 58a5517c7d21322c08c2e7cc
tests.api.v1_1.workflows_tests: ERROR: Timeout for Graph.noop-example, node 58a551857d21322c08c2e7cd
--------------------- >> end captured logging << ---------------------
Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v1_1/workflows_tests.py", line 249, in test_node_workflows_post
self.post_workflows("Graph.noop-example")
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v1_1/workflows_tests.py", line 179, in post_workflows
self.run_workflow_tasks(self.__tasks, timeout_sec)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v1_1/workflows_tests.py", line 211, in run_workflow_tasks
fail('Failure running {0}'.format(self.__graph_name))
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/asserts.py", line 220, in fail
raise ASSERTION_ERROR(message)
'Failure running Graph.noop-example\n-------------------- >> begin captured logging << --------------------\ntests.api.v1_1.workflows_tests: INFO: Starting AMQP listener for node 58a5517c7d21322c08c2e7cb\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/\', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nurllib3.connectionpool: WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /api/1.1/nodes/58a5517c7d21322c08c2e7cb/workflows/active\ntests.api.v1_1.workflows_tests: INFO: Starting AMQP listener for node 58a5517c7d21322c08c2e7cc\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/\', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nurllib3.connectionpool: WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /api/1.1/nodes/58a5517c7d21322c08c2e7cc/workflows/active\ntests.api.v1_1.workflows_tests: INFO: Starting AMQP listener for node 58a551857d21322c08c2e7cd\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/\', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nurllib3.connectionpool: WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /api/1.1/nodes/58a551857d21322c08c2e7cd/workflows/active\nkombu: INFO: Starting AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>\nkombu: INFO: Starting AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>\nkombu: INFO: Starting AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/\', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/\', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/\', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu.mixins: INFO: Connected to amqp://guest:@127.0.0.1:9091//\namqp: DEBUG: using channel_id: 1\namqp: DEBUG: Open OK!\nkombu.mixins: INFO: Connected to amqp://guest:@127.0.0.1:9091//\namqp: DEBUG: using channel_id: 1\namqp: DEBUG: Channel open\namqp: DEBUG: Open OK!\nkombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091//\namqp: DEBUG: using channel_id: 1\namqp: DEBUG: Channel open\namqp: DEBUG: Channel open\nmodules.worker: ERROR: subtask timeout after 300 seconds, (id=58a5517c7d21322c08c2e7cb), stopping..\nkombu: INFO: Stopping AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>\nmodules.worker: INFO: stopping subtask for 58a5517c7d21322c08c2e7cb\namqp: DEBUG: Closed channel #1\nmodules.worker: ERROR: subtask timeout after 301 seconds, (id=58a5517c7d21322c08c2e7cc), stopping..\nkombu: INFO: Stopping AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>\nmodules.worker: INFO: stopping subtask for 58a5517c7d21322c08c2e7cc\namqp: DEBUG: Closed channel #1\nmodules.worker: ERROR: subtask timeout after 302 seconds, (id=58a551857d21322c08c2e7cd), stopping..\nkombu: INFO: Stopping AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>\nmodules.worker: INFO: stopping subtask for 58a551857d21322c08c2e7cd\namqp: DEBUG: Closed channel #1\ntests.api.v1_1.workflows_tests: ERROR: Timeout for Graph.noop-example, node 58a5517c7d21322c08c2e7cb\ntests.api.v1_1.workflows_tests: ERROR: Timeout for Graph.noop-example, node 58a5517c7d21322c08c2e7cc\ntests.api.v1_1.workflows_tests: ERROR: Timeout for Graph.noop-example, node 58a551857d21322c08c2e7cd\n--------------------- >> end captured logging << ---------------------'

Test Name: redfish_discovery_test
Error Details: Failure running Graph.Redfish.Discovery
-------------------- >> begin captured logging << --------------------
tests.api.v1_1.workflows_tests: INFO: Starting AMQP listener for Graph.Redfish.Discovery
amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
amqp: DEBUG: Open OK!
kombu: INFO: Starting AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>
amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
amqp: DEBUG: Open OK!
kombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091//
amqp: DEBUG: using channel_id: 1
amqp: DEBUG: Channel open
modules.worker: ERROR: subtask timeout after 300 seconds, (id=Graph.Redfish.Discovery), stopping..
kombu: INFO: Stopping AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>
modules.worker: INFO: stopping subtask for Graph.Redfish.Discovery
amqp: DEBUG: Closed channel #1
tests.api.v1_1.workflows_tests: ERROR: Timeout for Graph.Redfish.Discovery, node Graph.Redfish.Discovery
--------------------- >> end captured logging << ---------------------
Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v1_1/redfish_endpoint_tests.py", line 176, in redfish_discovery_test
self.__post_unbound_workflow('Graph.Redfish.Discovery', body)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v1_1/redfish_endpoint_tests.py", line 88, in __post_unbound_workflow
workflow.post_unbound_workflow(graph_name, data=body)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v1_1/workflows_tests.py", line 197, in post_unbound_workflow
self.run_workflow_tasks(self.__tasks, timeout_sec)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v1_1/workflows_tests.py", line 211, in run_workflow_tasks
fail('Failure running {0}'.format(self.__graph_name))
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/asserts.py", line 220, in fail
raise ASSERTION_ERROR(message)
"Failure running Graph.Redfish.Discovery\n-------------------- >> begin captured logging << --------------------\ntests.api.v1_1.workflows_tests: INFO: Starting AMQP listener for Graph.Redfish.Discovery\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu: INFO: Starting AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091//\namqp: DEBUG: using channel_id: 1\namqp: DEBUG: Channel open\nmodules.worker: ERROR: subtask timeout after 300 seconds, (id=Graph.Redfish.Discovery), stopping..\nkombu: INFO: Stopping AMQP worker <unbound Queue graph.finished -> <unbound Exchange on.events(topic)> -> graph.finished.>\nmodules.worker: INFO: stopping subtask for Graph.Redfish.Discovery\namqp: DEBUG: Closed channel #1\ntests.api.v1_1.workflows_tests: ERROR: Timeout for Graph.Redfish.Discovery, node Graph.Redfish.Discovery\n--------------------- >> end captured logging << ---------------------"

Test Name: test_node_catalogs
Error Details: Node catalog is empty!
Stack Trace: Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v2_0/nodes_tests.py", line 269, in test_node_catalogs
assert_not_equal(0, len(resp), message='Node catalog is empty!')
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/asserts.py", line 151, in assert_not_equal
raise ASSERTION_ERROR(message)
AssertionError: Node catalog is empty!

Test Name: test_catalogs
Error Details: Catalog dmi not found in node 58a5517c7d21322c08c2e7cb!
Stack Trace: Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v2_0/catalogs_tests.py", line 48, in test_catalogs
fail('Catalog {0} not found in node {1}!'.format(source,node.get('id')))
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/asserts.py", line 220, in fail
raise ASSERTION_ERROR(message)
AssertionError: Catalog dmi not found in node 58a5517c7d21322c08c2e7cb!

Test Name: pollers_data_get
Error Details: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Content-Length': '137', 'X-Powered-By': 'Express', 'Connection': 'keep-alive', 'ETag': 'W/"89-2Bp3+4FnxmzFtDUYU8gyGw"', 'Date': 'Thu, 16 Feb 2017 07:45:07 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json; charset=utf-8'})
HTTP response body: {"message":"Request Timed Out (on.task:methods.requestPollerCache:Object).","status":"400","UUID":"768ea81e-ff85-42d8-8756-dfca00fc812c"}

Stack Trace: Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v2_0/pollers_tests.py", line 151, in pollers_data_get
Api().pollers_data_get(poller['id'])
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/apis/api_api.py", line 4280, in pollers_data_get
callback=params.get('callback'))
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 322, in call_api
response_type, auth_settings, callback)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 149, in __call_api
post_params=post_params, body=body)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 342, in request
headers=headers)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/rest.py", line 184, in GET
query_params=query_params)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/rest.py", line 177, in request
raise ApiException(http_resp=r)
ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Content-Length': '137', 'X-Powered-By': 'Express', 'Connection': 'keep-alive', 'ETag': 'W/"89-2Bp3+4FnxmzFtDUYU8gyGw"', 'Date': 'Thu, 16 Feb 2017 07:45:07 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json; charset=utf-8'})
HTTP response body: {"message":"Request Timed Out (on.task:methods.requestPollerCache:Object).","status":"400","UUID":"768ea81e-ff85-42d8-8756-dfca00fc812c"}

Test Name: test_node_workflows_post
Error Details: status should be 404
-------------------- >> begin captured logging << --------------------
tests.api.v2_0.workflows_tests: INFO: starting amqp listener for node 58a5517c7d21322c08c2e7cb
amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
amqp: DEBUG: Open OK!
--------------------- >> end captured logging << ---------------------
Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v2_0/workflows_tests.py", line 276, in test_node_workflows_post
self.post_workflows("Graph.noop-example")
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v2_0/workflows_tests.py", line 210, in post_workflows
assert_equal(404,e.status, message='status should be 404')
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
raise ASSERTION_ERROR(message)
"status should be 404\n-------------------- >> begin captured logging << --------------------\ntests.api.v2_0.workflows_tests: INFO: starting amqp listener for node 58a5517c7d21322c08c2e7cb\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\n--------------------- >> end captured logging << ---------------------"

Test Name: test_tag_create
Error Details: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Length': '110', 'X-Powered-By': 'Express', 'Connection': 'keep-alive', 'ETag': 'W/"6e-0H5+y7uobnOc5yIFrBOTgA"', 'Date': 'Thu, 16 Feb 2017 07:45:16 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json; charset=utf-8'})
HTTP response body: {"message":"No Catalogs Found for Source (dmi).","status":"404","UUID":"92034ff2-219c-463c-b623-efea0403f320"}

Stack Trace: Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v2_0/tags_tests.py", line 45, in test_tag_create
tagsWithRules = self.__create_tag_rule(n.get('id'))
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v2_0/tags_tests.py", line 26, in __create_tag_rule
Api().nodes_get_catalog_source_by_id(identifier=id,source='dmi')
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/apis/api_api.py", line 1862, in nodes_get_catalog_source_by_id
callback=params.get('callback'))
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 322, in call_api
response_type, auth_settings, callback)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 149, in __call_api
post_params=post_params, body=body)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 342, in request
headers=headers)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/rest.py", line 184, in GET
query_params=query_params)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/rest.py", line 177, in request
raise ApiException(http_resp=r)
ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Length': '110', 'X-Powered-By': 'Express', 'Connection': 'keep-alive', 'ETag': 'W/"6e-0H5+y7uobnOc5yIFrBOTgA"', 'Date': 'Thu, 16 Feb 2017 07:45:16 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json; charset=utf-8'})
HTTP response body: {"message":"No Catalogs Found for Source (dmi).","status":"404","UUID":"92034ff2-219c-463c-b623-efea0403f320"}

Test Name: get_sku_nodes
Error Details: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Length': '110', 'X-Powered-By': 'Express', 'Connection': 'keep-alive', 'ETag': 'W/"6e-BdPsIVpt9uAH7/k7K52lMg"', 'Date': 'Thu, 16 Feb 2017 07:45:18 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json; charset=utf-8'})
HTTP response body: {"message":"No Catalogs Found for Source (dmi).","status":"404","UUID":"ee14155f-59ad-4d96-ae77-3b2da5a654a7"}

Stack Trace: Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v2_0/skupack_tests.py", line 148, in get_sku_nodes
Api().nodes_get_catalog_source_by_id(identifier=node_id,source='dmi')
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/apis/api_api.py", line 1862, in nodes_get_catalog_source_by_id
callback=params.get('callback'))
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 322, in call_api
response_type, auth_settings, callback)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 149, in __call_api
post_params=post_params, body=body)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 342, in request
headers=headers)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/rest.py", line 184, in GET
query_params=query_params)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/rest.py", line 177, in request
raise ApiException(http_resp=r)
ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Length': '110', 'X-Powered-By': 'Express', 'Connection': 'keep-alive', 'ETag': 'W/"6e-BdPsIVpt9uAH7/k7K52lMg"', 'Date': 'Thu, 16 Feb 2017 07:45:18 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json; charset=utf-8'})
HTTP response body: {"message":"No Catalogs Found for Source (dmi).","status":"404","UUID":"ee14155f-59ad-4d96-ae77-3b2da5a654a7"}

Test Name: post_skupacks
Error Details: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Length': '110', 'X-Powered-By': 'Express', 'Connection': 'keep-alive', 'ETag': 'W/"6e-jqDUZv2rf9DKhB2l8Z68og"', 'Date': 'Thu, 16 Feb 2017 07:45:18 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json; charset=utf-8'})
HTTP response body: {"message":"No Catalogs Found for Source (dmi).","status":"404","UUID":"edefff47-8ce7-481e-9372-45e7cfb9b5ff"}

Stack Trace: Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/tests/api/v2_0/sel_alert_poller_tests.py", line 54, in post_skupacks
Api().nodes_get_catalog_source_by_id(identifier=node_id, source='dmi')
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/apis/api_api.py", line 1862, in nodes_get_catalog_source_by_id
callback=params.get('callback'))
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 322, in call_api
response_type, auth_settings, callback)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 149, in __call_api
post_params=post_params, body=body)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/api_client.py", line 342, in request
headers=headers)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/rest.py", line 184, in GET
query_params=query_params)
File "/home/jenkins/workspace/Templates/SmokeTest-CIT/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/on_http_api2_0/rest.py", line 177, in request
raise ApiException(http_resp=r)
ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Length': '110', 'X-Powered-By': 'Express', 'Connection': 'keep-alive', 'ETag': 'W/"6e-jqDUZv2rf9DKhB2l8Z68og"', 'Date': 'Thu, 16 Feb 2017 07:45:18 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json; charset=utf-8'})
HTTP response body: {"message":"No Catalogs Found for Source (dmi).","status":"404","UUID":"edefff47-8ce7-481e-9372-45e7cfb9b5ff"}

BUILD Install-CentOs-6.5 #433 Error Logs ▼Test Name: test_nodes_discovery Error Details: timeout waiting for task discovery -------------------- >> begin captured logging << -------------------- tests.api.v1_1.nodes_tests: INFO: Wait start time: 2017-02-16 02:14:00.016771 amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] amqp: DEBUG: Open OK! kombu: INFO: Starting AMQP worker -> graph.finished.*> amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] amqp: DEBUG: Open OK! kombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091// amqp: DEBUG: using channel_id: 1 amqp: DEBUG: Channel open modules.worker: ERROR: subtask timeout after 1200 seconds, (id=discovery), stopping.. kombu: INFO: Stopping AMQP worker -> graph.finished.*> modules.worker: INFO: stopping subtask for discovery amqp: DEBUG: Closed channel #1 --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run testMethod() File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest self._testFunc() File "/home/jenkins/workspace/Templates/Install-CentOs-6.5/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func compatability.capture_type_error(s_func) File "/home/jenkins/workspace/Templates/Install-CentOs-6.5/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error func() File "/home/jenkins/workspace/Templates/Install-CentOs-6.5/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func func(test_case.state.get_state()) File "/home/jenkins/workspace/Templates/Install-CentOs-6.5/RackHD/test/tests/api/v1_1/nodes_tests.py", line 123, in test_nodes_discovery message='timeout waiting for task {0}'.format(self.__task.id)) File "/home/jenkins/workspace/Templates/Install-CentOs-6.5/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/asserts.py", line 67, in assert_false raise ASSERTION_ERROR(message) "timeout waiting for task discovery\n-------------------- >> begin captured logging << --------------------\ntests.api.v1_1.nodes_tests: INFO: Wait start time: 2017-02-16 02:14:00.016771\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu: INFO: Starting AMQP worker -> graph.finished.*>\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091//\namqp: DEBUG: using channel_id: 1\namqp: DEBUG: Channel open\nmodules.worker: ERROR: subtask timeout after 1200 seconds, (id=discovery), stopping..\nkombu: INFO: Stopping AMQP worker -> graph.finished.*>\nmodules.worker: INFO: stopping subtask for discovery\namqp: DEBUG: Closed channel #1\n--------------------- >> end captured logging << ---------------------"

@JenkinsRHD
Copy link
Contributor

BUILD on-taskgraph #2090 : ABORTED

BUILD unit-tests #11760 Error Logs ▼Test Name: Task Scheduler "before all" hook Error Details: No provider for Progress! (TaskGraph.TaskScheduler -> Progress) Stack Trace: Error: No provider for Progress! (TaskGraph.TaskScheduler -> Progress) at Injector.token.get (node_modules/on-core/node_modules/di/dist/cjs/injector.js:160:15) at node_modules/on-core/node_modules/di/dist/cjs/injector.js:176:19 at Array.map (native) at Injector.token.get (node_modules/on-core/node_modules/di/dist/cjs/injector.js:172:32) at Context.<anonymous> (spec/lib/task-scheduler-spec.js:78:41)

Test Name: Task Runner "before all" hook
Error Details: No provider for Progress! (TaskGraph.TaskRunner -> Progress)
Stack Trace: Error: No provider for Progress! (TaskGraph.TaskRunner -> Progress)
at Injector.token.get (node_modules/on-core/node_modules/di/dist/cjs/injector.js:160:15)
at node_modules/on-core/node_modules/di/dist/cjs/injector.js:176:19
at Array.map (native)
at Injector.token.get (node_modules/on-core/node_modules/di/dist/cjs/injector.js:172:32)
at Context.<anonymous> (spec/lib/task-runner-spec.js:71:38)

BUILD Install-CentOs-6.5 #435 Error Logs ▼Test Name: test_nodes_discovery Error Details: timeout waiting for task discovery -------------------- >> begin captured logging << -------------------- tests.api.v1_1.nodes_tests: INFO: Wait start time: 2017-02-16 02:30:07.048624 amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] amqp: DEBUG: Open OK! kombu: INFO: Starting AMQP worker -> graph.finished.*> amqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] amqp: DEBUG: Open OK! kombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091// amqp: DEBUG: using channel_id: 1 amqp: DEBUG: Channel open modules.worker: ERROR: subtask timeout after 1200 seconds, (id=discovery), stopping.. kombu: INFO: Stopping AMQP worker -> graph.finished.*> modules.worker: INFO: stopping subtask for discovery amqp: DEBUG: Closed channel #1 --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run testMethod() File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest self._testFunc() File "/home/jenkins/workspace/Templates/Install-CentOs-6.5/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func compatability.capture_type_error(s_func) File "/home/jenkins/workspace/Templates/Install-CentOs-6.5/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error func() File "/home/jenkins/workspace/Templates/Install-CentOs-6.5/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func func(test_case.state.get_state()) File "/home/jenkins/workspace/Templates/Install-CentOs-6.5/RackHD/test/tests/api/v1_1/nodes_tests.py", line 123, in test_nodes_discovery message='timeout waiting for task {0}'.format(self.__task.id)) File "/home/jenkins/workspace/Templates/Install-CentOs-6.5/RackHD/test/.venv/on-build-config/local/lib/python2.7/site-packages/proboscis/asserts.py", line 67, in assert_false raise ASSERTION_ERROR(message) "timeout waiting for task discovery\n-------------------- >> begin captured logging << --------------------\ntests.api.v1_1.nodes_tests: INFO: Wait start time: 2017-02-16 02:30:07.048624\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu: INFO: Starting AMQP worker -> graph.finished.*>\namqp: DEBUG: Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']\namqp: DEBUG: Open OK!\nkombu.mixins: INFO: Connected to amqp://guest:**@127.0.0.1:9091//\namqp: DEBUG: using channel_id: 1\namqp: DEBUG: Channel open\nmodules.worker: ERROR: subtask timeout after 1200 seconds, (id=discovery), stopping..\nkombu: INFO: Stopping AMQP worker -> graph.finished.*>\nmodules.worker: INFO: stopping subtask for discovery\namqp: DEBUG: Closed channel #1\n--------------------- >> end captured logging << ---------------------"

@changev
Copy link
Member

changev commented Feb 16, 2017

test this please

@JenkinsRHD
Copy link
Contributor

BUILD on-multi #108 : ABORTED

@lanchongyizu
implement workflow level progress
046845a 
https://rackhd.atlassian.net/browse/RAC-3923
@lanchongyizu
Before, it's not so easy to implement both showing status when workflow
e0d6381 
failed, canceled, timeout and calculating progress when failed.
So refactor progress.
@lanchongyizu
Rename Progress to GraphProgress, and change the interfaces.
9af868c
@lanchongyizu
test: add unit tests
6d11cab 
publishTaskStartedProgressEvent should swallow the Errors
TaskFinishedProgressEvent should swallow the Errors
GraphFinishedProgressEvent should swallow the Errors
@lanchongyizu
Copy link
Member Author

test this please.

@JenkinsRHD
Copy link
Contributor

BUILD on-multi #125 : ABORTED

@lanchongyizu
Copy link
Member Author

test this please.

@yyscamper yyscamper merged commit 04627b6 into RackHD:master Feb 21, 2017
sushilrai pushed a commit to sushilrai/on-taskgraph that referenced this pull request Dec 5, 2017
@tannoa2 @jlongever
Catalog platformA elements (RackHD#214)
0b4076a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@houndci-bot houndci-bot houndci-bot left review comments

@pengz1 pengz1 pengz1 left review comments

@yyscamper yyscamper yyscamper approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
6 participants
@lanchongyizu @JenkinsRHD @changev @yyscamper @houndci-bot @pengz1