Bug 1798498: gather: enable bootstrap log gathering for baremetal #3617

stbenjam · 2020-05-19T02:03:48Z

This enables bootstrap log gathering for baremetal. We fetch the IP's
for the control plane hosts using ironic introspection data, and use
the provisioning IP for the bootstrap.

fixes #2009

stbenjam · 2020-05-19T02:04:17Z

/label platform/baremetal

stbenjam · 2020-05-19T10:50:20Z

/test e2e-metal-ipi

stbenjam · 2020-05-19T11:32:18Z

/test e2e-metal-ipi

stbenjam · 2020-05-19T12:55:02Z

/retitle Bug 1798498: gather: enable bootstrap log gathering for baremetal

openshift-ci-robot · 2020-05-19T12:55:22Z

@stbenjam: An error was encountered adding this pull request to the external tracker bugs for bug 1798498 on the Bugzilla server at https://bugzilla.redhat.com:

JSONRPC error 32000: There was an error reported for a GitHub REST call. URL: https://api.github.com/repos/openshift/installer/pulls/3617 Error: 403 Forbidden at /loader/0x563e184368f8/Bugzilla/Extension/ExternalBugs/Type/GitHub.pm line 111. at /loader/0x563e184368f8/Bugzilla/Extension/ExternalBugs/Type/GitHub.pm line 111. eval {...} called at /loader/0x563e184368f8/Bugzilla/Extension/ExternalBugs/Type/GitHub.pm line 98 Bugzilla::Extension::ExternalBugs::Type::GitHub::_do_rest_call('Bugzilla::Extension::ExternalBugs::Type::GitHub=HASH(0x563e26...', 'https://api.github.com/repos/openshift/installer/pulls/3617', 'GET') called at /loader/0x563e184368f8/Bugzilla/Extension/ExternalBugs/Type/GitHub.pm line 62 Bugzilla::Extension::ExternalBugs::Type::GitHub::get_data('Bugzilla::Extension::ExternalBugs::Type::GitHub=HASH(0x563e26...', 'Bugzilla::Extension::ExternalBugs::Bug=HASH(0x563e268c08b8)') called at /loader/0x563e184368f8/Bugzilla/Extension/ExternalBugs/Bug.pm line 302 eval {...} called at /loader/0x563e184368f8/Bugzilla/Extension/ExternalBugs/Bug.pm line 302 Bugzilla::Extension::ExternalBugs::Bug::update_ext_info('Bugzilla::Extension::ExternalBugs::Bug=HASH(0x563e268c08b8)', 1) called at /loader/0x563e184368f8/Bugzilla/Extension/ExternalBugs/Bug.pm line 125 Bugzilla::Extension::ExternalBugs::Bug::create('Bugzilla::Extension::ExternalBugs::Bug', 'HASH(0x563e26873408)') called at /var/www/html/bugzilla/extensions/ExternalBugs/Extension.pm line 877 Bugzilla::Extension::ExternalBugs::bug_start_of_update('Bugzilla::Extension::ExternalBugs=HASH(0x563e264e1678)', 'HASH(0x563e268427f0)') called at /var/www/html/bugzilla/Bugzilla/Hook.pm line 21 Bugzilla::Hook::process('bug_start_of_update', 'HASH(0x563e268427f0)') called at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1170 Bugzilla::Bug::update('Bugzilla::Bug=HASH(0x563e26841478)') called at /loader/0x563e184368f8/Bugzilla/Extension/ExternalBugs/WebService.pm line 88 Bugzilla::Extension::ExternalBugs::WebService::add_external_bug('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x563e265ab7f0)') called at (eval 1965) line 1 eval ' $procedure->{code}->($self, @params) ;' called at /usr/share/perl5/vendor_perl/JSON/RPC/Legacy/Server.pm line 220 JSON::RPC::Legacy::Server::_handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x563e265d27f8)') called at /var/www/html/bugzilla/Bugzilla/WebService/Server/JSONRPC.pm line 295 Bugzilla::WebService::Server::JSONRPC::_handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x563e265d27f8)') called at /usr/share/perl5/vendor_perl/JSON/RPC/Legacy/Server.pm line 126 JSON::RPC::Legacy::Server::handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...') called at /var/www/html/bugzilla/Bugzilla/WebService/Server/JSONRPC.pm line 70 Bugzilla::WebService::Server::JSONRPC::handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...') called at /var/www/html/bugzilla/jsonrpc.cgi line 31 ModPerl::ROOT::Bugzilla::ModPerl::ResponseHandler::var_www_html_bugzilla_jsonrpc_2ecgi::handler('Apache2::RequestRec=SCALAR(0x563e25f8bda0)') called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 207 eval {...} called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 207 ModPerl::RegistryCooker::run('Bugzilla::ModPerl::ResponseHandler=HASH(0x563e264dafa0)') called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 173 ModPerl::RegistryCooker::default_handler('Bugzilla::ModPerl::ResponseHandler=HASH(0x563e264dafa0)') called at /usr/lib64/perl5/vendor_perl/ModPerl/Registry.pm line 32 ModPerl::Registry::handler('Bugzilla::ModPerl::ResponseHandler', 'Apache2::RequestRec=SCALAR(0x563e25f8bda0)') called at /var/www/html/bugzilla/mod_perl.pl line 139 Bugzilla::ModPerl::ResponseHandler::handler('Bugzilla::ModPerl::ResponseHandler', 'Apache2::RequestRec=SCALAR(0x563e25f8bda0)') called at (eval 1965) line 0 eval {...} called at (eval 1965) line 0
Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

In response to this:

Bug 1798498: gather: enable bootstrap log gathering for baremetal

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

stbenjam · 2020-05-19T12:55:49Z

/bugzilla refresh

openshift-ci-robot · 2020-05-19T12:55:53Z

@stbenjam: This pull request references Bugzilla bug 1798498, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.5.0) matches configured target release for branch (4.5.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

stbenjam · 2020-05-19T13:05:59Z

/test images

stbenjam · 2020-05-19T13:18:48Z

/test e2e-aws

stbenjam · 2020-05-19T15:40:41Z

/test e2e-metal-ipi

stbenjam · 2020-05-19T17:27:30Z

/test e2e-metal-ipi

imain · 2020-05-19T21:54:16Z

It looks good! It would be nice if we could get info on the ironic containers. I see you get logs for all containers but ironic stuff isn't in there. Could be a later patch if deemed reasonable. Otherwise lgtm.

stbenjam · 2020-05-20T01:03:06Z

It looks good! It would be nice if we could get info on the ironic containers. I see you get logs for all containers but ironic stuff isn't in there. Could be a later patch if deemed reasonable. Otherwise lgtm.

The container logs should be in bootstrap/pods/, but good call about services, I'll add ironic and the master update service to the list tomorrow.

This enables bootstrap log gathering for baremetal. We fetch the IP's for the control plane hosts using ironic introspection data, and pick the best one. We prefer to find an IP in a machine network, but otherwise we'll use the first valid IP found. For the bootstrap, the only IP address we can know is the provisioning IP for the bootstrap.

This updates terraform-provider-ironic to 0.2.1. This version of the terraform provider has several fixes that we need related to introspection. The installer will now use introspection data to collect IP's for the control plane hosts for bootstrap log gathering. After updating the version in go.mod, I ran `go mod vendor` and `go mod tidy`.

stbenjam · 2020-05-20T14:06:48Z

/test e2e-metal-ipi

abhinavdahiya · 2020-05-20T16:13:12Z

/approve

openshift-ci-robot · 2020-05-20T16:14:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

imain · 2020-05-20T16:57:49Z

It looks good! It would be nice if we could get info on the ironic containers. I see you get logs for all containers but ironic stuff isn't in there. Could be a later patch if deemed reasonable. Otherwise lgtm.

The container logs should be in bootstrap/pods/, but good call about services, I'll add ironic and the master update service to the list tomorrow.

Actually the more I think about it, this could be pretty important. If there are any problems with ironic (including configuration) they will likely surface during bootstrap since that's its first use.

openshift-bot · 2020-05-26T14:13:16Z

/retest

Please review the full test history for this PR and help us cut down flakes.

stbenjam · 2020-05-26T14:13:22Z

/test images

openshift-bot · 2020-05-26T14:26:13Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T15:18:11Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T15:31:08Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T15:44:08Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T15:57:10Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T16:40:13Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T17:02:11Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T18:34:54Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T18:46:16Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T19:25:12Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T20:04:18Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T20:17:09Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T20:56:23Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T22:14:08Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-26T23:19:09Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-27T00:24:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-27T01:29:47Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-27T02:23:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-27T02:37:10Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-05-27T03:31:53Z

@stbenjam: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-libvirt	`4550c25`	link	`/test e2e-libvirt`
ci/prow/e2e-metal-ipi	`4550c25`	link	`/test e2e-metal-ipi`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2020-05-27T03:39:05Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-05-27T03:49:08Z

@stbenjam: All pull requests linked via external trackers have merged: openshift/installer#3617. Bugzilla bug 1798498 has been moved to the MODIFIED state.

In response to this:

Bug 1798498: gather: enable bootstrap log gathering for baremetal

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This was implemented in openshift#3617

install gather was implemented in openshift#3617, and the provisioning net is fully configurable now.

openshift-ci-robot added the platform/baremetal IPI bare metal hosts platform label May 19, 2020

openshift-ci-robot requested review from celebdor and patrickdillon May 19, 2020 02:04

stbenjam force-pushed the gather branch from 3bf44cc to 4b95ac7 Compare May 19, 2020 02:05

openshift-ci-robot changed the title ~~gather: enable bootstrap log gathering for baremetal~~ Bug 1798498: gather: enable bootstrap log gathering for baremetal May 19, 2020

openshift-ci-robot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels May 19, 2020

stbenjam force-pushed the gather branch 2 times, most recently from 1c49fca to fab9ddd Compare May 19, 2020 14:04

stbenjam added 2 commits May 20, 2020 07:41

stbenjam force-pushed the gather branch from fab9ddd to 4550c25 Compare May 20, 2020 11:42

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 20, 2020

stbenjam mentioned this pull request May 26, 2020

baremetal: copy install-gather as well openshift/release#9305

Merged

openshift-merge-robot merged commit f568c9f into openshift:master May 27, 2020

cybertron added a commit to cybertron/installer that referenced this pull request Jul 8, 2020

Remove todo for install gather support

3ffd8e1

This was implemented in openshift#3617

cybertron added a commit to cybertron/installer that referenced this pull request Jul 8, 2020

Remove metal todo for install gather support

7821f01

This was implemented in openshift#3617

cybertron mentioned this pull request Jul 8, 2020

Remove metal todo for install gather support #3866

Merged

cybertron added a commit to cybertron/installer that referenced this pull request Jul 9, 2020

Remove metal todos for completed features

56aba67

install gather was implemented in openshift#3617, and the provisioning net is fully configurable now.

Bug 1798498: gather: enable bootstrap log gathering for baremetal #3617

Bug 1798498: gather: enable bootstrap log gathering for baremetal #3617

Conversation

stbenjam commented May 19, 2020

stbenjam commented May 19, 2020

stbenjam commented May 19, 2020

stbenjam commented May 19, 2020

stbenjam commented May 19, 2020

openshift-ci-robot commented May 19, 2020

stbenjam commented May 19, 2020

openshift-ci-robot commented May 19, 2020

stbenjam commented May 19, 2020

stbenjam commented May 19, 2020

stbenjam commented May 19, 2020

stbenjam commented May 19, 2020

imain commented May 19, 2020

stbenjam commented May 20, 2020

stbenjam commented May 20, 2020

abhinavdahiya commented May 20, 2020

openshift-ci-robot commented May 20, 2020

imain commented May 20, 2020

openshift-bot commented May 26, 2020

stbenjam commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 26, 2020

openshift-bot commented May 27, 2020

openshift-bot commented May 27, 2020

openshift-bot commented May 27, 2020

openshift-bot commented May 27, 2020

openshift-ci-robot commented May 27, 2020

openshift-bot commented May 27, 2020

openshift-ci-robot commented May 27, 2020