[release-4.11] ci: set up separate homedir for Prow builder #1070

mike-nguyen · 2022-11-22T02:50:55Z

This is a backport of the CI fix in the main branch. The 4.11 PR's are failing with the same error.

Currently, we try to fake that the OpenShift-allocated uid is a valid user by adding it to /etc/passwd and using a home dir of /home/builder. But that doesn't work because anything that wants to write to the home dir will get EACCES.

The exact error was:
+ cosa buildextend-extensions
Extracting a84ad9e41b55dae5dcdbad814604adc8973fcfb2681d27ab2908193ea54b4cd4
error: Permission denied (os error 13)
Using strace, the files it failed on were:
[pid  1498] mkdir("/home/builder/.cache", 0700 <unfinished ...>
[pid  1498] <... mkdir resumed>)        = -1 EACCES (Permission denied)
[pid  1498] openat(AT_FDCWD, "/home/builder/.cache/ostree/auth.json", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid  1498] <... openat resumed>)       = -1 EACCES (Permission denied)
Probably something to do with the container stack when invoked by rpm-ostree?

Anyway, there might be something to fix there, but for now hack around this by instead allocating a temporary separate homedir.

Currently, we try to fake that the OpenShift-allocated uid is a valid user by adding it to `/etc/passwd` and using a home dir of `/home/builder`. But that doesn't work because anything that wants to write to the home dir will get `EACCES`. The exact error was: ``` + cosa buildextend-extensions Extracting a84ad9e41b55dae5dcdbad814604adc8973fcfb2681d27ab2908193ea54b4cd4 error: Permission denied (os error 13) ``` Using strace, the files it failed on were: ``` [pid 1498] mkdir("/home/builder/.cache", 0700 <unfinished ...> [pid 1498] <... mkdir resumed>) = -1 EACCES (Permission denied) [pid 1498] openat(AT_FDCWD, "/home/builder/.cache/ostree/auth.json", O_RDONLY|O_CLOEXEC <unfinished ...> [pid 1498] <... openat resumed>) = -1 EACCES (Permission denied) ``` Probably something to do with the container stack when invoked by rpm-ostree? Anyway, there might be something to fix there, but for now hack around this by instead allocating a temporary separate homedir.

openshift-ci · 2022-11-22T02:50:58Z

@mike-nguyen: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

ci: set up separate homedir for Prow builder

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-11-22T02:51:12Z

@mike-nguyen: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.11] ci: set up separate homedir for Prow builder

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-11-22T02:53:03Z

@mike-nguyen: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.11] ci: set up separate homedir for Prow builder

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-11-22T05:21:56Z

@mike-nguyen: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

HuijingHei

LGTM

travier · 2022-11-22T08:13:09Z

/lgtm

travier · 2022-11-22T08:13:41Z

/label backport-risk-assessed

travier · 2022-11-22T08:14:30Z

/label cherry-pick-approved

openshift-ci · 2022-11-22T08:15:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: HuijingHei, mike-nguyen, travier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [HuijingHei,mike-nguyen,travier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

travier · 2022-11-22T10:29:06Z

/label valid-bug
This is for CI, we don't need a bug

openshift-ci · 2022-11-22T10:29:09Z

@travier: The label(s) /label valid-bug cannot be applied. These labels are supported: platform/aws, platform/azure, platform/baremetal, platform/google, platform/libvirt, platform/openstack, ga, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, px-approved, docs-approved, qe-approved, downstream-change-needed, approved, backport-risk-assessed, bugzilla/valid-bug, cherry-pick-approved, jira/valid-bug, staff-eng-approved. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label valid-bug
This is for CI, we don't need a bug

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

travier · 2022-11-22T10:29:38Z

@mrunalp Can we get an override for the bug validation? Thanks

mike-nguyen · 2022-11-22T15:00:13Z

/label bugzilla/valid-bug
/label jira/valid-bug

Attempt as seeing if the right label works.

openshift-ci · 2022-11-22T15:01:54Z

@mike-nguyen: Can not set label bugzilla/valid-bug: Must be member in one of these teams: [openshift-patch-managers openshift-staff-engineers]

In response to this:

/label bugzilla/valid-bug
/label jira/valid-bug

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-11-22T15:03:28Z

@mike-nguyen: Can not set label jira/valid-bug: Must be member in one of these teams: [openshift-patch-managers openshift-staff-engineers]

In response to this:

/label bugzilla/valid-bug
/label jira/valid-bug

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2023-02-21T01:01:09Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

travier · 2023-02-21T09:37:57Z

/remove-lifecycle stale

@mike-nguyen For when you get back: Do we still need this one?

openshift-bot · 2023-05-23T01:00:12Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

mike-nguyen · 2023-05-23T12:31:30Z

/retest

openshift-bot · 2023-06-23T00:30:16Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2023-07-23T08:00:42Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci · 2023-07-23T08:01:12Z

@openshift-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mike-nguyen changed the title ~~ci: set up separate homedir for Prow builder~~ [release-4.11] ci: set up separate homedir for Prow builder Nov 22, 2022

openshift-ci bot requested review from dustymabe and HuijingHei November 22, 2022 02:51

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 22, 2022

mike-nguyen mentioned this pull request Nov 22, 2022

OCPBUGS-3959: [release-4.11] Use kernel-rt from ose repo #1064

Merged

HuijingHei approved these changes Nov 22, 2022

View reviewed changes

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Nov 22, 2022

openshift-ci bot assigned travier Nov 22, 2022

openshift-ci bot added lgtm Indicates that a PR is ready to be merged. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. labels Nov 22, 2022

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 21, 2023

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 21, 2023

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 23, 2023

openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 23, 2023

openshift-ci bot closed this Jul 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[release-4.11] ci: set up separate homedir for Prow builder #1070

[release-4.11] ci: set up separate homedir for Prow builder #1070

mike-nguyen commented Nov 22, 2022 •

edited

openshift-ci bot commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

HuijingHei left a comment

travier commented Nov 22, 2022

travier commented Nov 22, 2022

travier commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

travier commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

travier commented Nov 22, 2022

mike-nguyen commented Nov 22, 2022 •

edited

openshift-ci bot commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

openshift-bot commented Feb 21, 2023

travier commented Feb 21, 2023 •

edited

openshift-bot commented May 23, 2023

mike-nguyen commented May 23, 2023

openshift-bot commented Jun 23, 2023

openshift-bot commented Jul 23, 2023

openshift-ci bot commented Jul 23, 2023

[release-4.11] ci: set up separate homedir for Prow builder #1070

[release-4.11] ci: set up separate homedir for Prow builder #1070

Conversation

mike-nguyen commented Nov 22, 2022 • edited

openshift-ci bot commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

HuijingHei left a comment

Choose a reason for hiding this comment

travier commented Nov 22, 2022

travier commented Nov 22, 2022

travier commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

travier commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

travier commented Nov 22, 2022

mike-nguyen commented Nov 22, 2022 • edited

openshift-ci bot commented Nov 22, 2022

openshift-ci bot commented Nov 22, 2022

openshift-bot commented Feb 21, 2023

travier commented Feb 21, 2023 • edited

openshift-bot commented May 23, 2023

mike-nguyen commented May 23, 2023

openshift-bot commented Jun 23, 2023

openshift-bot commented Jul 23, 2023

openshift-ci bot commented Jul 23, 2023

mike-nguyen commented Nov 22, 2022 •

edited

mike-nguyen commented Nov 22, 2022 •

edited

travier commented Feb 21, 2023 •

edited