Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update-pr-preview often failing #19378

Closed
foolip opened this issue Sep 27, 2019 · 9 comments

Comments

@foolip
Copy link
Contributor

@foolip foolip commented Sep 27, 2019

The "Synchronize the Pull Request Preview / update-pr-preview" check has been failing on many pull requests lately:
#19048
#19209 (comment)
#19323 (comment)
#19370 (comment)

All of them fail like in https://github.com/web-platform-tests/wpt/pull/19048/checks?check_run_id=221550321:

INFO:__main__:Issuing request: GET https://api.github.com/repos/web-platform-tests/wpt/collaborators/LukeZielinski
INFO:__main__:Adding label
INFO:__main__:Issuing request: POST https://api.github.com/repos/web-platform-tests/wpt/issues/19048/labels
Traceback (most recent call last):
  File "tools/ci/update_pr_preview.py", line 253, in <module>
    code = main(sys.argv[1])
  File "tools/ci/update_pr_preview.py", line 225, in main
    github.add_label(pr_number, active_label)
  File "tools/ci/update_pr_preview.py", line 161, in add_label
    request(url, 'post', json_data=data)
  File "tools/ci/update_pr_preview.py", line 65, in request
    resp.raise_for_status()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.github.com/repos/web-platform-tests/wpt/issues/19048/labels
##[error]Docker run failed with exit code 1
@foolip

This comment has been minimized.

Copy link
Contributor Author

@foolip foolip commented Sep 27, 2019

@jugglinmike we've already decided to improve the visibility of the the deployment in Q4 which will involve touching this job, so assigning to you.

If it keeps failing like this, since it's not possible to discover where it's been deployed to even when successful, then I'd like to disable it until that work starts.

@jugglinmike

This comment has been minimized.

Copy link
Contributor

@jugglinmike jugglinmike commented Oct 1, 2019

It appears that the permissions model of the GitHub Actions feature has changed and that the new implementation is fundamentally incompatible with the solution we designed. I believe we'll need to re-implement the "git tag" and "Pull Request label" operations on an external server.

Comparing the pull requests which have received the pull-request-has-preview label against the pull requests which have not received the label, the Action only seems to succeed when running for a pull request which has been opened in this repository. In other words: it doesn't work for forks. This is an unacceptable limitation for a feature intended to support all contributors generally.

When initially implemented, we demonstrated that this feature would work for contributors that do not have "write" access to the repository. About three weeks ago, a GitHub employee wrote:

For the recent update for GitHub Actions we have made a lot of changes to how workflows are run.

The documentation referenced in our script no longer exists, and the new documentation states:

The permissions for the GITHUB_TOKEN in forked repositories is read-only. For more information about the GITHUB_TOKEN, see "Virtual environments for GitHub Actions."

Which leads me to believe that we'll need to grant the necessary permissions to an external service and re-write the GitHub Action to notify that service.

This is a non-trivial amount of work, though, so I'd appreciate it if @foolip or any of the other infrastructure maintainers could verify my interpretation!

foolip added a commit that referenced this issue Oct 1, 2019
As discovered in #19378,
the existing setup isn't compatible with the GitHub Actions read-only tokens
for pull requests from forks.

Since the location of the staging deployment isn't very discoverable even
when it does work, disable the job pending a new setup.
@foolip

This comment has been minimized.

Copy link
Contributor Author

@foolip foolip commented Oct 1, 2019

A token with read-only access to the target repository makes good sense for PRs from forks, and I can understand why GitHub changes it to work like that.

Without write access we could still deploy (by polling open pull requests) but can't in any way point to it from the pull request. I agree it seems like we need to have an external service doing the commenting/tagging/whatever.

Pending a new solution I've sent #19456, since it likely won't remain in that form and it's not working as intended.

@foolip

This comment has been minimized.

Copy link
Contributor Author

@foolip foolip commented Oct 1, 2019

Whether we post comments or use GitHub's deployments API I'm inclined to say that the same code that monitors for open PRs in need of deployment would also post back when the deployment is done. That there are multiple instances for load balancing probably complicates this somewhat, but it has to be possible to answer the question "will the URL load" somehow.

foolip added a commit that referenced this issue Oct 1, 2019
As discovered in #19378,
the existing setup isn't compatible with the GitHub Actions read-only tokens
for pull requests from forks.

Since the location of the staging deployment isn't very discoverable even
when it does work, disable the job pending a new setup.
@jugglinmike

This comment has been minimized.

Copy link
Contributor

@jugglinmike jugglinmike commented Oct 8, 2019

GitHub Actions supports triggering Workflow events on a pre-defined schedule. A quick experiment on the wpt-actions-test project proved that the token supplied to these executions has write permissions.

By querying the GitHub API, we could implement issue labeling and commit tagging completely in-repository. From the perspective of maintenance and simplicity, this seems preferable to the solution I suggested above. Some risks come to mind, though.

  1. Noise. We'll be creating a lot of no-op invocations to the project's Actions log. The UI supports filtering by Workflow title, so if anyone needs to review the history, they can easily ignore the extra information.
  2. API rate limiting. Probably we won't need to issue too many requests to achieve this, but it's worth considering. We could mitigate the risk by only considering pull requests that have had activity in the past 24 hours (via the updated: search syntax) and those that have been submitted directly to WPT (filtering out bots using the -author: search syntax).
  3. Responsiveness. We've designed wpt.live to manage pull request deployments by polling the git repo for tags. If we define those tags through a polling approach, we'll have layered two polling strategies.
@foolip

This comment has been minimized.

Copy link
Contributor Author

@foolip foolip commented Oct 8, 2019

Rather than mirroring pull requests from forks automatically, perhaps we can trigger it by a special label being added and reacting to that event? Can you check if the token for those actions has write access?

@jugglinmike

This comment has been minimized.

Copy link
Contributor

@jugglinmike jugglinmike commented Oct 8, 2019

The documentation for the "Pull Request Event" in GitHub Actions has the following note about "pull request" Events for forked repositories:

The permissions for the GITHUB_TOKEN in forked repositories is read-only. For more information about the GITHUB_TOKEN, see "Virtual environments for GitHub Actions."

The referenced document shows that "Access by forked repos" is read-only for all available Events.

@foolip

This comment has been minimized.

Copy link
Contributor Author

@foolip foolip commented Oct 9, 2019

Alright, I guess that does make sense, it'd be a more complicated model to reason about if what I suggested did work.

Since the workflow no longer exists I'm going to close this issue. @jugglinmike can you file another, if there isn't one already, for getting deployments to wptpr.live working and reported on PRs? We can discuss there what the tradeoffs are, and I'd like to get @Hexcles's feedback there as well.

@jugglinmike

This comment has been minimized.

Copy link
Contributor

@jugglinmike jugglinmike commented Oct 9, 2019

Sure! See gh-19607.

moz-v2v-gh pushed a commit to mozilla/gecko-dev that referenced this issue Oct 14, 2019
…eview job, a=testonly

Automatic update from web-platform-tests
[GitHub Actions] remove the update-pr-preview job (#19456)

As discovered in web-platform-tests/wpt#19378,
the existing setup isn't compatible with the GitHub Actions read-only tokens
for pull requests from forks.

Since the location of the staging deployment isn't very discoverable even
when it does work, disable the job pending a new setup.
--

wpt-commits: 81b79d799e8f87639725ef965ac16b436b7498bb
wpt-pr: 19456
xeonchen pushed a commit to xeonchen/gecko that referenced this issue Oct 14, 2019
…eview job, a=testonly

Automatic update from web-platform-tests
[GitHub Actions] remove the update-pr-preview job (#19456)

As discovered in web-platform-tests/wpt#19378,
the existing setup isn't compatible with the GitHub Actions read-only tokens
for pull requests from forks.

Since the location of the staging deployment isn't very discoverable even
when it does work, disable the job pending a new setup.
--

wpt-commits: 81b79d799e8f87639725ef965ac16b436b7498bb
wpt-pr: 19456
gecko-dev-updater pushed a commit to marco-c/gecko-dev-comments-removed that referenced this issue Oct 16, 2019
…eview job, a=testonly

Automatic update from web-platform-tests
[GitHub Actions] remove the update-pr-preview job (#19456)

As discovered in web-platform-tests/wpt#19378,
the existing setup isn't compatible with the GitHub Actions read-only tokens
for pull requests from forks.

Since the location of the staging deployment isn't very discoverable even
when it does work, disable the job pending a new setup.
--

wpt-commits: 81b79d799e8f87639725ef965ac16b436b7498bb
wpt-pr: 19456

UltraBlame original commit: 5c670cc39b282dd3ca862302210f3da89fda10cf
gecko-dev-updater pushed a commit to marco-c/gecko-dev-wordified-and-comments-removed that referenced this issue Oct 16, 2019
…eview job, a=testonly

Automatic update from web-platform-tests
[GitHub Actions] remove the update-pr-preview job (#19456)

As discovered in web-platform-tests/wpt#19378,
the existing setup isn't compatible with the GitHub Actions read-only tokens
for pull requests from forks.

Since the location of the staging deployment isn't very discoverable even
when it does work, disable the job pending a new setup.
--

wpt-commits: 81b79d799e8f87639725ef965ac16b436b7498bb
wpt-pr: 19456

UltraBlame original commit: 5c670cc39b282dd3ca862302210f3da89fda10cf
gecko-dev-updater pushed a commit to marco-c/gecko-dev-wordified that referenced this issue Oct 16, 2019
…eview job, a=testonly

Automatic update from web-platform-tests
[GitHub Actions] remove the update-pr-preview job (#19456)

As discovered in web-platform-tests/wpt#19378,
the existing setup isn't compatible with the GitHub Actions read-only tokens
for pull requests from forks.

Since the location of the staging deployment isn't very discoverable even
when it does work, disable the job pending a new setup.
--

wpt-commits: 81b79d799e8f87639725ef965ac16b436b7498bb
wpt-pr: 19456

UltraBlame original commit: 5c670cc39b282dd3ca862302210f3da89fda10cf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.