[JENKINS-72469] Avoid repeated tool downloads from misconfigured HTTP servers #8814

MarkEWaite · 2023-12-31T16:09:00Z

[JENKINS-72469] Avoid repeated tool downloads from misconfigured HTTP servers

The Azul Systems content delivery network stopped providing the last-modified header in their URL responses. They only provide the ETag header.

Add ETag support (based on the ETag specification) to the Jenkins FilePath URL download method so that if ETag is provided, we use the ETag value. If last-modified is provided and matches, we continue to honor it as well.

JENKINS-72469 has more details.

The community post also includes more details.

Testing done

Automated test added to FilePathTest for code changes on the controller. The automated test confirms that even without a last-modified value, the later downloads are skipped if a matching ETag is received. The automated test also confirms that download is skipped if OK is received with a matching ETag. No automated test was added to confirm download on the agent because that path is not tested by any of the other test automation of this class.
Interactive test with the Azul Systems JDK installer on the controller. I created a tool installer for the Azul JDK. I verified that before this change it was downloaded each time the job was run. I verified that after the change it was downloaded only once.
Interactive test with the Azul Systems JDK installer on an agent. I created a tool installer for the Azul JDK. I verified that before this change it was downloaded each time the job was run. I verified that after the change it was downloaded only once.
Interactive test on the controller with a file download from an NGINX web server confirmed that the tool is downloaded once and then later runs of the job did not download the file again.

Proposed changelog entries

Avoid repeated tool downloads from misconfigured HTTP servers.

Proposed upgrade guidelines

N/A

Submitter checklist

Give feedback

The Jira issue, if it exists, is well-described.

The Jira issue, if it exists, is well-described.
Options
Successfully updated the issue's project

There was an error updating the issue's project
The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see [examples](https://github.com/jenkins-infra/jenkins.io/blob/master/content/_data/changelogs/weekly.yml)). Fill in the **Proposed upgrade guidelines** section only if there are breaking changes or changes that may require extra steps from users during upgrade.

The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples). Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during upgrade.
Options
Successfully updated the issue's project

There was an error updating the issue's project
There is automated testing or an explanation as to why this change has no tests.

There is automated testing or an explanation as to why this change has no tests.
Options
Successfully updated the issue's project

There was an error updating the issue's project
New public classes, fields, and methods are annotated with `@Restricted` or have `@since TODO` Javadocs, as appropriate.

New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadocs, as appropriate.
Options
Successfully updated the issue's project

There was an error updating the issue's project
New deprecations are annotated with `@Deprecated(since = "TODO")` or `@Deprecated(forRemoval = true, since = "TODO")`, if applicable.

New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO"), if applicable.
Options
Successfully updated the issue's project

There was an error updating the issue's project
New or substantially changed JavaScript is not defined inline and does not call `eval` to ease future introduction of Content Security Policy (CSP) directives (see [documentation](https://www.jenkins.io/doc/developer/security/csp/)).

New or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content Security Policy (CSP) directives (see documentation).
Options
Successfully updated the issue's project

There was an error updating the issue's project
For dependency updates, there are links to external changelogs and, if possible, full differentials.

For dependency updates, there are links to external changelogs and, if possible, full differentials.
Options
Successfully updated the issue's project

There was an error updating the issue's project
For new APIs and extension points, there is a link to at least one consumer.

For new APIs and extension points, there is a link to at least one consumer.
Options
Successfully updated the issue's project

There was an error updating the issue's project
Options

Desired reviewers

N/A

Before the changes are marked as ready-for-merge:

Maintainer checklist

Give feedback

There are at least two (2) approvals for the pull request and no outstanding requests for change.

There are at least two (2) approvals for the pull request and no outstanding requests for change.
Options
Successfully updated the issue's project

There was an error updating the issue's project
Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.

Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
Options
Successfully updated the issue's project

There was an error updating the issue's project
Changelog entries in the pull request title and/or **Proposed changelog entries** are accurate, human-readable, and in the imperative mood.

Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
Options
Successfully updated the issue's project

There was an error updating the issue's project
Proper changelog labels are set so that the changelog can be generated automatically.

Proper changelog labels are set so that the changelog can be generated automatically.
Options
Successfully updated the issue's project

There was an error updating the issue's project
If the change needs additional upgrade steps from users, the `upgrade-guide-needed` label is set and there is a **Proposed upgrade guidelines** section in the pull request title (see [example](https://github.com/jenkinsci/jenkins/pull/4387)).

If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
Options
Successfully updated the issue's project

There was an error updating the issue's project
If it would make sense to backport the change to LTS, a Jira issue must exist, be a _Bug_ or _Improvement_, and be labeled as `lts-candidate` to be considered (see [query](https://issues.jenkins.io/issues/?filter=12146)).

If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).
Options
Successfully updated the issue's project

There was an error updating the issue's project
Options

… servers The Azul Systems content delivery network stopped providing the last-modified header in their URL responses. They only provide the ETag header. Add ETag support to the Jenkins FilePath URL download method so that if ETag is provided, we use the ETag value. If last-modified is provided and matches, we continue to honor it as well. https://issues.jenkins.io/browse/JENKINS-72469 has more details. https://community.jenkins.io/t/job-stuck-on-unpacking-global-jdk-tool/11272 also includes more details. Testing done * Automated test added to FilePathTest for code changes on the controller. The automated test confirms that even without a last-modified value, the later downloads are skipped if a matching ETag is received. The automated test also confirms that download is skipped if OK is received with a matching ETag. No automated test was added to confirm download on the agent because that path is not tested by any of the other test automation of this class. * Interactive test with the Azul Systems JDK installer on the controller. I created a tool installer for the Azul JDK. I verified that before this change it was downloaded each time the job was run. I verified that after the change it was downloaded only once. * Interactive test with the Azul Systems JDK installer on an agent. I created a tool installer for the Azul JDK. I verified that before this change it was downloaded each time the job was run. I verified that after the change it was downloaded only once. * Interactive test on the controller with a file download from an NGINX web server confirmed that the tool is downloaded once and then later runs of the job did not download the file again.

Don't risk that a substring of an earlier ETag might cause a later ETag to incorrectly assume it does not need to download a modified installer.

core/src/main/java/hudson/FilePath.java

@mawinter69

https://httpwg.org/specs/rfc9110.html#field.etag describes weak comparison cases and notes that content providers may provide weak or strong entity tags. Updated code to correctly compare weak and strong entity tags. Also improves the null checks based on the suggestions from @mawinter69 in jenkinsci#8814 (comment)

Cover more branches in the equalEtags method as well

mawinter69

Looks good now

timja

/label ready-for-merge

This PR is now ready for merge, after ~24 hours, we will merge it if there's no negative feedback.

Thanks!

MarkEWaite · 2024-01-01T22:01:16Z

Thanks for the approval @timja.

I plan to merge this pull request shortly before I go to sleep tonight so that it will be included in the weekly build tomorrow. I'd like it in at least one weekly before we select the next LTS baseline.

@mawinter69

… servers (jenkinsci#8814) * [JENKINS-72469] Avoid repeated tool downloads from misconfigured HTTP servers The Azul Systems content delivery network stopped providing the last-modified header in their URL responses. They only provide the ETag header. Add ETag support to the Jenkins FilePath URL download method so that if ETag is provided, we use the ETag value. If last-modified is provided and matches, we continue to honor it as well. https://issues.jenkins.io/browse/JENKINS-72469 has more details. https://community.jenkins.io/t/job-stuck-on-unpacking-global-jdk-tool/11272 also includes more details. Testing done * Automated test added to FilePathTest for code changes on the controller. The automated test confirms that even without a last-modified value, the later downloads are skipped if a matching ETag is received. The automated test also confirms that download is skipped if OK is received with a matching ETag. No automated test was added to confirm download on the agent because that path is not tested by any of the other test automation of this class. * Interactive test with the Azul Systems JDK installer on the controller. I created a tool installer for the Azul JDK. I verified that before this change it was downloaded each time the job was run. I verified that after the change it was downloaded only once. * Interactive test with the Azul Systems JDK installer on an agent. I created a tool installer for the Azul JDK. I verified that before this change it was downloaded each time the job was run. I verified that after the change it was downloaded only once. * Interactive test on the controller with a file download from an NGINX web server confirmed that the tool is downloaded once and then later runs of the job did not download the file again. * Use equals instead of contains to check ETag Don't risk that a substring of an earlier ETag might cause a later ETag to incorrectly assume it does not need to download a modified installer. * Use weak comparison for ETag values https://httpwg.org/specs/rfc9110.html#field.etag describes weak comparison cases and notes that content providers may provide weak or strong entity tags. Updated code to correctly compare weak and strong entity tags. Also improves the null checks based on the suggestions from @mawinter69 in jenkinsci#8814 (comment) * Test comparison of weak and strong validators * Do not duplicate test args, more readable * Use better variable names in test Cover more branches in the equalEtags method as well * Fix variable declaration order (cherry picked from commit c8156d4)

MarkEWaite added the bug For changelog: Minor bug. Will be listed after features label Dec 31, 2023

Use equals instead of contains to check ETag

cd0487e

Don't risk that a substring of an earlier ETag might cause a later ETag to incorrectly assume it does not need to download a modified installer.

mawinter69 reviewed Dec 31, 2023

View reviewed changes

core/src/main/java/hudson/FilePath.java Outdated Show resolved Hide resolved

core/src/main/java/hudson/FilePath.java Outdated Show resolved Hide resolved

mawinter69 suggested changes Dec 31, 2023

View reviewed changes

core/src/main/java/hudson/FilePath.java Outdated Show resolved Hide resolved

MarkEWaite requested a review from mawinter69 December 31, 2023 22:40

MarkEWaite added the squash-merge-me Unclean or useless commit history, should be merged only with squash-merge label Dec 31, 2023

MarkEWaite added 4 commits December 31, 2023 16:59

Test comparison of weak and strong validators

cc50570

Do not duplicate test args, more readable

b068176

Use better variable names in test

056ec5d

Cover more branches in the equalEtags method as well

Fix variable declaration order

aa3578d

mawinter69 approved these changes Jan 1, 2024

View reviewed changes

MarkEWaite requested a review from a team January 1, 2024 18:03

timja approved these changes Jan 1, 2024

View reviewed changes

comment-ops-bot bot added the ready-for-merge The PR is ready to go, and it will be merged soon if there is no negative feedback label Jan 1, 2024

MarkEWaite merged commit c8156d4 into jenkinsci:master Jan 2, 2024
17 checks passed

MarkEWaite deleted the add-etag-support-to-tool-downloader branch January 2, 2024 03:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JENKINS-72469] Avoid repeated tool downloads from misconfigured HTTP servers #8814

[JENKINS-72469] Avoid repeated tool downloads from misconfigured HTTP servers #8814

MarkEWaite commented Dec 31, 2023 •

edited

Submitter checklist

Maintainer checklist

mawinter69 left a comment

timja left a comment

MarkEWaite commented Jan 1, 2024

[JENKINS-72469] Avoid repeated tool downloads from misconfigured HTTP servers #8814

[JENKINS-72469] Avoid repeated tool downloads from misconfigured HTTP servers #8814

Conversation

MarkEWaite commented Dec 31, 2023 • edited