Is your feature request related to a problem? Please describe.
We download the Maven-all.zip, unzip it, and match dependencies to vulns.
In order to make sure we're current, we call head on the zip first, and compare last-modified to a file, saved the previous time we downloaded the zip, that contains the previous last-modified (and if there is no such file, we do the initial download).
Unfortunately, last-modified seems to change once an hour, and yet when we unzip and compare to the previous contents, often nothing has changed.
We can't use size info because it changes somehow, even though the contents haven't changed.
The available fields in head are:
x-guploader-uploadid: ABPtcPpXXBXsd5ctGlbi3eURZuo-XTgoPRoPGHgv2YTIw2vwyg6F4eBuX4FzrxhMwMen2wJHc1bJJnq-DA
x-goog-generation: 1708026318109754
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 6444072
x-goog-hash: crc32c=SkCpMA==
x-goog-hash: md5=A417294/XUvR65kzO7geNQ==
x-goog-storage-class: STANDARD
accept-ranges: bytes
content-length: 6444072
server: UploadServer
date: Thu, 15 Feb 2024 19:49:19 GMT
expires: Thu, 15 Feb 2024 20:49:19 GMT
cache-control: public, max-age=3600
last-modified: Thu, 15 Feb 2024 19:45:18 GMT
etag: "038d7bdbde3f5d4bd1eb99333bb81e35"
content-type: application/zip
age: 2170
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Describe the solution you'd like
Some field in head that can be relied on to change only when there is a change to the contents of the zip that is meaningful (for example, adding a vuln, retiring a vuln, adding a CVE to a vuln, etc.)
Is your feature request related to a problem? Please describe.
We download the Maven-all.zip, unzip it, and match dependencies to vulns.
In order to make sure we're current, we call head on the zip first, and compare last-modified to a file, saved the previous time we downloaded the zip, that contains the previous last-modified (and if there is no such file, we do the initial download).
Unfortunately, last-modified seems to change once an hour, and yet when we unzip and compare to the previous contents, often nothing has changed.
We can't use size info because it changes somehow, even though the contents haven't changed.
The available fields in head are:
x-guploader-uploadid: ABPtcPpXXBXsd5ctGlbi3eURZuo-XTgoPRoPGHgv2YTIw2vwyg6F4eBuX4FzrxhMwMen2wJHc1bJJnq-DA
x-goog-generation: 1708026318109754
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 6444072
x-goog-hash: crc32c=SkCpMA==
x-goog-hash: md5=A417294/XUvR65kzO7geNQ==
x-goog-storage-class: STANDARD
accept-ranges: bytes
content-length: 6444072
server: UploadServer
date: Thu, 15 Feb 2024 19:49:19 GMT
expires: Thu, 15 Feb 2024 20:49:19 GMT
cache-control: public, max-age=3600
last-modified: Thu, 15 Feb 2024 19:45:18 GMT
etag: "038d7bdbde3f5d4bd1eb99333bb81e35"
content-type: application/zip
age: 2170
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Describe the solution you'd like
Some field in head that can be relied on to change only when there is a change to the contents of the zip that is meaningful (for example, adding a vuln, retiring a vuln, adding a CVE to a vuln, etc.)