Add squid proxy as egress cache #62103
Conversation
[ghstack-poisoned]
zhouzhuojie
requested review from
driazati,
janeyx99 and
seemethere
as code owners
🔗 Helpful links
💊 CI failures summary and remediationsAs of commit fbe2640 (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions to the (internal) Dr. CI Users group. |
[ghstack-poisoned]
|
Can you please add a bit more details to PR description? I.e. why both |
[ghstack-poisoned]
[ghstack-poisoned]
This PR adds a squid proxy that's deployed dedicated for PyTorch CI. Initially we only roll out to GHA, and if things are ok we will extend this to circleci tests if necessary. `http_proxy` and `https_proxy` are compatible with the following http clients: - curl - wget - python [ghstack-poisoned]
This PR adds a squid proxy that's deployed dedicated for PyTorch CI. Initially we only roll out to GHA, and if things are ok we will extend this to circleci tests if necessary. `http_proxy` and `https_proxy` are compatible with the following http clients: - curl - wget - python Existing cache policy: ``` refresh_pattern -i .(7z|deb|rpm|exe|zip|tar|tgz|gz|ram|rar|bin|tiff|bz2|run|csv|sh)$ 1440 80% 2880 ``` It uses the standard squid refresh_pattern for cache requests. In our setup, we tried to cache at least (1440 minutes - 1 day) and at max (2880 minutes - 2 days), with last-modified factor 80% ([squid doc](http://www.squid-cache.org/Doc/config/refresh_pattern/)). Please refer to [pytorch/test-infra](https://github.com/pytorch/test-infra/tree/master/aws/websites/squid-proxy) for details. Right now, it only applies to the `build` and `test` step, to limit the scope and make sure build and test are more reliable with egress cache. [ghstack-poisoned]
|
@zhouzhuojie has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Yes, the proxy is in the critical path, and we will beef up the monitoring of it |
The common http clients usually respect both format, and lowercase is kind of the default. Found a good summary on this topic https://about.gitlab.com/blog/2021/01/27/we-need-to-talk-no-proxy/
|
|
@zhouzhuojie merged this pull request in e63160d. |
|
This pull request has been reverted by f16102f. |
6 participants
Stack from ghstack:
This PR adds a squid proxy that's deployed dedicated for PyTorch CI. Initially we only roll out to GHA, and if things are ok we will extend this to circleci tests if necessary.
http_proxyandhttps_proxyare compatible with the following http clients:Existing cache policy:
It uses the standard squid refresh_pattern for cache requests. In our setup, we tried
to cache at least (1440 minutes - 1 day) and at max (2880 minutes - 2 days), with
last-modified factor 80% (squid doc). Please refer to pytorch/test-infra for details.
Right now, it only applies to the
buildandteststep, to limit the scope and make sure build and test are more reliable with egress cache.Differential Revision: D29892919