New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg install produces cache misses when using http proxy #1181

Closed
tomster opened this Issue Feb 21, 2015 · 3 comments

Comments

Projects
None yet
2 participants
@tomster
Copy link

tomster commented Feb 21, 2015

i'm using squid to speed up repeated jail setups and while pkg successfully honors its pkg_env setting, i'm experiencing nothing but cache misses when performing repeated installations.

the log below shows how the same urls are requested repeatedly, but all of them produce misses - in the beginning i've included some HIT entries (produced via zfsinstall) to demonstrate that my squid instance is working in principal.

any idea why this is not working?

in addition, here's my /usr/local/etc/pkg/pkg.conf:

pkg_env: {
  HTTP_PROXY: "http://192.168.56.1:8081",
  HTTPS_PROXY: "http://192.168.56.1:8081"
}
ASSUME_ALWAYS_YES: YES
REPO_AUTOUPDATE: NO

and my /usr/local/etc/pkg/repos/FreeBSD.conf:

FreeBSD: {
    url: "http://pkg.freeBSD.org/${ABI}/quarterly",
    mirror_type: "http",
    enabled: yes
}

note, that i've tried it with mirror types none and srv with the same result.

is this a bug or what effect is the setting of http proxy supposed to have? or am i missing something in my squid configuration?

here's the sample of squid's access log:

1424528542.910   1324 192.168.56.105 TCP_HIT/200 71183185 GET http://ftp4.de.freebsd.org/pub/FreeBSD/releases/amd64/9.3-RELEASE//base.txz - HIER_NONE/- -
1424528544.634   1708 192.168.56.105 TCP_HIT/200 89157621 GET http://ftp4.de.freebsd.org/pub/FreeBSD/releases/amd64/9.3-RELEASE//kernel.txz - HIER_NONE/- -
1424528567.035   1014 192.168.56.105 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528567.899    161 192.168.56.105 TCP_MISS/200 1359 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/meta.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528569.792   1885 192.168.56.105 TCP_MISS/200 5273054 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/packagesite.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528582.465   4995 192.168.56.105 TCP_MISS/200 10480055 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/python27-2.7.9.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528582.723     58 192.168.56.105 TCP_MISS/200 33586 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libffi-3.0.13_3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528582.780     53 192.168.56.105 TCP_MISS/200 5269 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/indexinfo-0.2.2.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528582.895    108 192.168.56.105 TCP_MISS/200 148788 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/gettext-runtime-0.19.3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528583.193    278 192.168.56.105 TCP_MISS/200 603620 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libiconv-1.14_6.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424522846.059  22926 192.168.56.100 TCP_MISS/200 71183175 GET http://ftp4.de.freebsd.org/pub/FreeBSD/releases/amd64/9.3-RELEASE//base.txz - HIER_DIRECT/194.8.197.22 -
1424522892.543  46480 192.168.56.100 TCP_MISS_ABORTED/200 80116214 GET http://ftp4.de.freebsd.org/pub/FreeBSD/releases/amd64/9.3-RELEASE//kernel.txz - HIER_DIRECT/194.8.197.22 -
1424522991.329  54304 192.168.56.100 TCP_MISS/200 89157611 GET http://ftp4.de.freebsd.org/pub/FreeBSD/releases/amd64/9.3-RELEASE//kernel.txz - HIER_DIRECT/194.8.197.22 -
1424523026.226   1896 192.168.56.100 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523027.227    173 192.168.56.100 TCP_MISS/200 1359 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/meta.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523028.585   1348 192.168.56.100 TCP_MISS/200 5273054 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/packagesite.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523039.242   3490 192.168.56.100 TCP_MISS/200 10480055 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/python27-2.7.9.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523042.409    134 192.168.56.100 TCP_MISS/200 33586 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libffi-3.0.13_3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523042.485     71 192.168.56.100 TCP_MISS/200 5269 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/indexinfo-0.2.2.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523042.580     91 192.168.56.100 TCP_MISS/200 148788 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/gettext-runtime-0.19.3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523042.855    270 192.168.56.100 TCP_MISS/200 603620 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libiconv-1.14_6.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523181.008    897 192.168.56.101 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523181.672    126 192.168.56.101 TCP_MISS/200 1359 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/meta.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523183.207   1527 192.168.56.101 TCP_MISS/200 5273054 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/packagesite.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523189.674   2366 192.168.56.101 TCP_MISS/200 10480055 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/python27-2.7.9.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523189.815     60 192.168.56.101 TCP_MISS/200 33586 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libffi-3.0.13_3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523189.871     53 192.168.56.101 TCP_MISS/200 5269 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/indexinfo-0.2.2.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523189.955     80 192.168.56.101 TCP_MISS/200 148788 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/gettext-runtime-0.19.3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523190.125    167 192.168.56.101 TCP_MISS/200 603620 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libiconv-1.14_6.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424523675.450    806 192.168.56.102 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424524287.522    801 192.168.56.102 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424525781.645    842 192.168.56.102 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424525846.953    705 192.168.56.102 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526044.884    697 192.168.56.102 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526230.942   1013 192.168.56.102 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526249.550    713 192.168.56.102 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526262.892    700 192.168.56.102 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526263.648    160 192.168.56.102 TCP_MISS/200 1359 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/meta.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526264.878   1220 192.168.56.102 TCP_MISS/200 5273054 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/packagesite.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526273.420   1875 192.168.56.102 TCP_MISS/200 10480055 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/python27-2.7.9.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526273.605     58 192.168.56.102 TCP_MISS/200 33586 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libffi-3.0.13_3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526273.662     53 192.168.56.102 TCP_MISS/200 5269 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/indexinfo-0.2.2.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526273.746     79 192.168.56.102 TCP_MISS/200 148788 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/gettext-runtime-0.19.3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424526273.937    186 192.168.56.102 TCP_MISS/200 603620 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libiconv-1.14_6.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527003.416    910 192.168.56.103 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527004.325    161 192.168.56.103 TCP_MISS/200 1359 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/meta.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527005.936   1600 192.168.56.103 TCP_MISS/200 5273054 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/packagesite.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527015.255   2583 192.168.56.103 TCP_MISS/200 10480055 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/python27-2.7.9.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527015.443     58 192.168.56.103 TCP_MISS/200 33586 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libffi-3.0.13_3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527015.499     53 192.168.56.103 TCP_MISS/200 5269 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/indexinfo-0.2.2.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527015.582     78 192.168.56.103 TCP_MISS/200 148788 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/gettext-runtime-0.19.3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527015.769    183 192.168.56.103 TCP_MISS/200 603620 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libiconv-1.14_6.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527197.116      0 192.168.56.104 TCP_MISS/503 3689 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/2001:41c8:112:8300::50:1 text/html
1424527472.188    828 192.168.56.104 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527473.187    161 192.168.56.104 TCP_MISS/200 1359 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/meta.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527474.774   1567 192.168.56.104 TCP_MISS/200 5273054 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/packagesite.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527487.589   4751 192.168.56.104 TCP_MISS/200 10480055 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/python27-2.7.9.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527487.785     58 192.168.56.104 TCP_MISS/200 33586 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libffi-3.0.13_3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527487.840     52 192.168.56.104 TCP_MISS/200 5269 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/indexinfo-0.2.2.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527487.959    105 192.168.56.104 TCP_MISS/200 148788 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/gettext-runtime-0.19.3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424527488.227    264 192.168.56.104 TCP_MISS/200 603620 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libiconv-1.14_6.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528567.035   1014 192.168.56.105 TCP_MISS/200 2037690 GET http://pkg.freebsd.org/freebsd:9:x86:64/quarterly/Latest/pkg.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528567.899    161 192.168.56.105 TCP_MISS/200 1359 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/meta.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528569.792   1885 192.168.56.105 TCP_MISS/200 5273054 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/packagesite.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528582.465   4995 192.168.56.105 TCP_MISS/200 10480055 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/python27-2.7.9.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528582.723     58 192.168.56.105 TCP_MISS/200 33586 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libffi-3.0.13_3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528582.780     53 192.168.56.105 TCP_MISS/200 5269 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/indexinfo-0.2.2.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528582.895    108 192.168.56.105 TCP_MISS/200 148788 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/gettext-runtime-0.19.3.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
1424528583.193    278 192.168.56.105 TCP_MISS/200 603620 GET http://pkgmir.pkg.freebsd.org/FreeBSD:9:amd64/quarterly/All/libiconv-1.14_6.txz - HIER_DIRECT/213.138.116.73 application/octet-stream
@tomster

This comment has been minimized.

Copy link
Author

tomster commented Feb 21, 2015

ok, the issue is not with pkg itself but rather the response headers from the pkg servers, which set IMHO rather aggressive Cache-Control:max-age=0, Cache-Control:private and extremely short expiration time.

once i've instructed squid to ignore these, everything works as expected.

@tomster tomster closed this Feb 21, 2015

@tomster

This comment has been minimized.

Copy link
Author

tomster commented Feb 22, 2015

FTR and in case anybody arrives here via search, here's a working squid configuration that effectively caches packages forever (2 years) and meta information slightly less aggressively (four weeks):

# match against download urls for specific packages - their content never changes for the same url, so we cache aggressively
refresh_pattern -i (quarterly|latest)\/All\/.*(\.txz) 1440 100% 1051200 ignore-private ignore-must-revalidate override-expire ignore-no-cache

# match against meta-information - this shouldn't be cached quite so aggressively
refresh_pattern -i (quarterly|latest)\/.*(\.txz) 1440 100% 40080 ignore-private ignore-must-revalidate override-expire ignore-no-cache

refresh_pattern ^ftp:       1440    20% 1008000
refresh_pattern ^gopher:    1440    0%  1440
refresh_pattern -i (/cgi-bin/|\?) 0 0%  0
refresh_pattern .       0   20% 4320

maximum_object_size_in_memory 32 KB
maximum_object_size 2000 MB
@johnko

This comment has been minimized.

Copy link

johnko commented Jun 19, 2015

👍 thanks @tomster

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment