CVMFS_HTTP_PROXY list ignored: proxy set to DIRECT

[adriansev]

Hi! I see in our logs (cvmfs debug log) this sequence:

(download) first fallback proxy group 3    [11-10-2023 18:14:06 EET]
(download) full proxy list http://alien.spacescience.ro:3128; http://issaf.spacescience.ro:3128; http://monitor.spacescience.ro:3128    [11-10-2023 18:14:06 EET]
(download) resolving 3 proxy addresses    [11-10-2023 18:14:06 EET]
(dns) add address alien.spacescience.ro -> 85.120.46.17    [11-10-2023 18:14:06 EET]
(dns) failed to resolve  http - 4 (unknown host name), ttl 60    [11-10-2023 18:14:06 EET]
(dns) failed to resolve  http - 4 (unknown host name), ttl 60    [11-10-2023 18:14:06 EET]
(download) failed to resolve IP addresses for  http (4 - unknown host name)    [11-10-2023 18:14:06 EET]
(download) failed to resolve IP addresses for  http (4 - unknown host name)    [11-10-2023 18:14:06 EET]
(download) installed 3 proxies in 3 load-balance groups    [11-10-2023 18:14:06 EET]
(download) switching proxy from (none) to http://85.120.46.17:3128 (set proxies)    [11-10-2023 18:14:06 EET]
(download) requesting ordered server list from http://cvmfs-stratum-one.cern.ch/cvmfs/cvmfs-config.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [11-10-2023 18:14:06 EET]
(download) replacing @proxy@ by alien.spacescience.ro    [11-10-2023 18:14:06 EET]

then a bit later

(download) switching proxy from (none) to http://85.120.46.17:3128 (cloned)    [11-10-2023 18:14:06 EET]
(download) first fallback proxy group 1    [11-10-2023 18:14:06 EET]
(download) full proxy list DIRECT    [11-10-2023 18:14:06 EET]
(download) resolving 1 proxy addresses    [11-10-2023 18:14:06 EET]
(dns) empty hostname    [11-10-2023 18:14:06 EET]
(download) installed 1 proxies in 1 load-balance groups    [11-10-2023 18:14:06 EET]
(download) switching proxy from http://85.120.46.17:3128 to DIRECT (set proxies)    [11-10-2023 18:14:06 EET]
(cvmfs) DNS roaming is disabled for this repository.    [11-10-2023 18:14:06 EET]

full log and bug report here: https://asevcenc.web.cern.ch/asevcenc/cvmfs_bug_report/

Any idea why the proxy list is not honored?
Thanks a lot!

[vvolkl]

Hi Adrian,

I think the wording of the log messages might be misleading here. Can you post the output of

cvmfs_config stat -v alice.cern.ch

on your client?

[adriansev]

Hi! Hmm right, on all involved clients i get something like:

cvmfs_config stat -v alice.cern.ch
Version: 2.11.2.0
PID: 7356
Uptime: 38941 minutes
Memory Usage: 593352k
File Catalog Revision: 17074 (expires in 1 minutes)
File Catalog ID: cb806c1cda6dd3af5a0f5b8475ce39501c19b1b1
No. Active File Catalogs: 20
Cache Usage: 27518073k / 52428801k
File Descriptor Usage: 87 / 1048064
No. Open Directories: 0
No. IO Errors: 0
Connection: http://cvmfs-stratum-one.cern.ch:8000/cvmfs/alice.cern.ch through proxy http://85.120.46.17:3128 (online)
Usage: 126917 open() calls (hitrate 99.988%), 5208 opendir() calls
Transfer Statistics: 73766k read, avg. speed: 1729k/s

so, from my side it looks good and i would close the issue... but could you confirm please that you see no other potential problem in the logs, so to have an "all green" confirmation ? :D
Thanks a lot!! :)

[DrDaveD]

Let's leave the issue open for improving the log message to not be deceiving.

[HereThereBeDragons]

improvement of log messages should include the following:

• dns/download: add hostname which failed to resolve
• add what type of proxies are handled: normal proxies vs fallback proxies
• add what type of downloadmanager (external vs normal) is emitting the messages

[DrDaveD]

We have just had another case where we have been misled by that "switching proxy from ... to DIRECT". Please include a fix for this in the next release. I'll set this to a 2.11.3 milestone.

[HereThereBeDragons]

hi @adriansev

i am currently working on enhancing the debug messages.
one thing i can already tell you is that your config is faulty: CVMFS_HTTP_PROXY does not like whitespaces. please remove them
(i looked in the documentation: nothing is written about whitespace-usage but also no example given that uses whitespaces)

see broken (CVMFS_HTTP_PROXY='http://alien.spacescience.ro:3128; http://issaf.spacescience.ro:3128; http://monitor.spacescience.ro:3128':

(download) (manager 'standard') first fallback proxy group 3    [04-09-2024 15:00:34 CEST]
(download) (manager 'standard') full proxy list http://alien.spacescience.ro:3128; http://issaf.spacescience.ro:3128; http://monitor.spacescience.ro:3128    [04-09-2024 15:00:34 CEST]
(download) (manager 'standard') resolving 3 proxy addresses    [04-09-2024 15:00:34 CEST]
(dns) add address alien.spacescience.ro -> 85.120.46.17    [04-09-2024 15:00:34 CEST]
(dns) add address alien.spacescience.ro -> 2001:b30:4210:1::17    [04-09-2024 15:00:34 CEST]
(dns) failed to resolve  http - 1 (invalid resolver addresses), ttl 60    [04-09-2024 15:00:34 CEST]
(dns) failed to resolve  http - 1 (invalid resolver addresses), ttl 60    [04-09-2024 15:00:34 CEST]
(download) (manager 'standard') failed to resolve IP addresses for  http (1 - invalid resolver addresses)    [04-09-2024 15:00:34 CEST]
(download) (manager 'standard') failed to resolve IP addresses for  http (1 - invalid resolver addresses)    [04-09-2024 15:00:34 CEST]
(download) (manager 'standard') installed 3 proxies in 3 load-balance groups    [04-09-2024 15:00:34 CEST]
(download) (manager 'standard') switching proxy from (none) to http://85.120.46.17:3128 (set proxies)    [04-09-2024 15:00:34 CEST]
(download) (manager 'standard') requesting ordered server list from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/alice-ocdb.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [04-09-2024 15:00:34 CEST]

see fixed (CVMFS_HTTP_PROXY='http://alien.spacescience.ro:3128;http://issaf.spacescience.ro:3128;http://monitor.spacescience.ro:3128'):

(download) (manager 'standard') first fallback proxy group 3    [04-09-2024 15:01:19 CEST]
(download) (manager 'standard') full proxy list http://alien.spacescience.ro:3128;http://issaf.spacescience.ro:3128;http://monitor.spacescience.ro:3128    [04-09-2024 15:01:19 CEST]
(download) (manager 'standard') resolving 3 proxy addresses    [04-09-2024 15:01:19 CEST]
(dns) add address alien.spacescience.ro -> 85.120.46.17    [04-09-2024 15:01:19 CEST]
(dns) add address alien.spacescience.ro -> 2001:b30:4210:1::17    [04-09-2024 15:01:19 CEST]
(dns) add address issaf.spacescience.ro -> 46.243.114.226    [04-09-2024 15:01:19 CEST]
(dns) add address issaf.spacescience.ro -> 2001:b30:4210::226    [04-09-2024 15:01:19 CEST]
(dns) add address monitor.spacescience.ro -> 46.243.114.227    [04-09-2024 15:01:19 CEST]
(dns) add address monitor.spacescience.ro -> 2001:b30:4210::227    [04-09-2024 15:01:19 CEST]
(download) (manager 'standard') installed 3 proxies in 3 load-balance groups    [04-09-2024 15:01:19 CEST]
(download) (manager 'standard') switching proxy from (none) to http://85.120.46.17:3128 (set proxies)    [04-09-2024 15:01:19 CEST]
(download) (manager 'standard') requesting ordered server list from http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/alice-ocdb.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [04-09-2024 15:01:19 CEST]
(download) (id 0) replacing @proxy@ by alien.spacescience.ro    [04-09-2024 15:01:19 CEST]

[adriansev]

@HereThereBeDragons Thanks a lot for this!!! i assumed that during tokenization there will be some stripping on the tokens and i did not payed any attention to this..

[HereThereBeDragons]

closing, PR was merged