Does cvdupdate work with proxy #9

marekgrajcar · 2021-03-16T06:58:44Z

Hello,
due to the latest changes I am trying to download signatures using cvdupdate tool. I am performing it from inside of company’s network and I get warning: “Failed to determine available version via DNS TXT query!”.
However when I am looking for address names resolution using our internal proxy I am able to get an answer:

# nslookup database.clamav.net <proxy_ip>
Server:         <proxy_ip>
Address:        <proxy_ip>#53

Non-authoritative answer:
database.clamav.net     canonical name = database.clamav.net.cdn.cloudflare.net.
Name:   database.clamav.net.cdn.cloudflare.net
Address: 104.16.219.84
Name:   database.clamav.net.cdn.cloudflare.net
Address: 104.16.218.84
Name:   database.clamav.net.cdn.cloudflare.net
Address: 2606:4700::6810:da54
Name:   database.clamav.net.cdn.cloudflare.net
Address: 2606:4700::6810:db54

Proxy server can forward DNS requests but cannot act as name server. When updating “nameserver” in /root/.cvdupdate/config.json with proxy address I got timeout like below:

# cvd update -V
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking main.cvd for update from https://database.clamav.net/main.cvd
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-03-16 07:40:28 cvdupdate-0.2.0 INFO Using nameserver specified in the config: 10.111.0.162
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking main.cvd version via DNS TXT advertisement.
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG main.cvd version advertised by DNS: 59
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Downloading CDIFFs first...
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking for main-59.cdiff
2021-03-16 07:40:28 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): database.clamav.net:443
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='database.clamav.net', port=443): Max retries exceeded with url: /main-59.cdiff (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/cvd", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 257, in update_alias
    ctx.forward(db_update)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 628, in forward
    return self.invoke(cmd, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 100, in db_update
    errors = m.db_update(db)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 781, in db_update
    if update(db) == False:
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 767, in update
    return self._download_cvd(db, advertised_version)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 612, in _download_cvd
    'User-Agent': f'ClamAV/{self.dns_version_tokens[0]} (cvdupdate-{self.version})',
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='database.clamav.net', port=443): Max retries exceeded with url: /main-59.cdiff (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

I am wondering if it’s and option to forward request via proxy correctly, i.e. putting proxy ip in /root/.cvdupdate/config.json as a feature.

The text was updated successfully, but these errors were encountered:

micahsnyder · 2021-03-16T19:37:30Z

It looks like another user was able to get cvdupdate working with a proxy this past weekend: #7

I'd like to verify their solution and (if it works) add it to the README file. If you get it working, would you consider helping me with the documentation?

devdroid0 · 2021-03-17T16:40:28Z

I am having problems with the proxy settings as well. Our proxy requires authentication and request is not parsing it:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 412, in send
conn = self.get_connection(request.url, proxies)
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 305, in get_connection
proxy_url = parse_url(proxy)
File "/usr/local/lib/python3.6/site-packages/urllib3-1.26.4-py3.6.egg/urllib3/util/url.py", line 392, in parse_url
return six.raise_from(LocationParseError(source_url), None)
File "", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: requests.exceptions.InvalidURL: Failed to parse: http://myUsername:myPassword@proxy.domain.com:80/

It seems to be using the http_proxy and https_proxy environment variables. Tried lots of different combinations but to get this far and the proxy to not reject the connection (407) this is a close as I've gotten.

marekgrajcar · 2021-03-18T09:39:22Z

Hi,

I was able to reproduce solution from https://github.com/micahsnyder/cvdupdate/issues/7. Now I am able to reach for clamav signatures being behind proxy. What I have done is:

In /root/.cvdupdate/config.json I have updated nameserver with proxy address:
"nameserver": "<proxy_ip>",
While running cvd update inside of script I set http_proxy and https_proxy and that usually does a trick:
#!/bin/bash
http_proxy=http://<proxy_ip>:<proxy_port>
export http_proxy
https_proxy=http://<proxy_ip>:<proxy_port>
export https_proxy
/usr/local/bin/cvd update -V

devdroid0 · 2021-03-18T11:02:00Z

Use a script similar to above with our specific proxy information. We have to use authentication with our proxy. All the attempt to not use authentication string in the URL resulted in a 407:

ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',

Iteration 1: Set the nameserver kev value in the config.json to one of the ip addresses of our proxy server and got this:

2021-03-18 06:31:50 cvdupdate-0.3.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-03-18 06:31:50 cvdupdate-0.3.0 INFO Using nameserver specified in the config: 10.10.7.8
2021-03-18 06:31:55 cvdupdate-0.3.0 DEBUG EXCEPTION OCCURRED: The DNS operation timed out after 5.509402751922607 seconds
2021-03-18 06:31:55 cvdupdate-0.3.0 WARNING Failed to determine available version via DNS TXT query!
Traceback (most recent call last):
File "/usr/local/bin/cvd", line 11, in
load_entry_point('cvdupdate==0.3.0', 'console_scripts', 'cvd')()
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/main.py", line 259, in update_alias
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 628, in forward
return self.invoke(cmd, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/main.py", line 101, in db_update
File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/cvdupdate.py", line 795, in db_update
KeyError: ''

Iteration 2: changed the nameserver key value in the config.json to an ip address that was a nameserver and got a little farther:

2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-03-18 06:33:09 cvdupdate-0.3.0 INFO Using nameserver specified in the config: 10.0.0.100
2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG Checking main.cvd for update from https://database.clamav.net/main.cvd
2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG Checking main.cvd version via DNS TXT advertisement.
2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG main.cvd version advertised by DNS: 59
2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG Downloading CDIFFs first...
2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG Checking for main-59.cdiff
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 412, in send
conn = self.get_connection(request.url, proxies)
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 305, in get_connection
proxy_url = parse_url(proxy)
File "/usr/local/lib/python3.6/site-packages/urllib3-1.26.4-py3.6.egg/urllib3/util/url.py", line 392, in parse_url
return six.raise_from(LocationParseError(source_url), None)
File "", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: http://myUsername:myPassword@proxy.domain.com:80/

During handling of the above exception, another exception occurred:```
Traceback (most recent call last):
File "/usr/local/bin/cvd", line 11, in
load_entry_point('cvdupdate==0.3.0', 'console_scripts', 'cvd')()
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/main.py", line 259, in update_alias
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 628, in forward
return self.invoke(cmd, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/main.py", line 101, in db_update
File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/cvdupdate.py", line 859, in db_update
File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/cvdupdate.py", line 845, in update
File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/cvdupdate.py", line 661, in _download_cvd
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 414, in send
raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Failed to parse: http://myUser:myPassword@proxy.domain.com:80/

devdroid0 · 2021-03-23T17:32:27Z

Finally got it to work. had to update our proxy with the required user-agent to allow it through unauthenticated.

Set the name server to one of our real name servers;
Set the httpd_proxy and https_proxy environment variables to our proxy FQDN:port (multiple proxy addresses)

micahsnyder · 2021-03-26T22:19:38Z

Great! I'll try to summarize all of this in the Readme when time permits. I'll keep this ticket open until I've done so.

This is for: - Proxy support issue: #9 - Cron examples issue: #11

micahsnyder self-assigned this Mar 19, 2021

micahsnyder added a commit that referenced this issue Apr 18, 2021

Add Proxy and Cron examples to the README

e16407c

This is for: - Proxy support issue: #9 - Cron examples issue: #11

micahsnyder mentioned this issue Apr 18, 2021

Add Proxy and Cron examples to the README #20

Merged

micahsnyder added a commit that referenced this issue Apr 18, 2021

Add Proxy and Cron examples to the README

e88172c

This is for: - Proxy support issue: #9 - Cron examples issue: #11

micahsnyder closed this as completed Apr 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Does cvdupdate work with proxy #9

Does cvdupdate work with proxy #9

marekgrajcar commented Mar 16, 2021

micahsnyder commented Mar 16, 2021

devdroid0 commented Mar 17, 2021

marekgrajcar commented Mar 18, 2021 •

edited

Loading

devdroid0 commented Mar 18, 2021

devdroid0 commented Mar 23, 2021 •

edited

Loading

micahsnyder commented Mar 26, 2021

Does cvdupdate work with proxy #9

Does cvdupdate work with proxy #9

Comments

marekgrajcar commented Mar 16, 2021

micahsnyder commented Mar 16, 2021

devdroid0 commented Mar 17, 2021

marekgrajcar commented Mar 18, 2021 • edited Loading

devdroid0 commented Mar 18, 2021

devdroid0 commented Mar 23, 2021 • edited Loading

micahsnyder commented Mar 26, 2021

marekgrajcar commented Mar 18, 2021 •

edited

Loading

devdroid0 commented Mar 23, 2021 •

edited

Loading