Impossible to fetch Fingerbank account information

[snooker52]

Hello,
I can't load Fingerbank content.

Message from PacketFence Web-UI:
Impossible to fetch Fingerbank account information: Can't connect to api.fingerbank.org:443 Connection refused at /usr/share/perl5/vendor_perl/LWP/Protocol/http.pm line 41.

fingerbank.log:
Aug 30 13:44:58 CZSTD-PF80-P1 /usr/local/fingerbank/collector/fingerbank-collector[1777]: t=2018-08-30T13:44:58+0200 lvl=eror msg="ERROR: Wasn't able to fetch the destination hosts from the Fingerbank API: Get https://api-ss.fingerbank.org:443/api/v2/download/destination-hosts?key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: dial tcp 35.196.72.95:443: getsockopt: connection refused" pid=1777

manually wget:
wget https://api-ss.fingerbank.org:443/api/v2/download/destination-hosts?key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
--2018-08-30 13:49:02-- https://api-ss.fingerbank.org/api/v2/download/destination-hosts?key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Connecting to 10.13.1.81:8181... connected.
ERROR: cannot verify api-ss.fingerbank.org's certificate, issued by ‘/C=CA/ST=Quebec/L=Montreal/O=Inverse Inc./CN=api-ss.fingerbank.org’:
Self-signed certificate encountered.
To connect to api-ss.fingerbank.org insecurely, use `--no-check-certificate'.

[julsemaan]

@snooker52

Just as a side note, you did leak your key since in the email notification I got, it was there in the last line.
I suggest you change your key.

Next, it seems that you're using a proxy. The collector supports the use of a proxy if set using environment variables (HTTP_PROXY, HTTPS_PROXY) but will not support man in the middle SSL decryption.

[snooker52]

Yes, i'm use proxy.
fingerbank.conf:
[proxy]
use_proxy=enabled
host=http://10.13.1.81
port=8181
verify_ssl=disabled

and
/etc/wgetrc:
https_proxy = http://10.13.1.81:8181/
http_proxy = http://10.13.1.81:8181/

This is wrong proxy definition?

[julsemaan]

Is your proxy performing man in the middle SSL decryption ? If so, its not compatible with the collector.

Also, your environment variables should be defined on the OS level so that all processes have access to it. Not only for wget.

If you're unable to do that, then you'll have to create the following unit file in /etc/systemd/system/packetfence-fingerbank-collector.service:

[Unit]
Description=PacketFence Fingerbank Collector
Wants=packetfence-config.service
After=packetfence-config.service

[Service]
StartLimitBurst=3
StartLimitInterval=10
Environment=GIN_MODE=release
Environment=HTTP_PROXY=http://10.13.1.81:8181/
ExecStartPre=/usr/local/pf/bin/pfcmd service fingerbank-collector generateconfig
ExecStart=/usr/local/fingerbank/collector/fingerbank-collector
Restart=on-failure
Slice=packetfence.slice

[Install]
WantedBy=packetfence.target

[julsemaan]

I'll have it as an action item to automatically set that environment variable for the collector based on the Fingerbank configuration in PacketFence

[snooker52]

Now messages changed, but always error to connect api.fingerbank.ogf
PacketFence Web-UI:
Impossible to fetch Fingerbank account information: Can't connect to api.fingerbank.org:443 Connection refused at /usr/share/perl5/vendor_perl/LWP/Protocol/http.pm line 41.

and
fingerbang.log:
Aug 30 14:36:57 CZSTD-PF80-P1 fingerbank: pfqueue(2095) ERROR: [mac:40:83:de:cb:72:34] Error while communicating with the Fingerbank API to check if device 35911 is linked to device 2. 500 Can't connect to api.fingerbank.org:443 (fingerbank::Model::Device::ANON)

[snooker52]

Aug 30 14:42:56 CZSTD-PF80-P1 fingerbank_httpd.admin: httpd.admin(1810) ERROR: [mac:unknown] Error while fetching account information (fingerbank::API::account_info)
Aug 30 14:42:59 CZSTD-PF80-P1 fingerbank_httpd.admin: httpd.admin(1810) ERROR: [mac:unknown] Error while fetching account information (fingerbank::API::account_info)
Aug 30 14:43:03 CZSTD-PF80-P1 fingerbank: pfqueue(3747) WARN: [mac:unknown] Failed to download latest version of file '/usr/local/fingerbank/db/fingerbank_Upstream.db.new' on 'https://api.fingerbank.org:443/api/v2/download/db' with the following return code: 500 Can't connect to api.fingerbank.org:443 (fingerbank::Util::fetch_file)
Aug 30 14:43:03 CZSTD-PF80-P1 fingerbank: pfqueue(3747) WARN: [mac:unknown] An error occured while updating file '/usr/local/fingerbank/db/fingerbank_Upstream.db' (fingerbank::Util::update_file)

[julsemaan]

I took some time to get everything tested and there was a bug in the proxy implementation of the fingerbank perl library.

I've pushed new packages for CentOS 7, try updating the fingerbank packages and try again:

yum update fingerbank\* --enablerepo=packetfence

Then, for the settings, put:

[proxy]
use_proxy=enabled
host=10.13.1.81
port=8181
verify_ssl=disabled

[snooker52]

Thank you

[gticomunica]

Hi, I'm having the same problems with proxy settings. Can anyone tell me what was the solution. I'm using the latests packages in CentoOS 7

I think the problems in my install are:
-Dns requests are not going through the proxy
-Requsts to the local server are wrongfully going through the proxy, my proxy logs shows:
1 SERVER_IP TCP_DENIED/403 3423 CONNECT SERVER_IP :2379 - HIER_NONE/- text/html

And if add fingerbank dns to /etc/hosts I also get :
51878 SERVER_IP X TCP_MISS/200 3524 CONNECT api-ss.fingerbank.org:443 - HIER_DIRECT/35.196.72.95

[nqb]

Hello @gticomunica,

Check #3699 (comment) and try to update fingerbank package.

[gticomunica]

Hi, I updated to latest devel version and have the same problem:

The message I get in packetfence web gui is:

"Impossible to fetch Fingerbank account information: 501 Protocol scheme 'connect' is not supported"

Squid logs show:
1540392389.221 300314 SERVER_IP TCP_MISS/200 3555 CONNECT api-ss.fingerbank.org:443 - HIER_DIRECT/35.196.72.95 -
1540392429.198 0 SERVER_IP TCP_DENIED/403 3423 CONNECT SERVER_IP:2379 - HIER_NONE/- text/html

[julsemaan]

Is your proxy terminating SSL ?

Also, this issue is unrelated, in this issue, the proxy wasn't respected. In yours, the proxy is respected but your proxy isn't able to connect to Fingerbank

[gticomunica]

The company web proxy it's an old microsoft ISA server (it does not terminate ssl).

But for this issue I installed a temporary debian squid proxy (I can surf the web just fine), with no restrictions , just clean install , all allowed .

I also removed the /etc/hosts entries

And the message in squid is :

1540394471.528 0 SERVER_IP TCP_DENIED/403 3423 CONNECT SERVER_IP:2379 - HIER_NONE/- text/html

And packetfence log:

pfqueue: pfqueue(2508) ERROR: [mac:unknown] Error while communicating with the Fingerbank collector. 501 Protocol scheme 'connect' is not supported (pf::fingerbank::endpoint_attributes)
pfqueue: pfqueue(2508) ERROR: [mac:unknown] Unable to fetch query arguments for Fingerbank query. Aborting. (pf::fingerbank::process)

If I changed to other proxy , packetfence gives the same massage:

pfqueue: pfqueue(2508) ERROR: [mac:unknown] Error while communicating with the Fingerbank collector. 501 Protocol scheme 'connect' is not supported (pf::fingerbank::endpoint_attributes)
pfqueue: pfqueue(2508) ERROR: [mac:unknown] Unable to fetch query arguments for Fingerbank query. Aborting. (pf::fingerbank::process)

[julsemaan]

Please provide:

rpm -qa | grep fingerbank

rpm -qa | grep packetfence

[gticomunica]

rpm -qa | grep packetfence
packetfence-ntlm-wrapper-8.1.0-2.el7.x86_64
packetfence-release-1.2-7.el7.noarch
packetfence-pfcmd-suid-8.1.0-2.el7.x86_64
packetfence-8.1.0-2.el7.noarch
packetfence-config-8.1.0-2.el7.noarch

rpm -qa | grep fingerbank
fingerbank-collector-1.0.2-2.1.x86_64
fingerbank-4.1.2-2.1.noarch

I also commented in squid :
#Deny CONNECT to other than secure SSL ports
#http_access deny CONNECT !SSL_ports

Still same error,

[julsemaan]

Try downgrading to the stable version of the fingerbank library:

yum downgrade fingerbank --enablerepo=packetfence

[gticomunica]

Hi, I tried downgrading and I get the same error:

pfqueue: pfqueue(2508) ERROR: [mac:unknown] Error while communicating with the Fingerbank collector. 501 Protocol scheme 'connect' is not supported (pf::fingerbank::endpoint_attributes)

[gticomunica]

Why I'm seeing attempts to connect to PacketFenceIP:2379 in the proxy server ? those shouln't go through the proxy . Isn't that part of the problem?

[julsemaan]

I think this is because we currently have to set the proxy for the whole OS for Fingerbank (the HTTP_PROXY variable) so that means everything uses that.

Port 2379 is for etcd so that means your etcd requests are going through your proxy.

Although this isn't a problem for Fingerbank, it might cause little glitches for pfdhcp but nothing major.

When you were running fingerbank-4.1.2 as @nqb suggested, did you restart all the PacketFence services after upgrading ?

[gticomunica]

Hi, when I tried 4.1.2 , I restarted fingerbank and just in case the whole server because is not in production yet (until I figure out fingerbank issue).

I have set HTTP_PROXY in /etc/environment :

export http_proxy=http://proxy_ip:3128
export https_proxy=http://proxy_ip:3128

I also tried unsetting those variables and the problem continues .

[gticomunica]

This are the logs:

/usr/local/pf/logs/fingerbank.log

Oct 25 13:30:29 servername fingerbank_httpd.admin: httpd.admin(2265) INFO: [mac:unknown] Database /usr/local/fingerb ank/db/fingerbank_Local.db was changed or handles weren't initialized. Creating handle. (fingerbank::DB::SQLite::build_handle)
Oct 25 13:30:29 servername fingerbank_httpd.admin: httpd.admin(2265) INFO: [mac:unknown] Searching for 'Combination' entries in schema(s) returned an empty set (fingerbank::Base::CRUD::search)
Oct 25 13:30:31 servername fingerbank_httpd.admin: httpd.admin(2265) ERROR: [mac:unknown] Error while fetching accou nt information (fingerbank::API::account_info)

/usr/local/pf/logs/packetfence.log

Oct 25 13:31:19 servername pfqueue: pfqueue(2813) ERROR: [mac:unknown] Error while communicating with the Fingerbank collector. 501 Protocol scheme 'connect' is not supported (pf::fingerbank::endpoint_attributes)
Oct 25 13:31:19 servername pfqueue: pfqueue(2813) ERROR: [mac:unknown] Unable to fetch query arguments for Fingerbank query. Aborting. (pf::fingerbank::process)

[julsemaan]

I'll likely have to dig into it and replicate this in lab.

[julsemaan]

Try applying the following:

cd /usr/local/fingerbank/
curl https://github.com/fingerbank/perl-client/commit/56eeec9c4701c68051950164119d4210ee8c66c0.diff | patch -p1

And then restart all services

[gticomunica]

Hello errors changed now, and it shows an empy account information on the web gui , the errors are as follows:

/usr/local/pf/logs/packetfence.log

Oct 25 15:27:07 lnx-nac pfqueue: pfqueue(2655) ERROR: [mac:unknown] Error while communicating with the Fingerbank collector. 400 could not load https::connect protocol support: Can't locate LWP/Protocol/https/connect.pm in @inc (@inc contains: /usr/local/pf/lib /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at (eval 2558) line 2. (pf::fingerbank::endpoint_attributes)
Oct 25 15:27:07 lnx-nac pfqueue: pfqueue(2655) ERROR: [mac:unknown] Unable to fetch query arguments for Fingerbank query. Aborting. (pf::fingerbank::process)

/usr/local/pf/logs/fingerbank.log

Oct 25 15:27:04 lnx-nac fingerbank_httpd.admin: httpd.admin(2223) ERROR: [mac:unknown] Error while fetching account information (fingerbank::API::account_info)
Oct 25 15:27:06 lnx-nac fingerbank_httpd.admin: httpd.admin(2223) INFO: [mac:unknown] Searching for 'Combination' entries in schema(s) returned an empty set (fingerbank::Base::CRUD::search)
Oct 25 15:27:07 lnx-nac fingerbank_httpd.admin: httpd.admin(2223) ERROR: [mac:unknown] Error while fetching account information (fingerbank::API::account_info)

Thanks for taking your time for this problem.

[julsemaan]

I think I finally got it right

Try:

# cd /usr/local/fingerbank
# curl https://github.com/fingerbank/perl-client/compare/56eeec9c4701c68051950164119d4210ee8c66c0...934485d0368bb65c5b05e9eaf34316f670bf218c.diff | patch -p1
# rpm -Uvh http://inverse.ca/downloads/PacketFence/CentOS7/devel/x86_64/RPMS/perl-LWP-Protocol-connect-6.09-11.1.noarch.rpm

Then restart and retry

[gticomunica]

Wow , I think we are almost there!

-I saw 4 request on my api.fingerbank.org profile.
-Fingerbank Profiling web gui page now shows My username , account type and request in the current hour

Now I'm seeing connect attempts to 127.0.0.1:4723 coming through the proxy.

This are the logs:

Squid:

1540569250.719 0 10.1.1.131 TCP_MISS/503 0 CONNECT 127.0.0.1:4723 - HIER_NONE/- -

/usr/local/pf/logs/packetfence.log:

Oct 26 12:58:02 lnx-nac pfqueue: pfqueue(2590) ERROR: [mac:unknown] Unable to fetch query arguments for Fingerbank query. Aborting. (pf::fingerbank::process)
Oct 26 12:58:04 lnx-nac pfqueue: pfqueue(2590) ERROR: [mac:unknown] Error while communicating with the Fingerbank collector. 500 error while CONNECT thru proxy: 500 Unable to connect (pf::fingerbank::endpoint_attributes)

/usr/local/pf/logs/fingerbank.log:

No errors just this warning
Oct 26 12:59:07 lnx-nac /usr/local/fingerbank/collector/fingerbank-collector[1981]: t=2018-10-26T12:59:07-0300 lvl=warn msg="Couldn't find a peer that has more uptime than this collector. Will not sync from anybody." pid=198

[julsemaan]

Try this:

# cd /usr/local/fingerbank
# curl https://github.com/fingerbank/perl-client/commit/befddf5478ac8c03c1ac891b101521b8fe6763fd.diff | patch -p1

And then as usual, restart and retry

[gticomunica]

Hi, I think it kinda works. How can I check fingerbank works correctly ?

I now randomly (only sometimes) see this connect attempt in the proxy:

1540570932.013 131386 10.1.1.131 TCP_MISS/200 58 CONNECT 10.1.1.131:2379 - HIER_DIRECT/10.1.1.131 -
1540570932.016 1 10.1.1.131 TCP_MISS/503 0 CONNECT 10.1.1.131:2379 - HIER_NONE/- -

This error shows randomly too:
/usr/local/pf/logs/fingerbank.log:
Oct 26 14:15:48 lnx-nac fingerbank_httpd.admin: httpd.admin(2261) ERROR: [mac:unknown] Error while fetching account information (fingerbank::API::account_info)

And this warning:
/usr/local/pf/logs/packetfence.log

Oct 26 14:15:15 lnx-nac /usr/local/fingerbank/collector/fingerbank-collector[2029]: t=2018-10-26T14:15:15-0300 lvl=warn msg="Couldn't find a peer that has more uptime than this collector. Will not sync from anybody." pid=2029

How can I check fingerbank works correctly ?

[gticomunica]

Well aside from the occasional error and random attempts in the proxy to connect to 10.1.1.131:2379 , I think it's working ok. I'm doing first steps in packetfence configuration.

Are these fixes going to be in the next versions ?

Thanks for all your help

[julsemaan]

The fixes will be in 8.2

As for the occasional hits on port 2379, this is etcd which will be removed in 8.2 so it will be "fixed" that way

[gticomunica]

Just confirming... Finished setting up packefence, and tested it with a new laptop and fingerbank works perfectly !