Skip to content
This repository has been archived by the owner on Sep 26, 2018. It is now read-only.

BTS connected but there's no telemetry data #79

Closed
i-am-nut opened this issue Sep 6, 2017 · 28 comments
Closed

BTS connected but there's no telemetry data #79

i-am-nut opened this issue Sep 6, 2017 · 28 comments

Comments

@i-am-nut
Copy link

i-am-nut commented Sep 6, 2017

I managed to connect a SysmoBTS to CCM. I'm running osmo-nitb on osmocom vm client and osmo-bts on sysmo. Issuing show bts 0 on osmo-nitb vty from vm client shows that it's connected.

Also, i've registrated snowflake ID of osmocom vm client on the cloud and checking the dashboard it says "active".

However, when clicking on "Tower Monitoring" to see the stats, it says:
This tower is running old software (version "None") and is not sending telemetry data in checkins.

I'm reading the scripts to understand what's going on but could get nothing so far.
What might be wrong? Any thoughts?

@kheimerl
Copy link
Contributor

kheimerl commented Sep 6, 2017 via email

@i-am-nut
Copy link
Author

Makes sense, but i still can't understand why endaga_version is not being sent to the checkin. May you clarify it a bit more?

@kheimerl
Copy link
Contributor

kheimerl commented Sep 21, 2017 via email

@i-am-nut
Copy link
Author

i-am-nut commented Sep 21, 2017

I did run 'endaga_db_get endaga_version'

vagrant@endaga-client-osmocom:~$ endaga_db_get endaga_version
0.8.1
vagrant@endaga-client-osmocom:~$ 

How may it not being sent in the checkin as it is correctly set?

@i-am-nut
Copy link
Author

I'm not sure if it helps in something, but on Web VM, output of python manage.py runserver 192.168.40.10:8000 shows the following:

[21/Sep/2017 15:24:02] "GET /dashboard/towers/68700135-6342-4821-9b9f-31254fd8cac5 HTTP/1.1" 200 8323
Error handling checkin (BTS 68700135-6342-4821-9b9f-31254fd8cac5): list index out of range
BTS status: {u'system_utilization': {u'memory_percent': 75.3, u'disk_percent': 41.6, u'bytes_sent_delta': 156942, u'bytes_received_delta': 327168, u'cpu_percent': 0.0}, u'uptime': 4051, u'versions': {u'openbts-public': None, u'python-gsm': u'', u'type': u'osmocom', u'osmocom-public': u'0.15.1.20170703', u'python-osmocom': u'1.0', u'python-openbts': None, u'endaga': u'0.8.1', u'freeswitch': u'1.6.16~33~e6d643b-1~jessie+1', u'python-endaga-core': u'0.6.1', u'gsm': u'0.15.1.20170703'}, u'camped_subscribers': [], u'openbts_load': {u'checkin.process_lat': 0.006463766098022461, u'checkin.raw_rsp_sz': 130982, u'checkin.rsp_sz': 130982, u'checkin.req_sz': 494, u'checkin.lat': 1.9099562168121338, u'checkin.post_lat': 0.868234395980835, u'checkin.raw_req_sz': 920}, u'openbts_noise': {u'some_noise_stats_here_tbd': 0}, u'radio': {u'band': u'GSM900', u'c0': u'63'}, u'subscribers': {}, u'usage': {u'events': []}}
Internal Server Error: /api/v1/checkin
Traceback (most recent call last):
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/rest_framework/views.py", line 466, in dispatch
    response = self.handle_exception(exc)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/rest_framework/views.py", line 463, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/vagrant/cloud/endagaweb/views/api.py", line 592, in post
    resp = checkin.CheckinResponder(bts).process(bts_status)
  File "/home/vagrant/cloud/endagaweb/checkin.py", line 161, in process
    resp['config'] = self._optimize('config', self.gen_config())
  File "/home/vagrant/cloud/endagaweb/checkin.py", line 394, in gen_config
    channel='stable').order_by('-date')[0].version
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/db/models/query.py", line 296, in __getitem__
    return list(qs)[0]
IndexError: list index out of range
[21/Sep/2017 15:26:42] ERROR [exception.py:124] Internal Server Error: /api/v1/checkin
Traceback (most recent call last):
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/rest_framework/views.py", line 466, in dispatch
    response = self.handle_exception(exc)
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/rest_framework/views.py", line 463, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/vagrant/cloud/endagaweb/views/api.py", line 592, in post
    resp = checkin.CheckinResponder(bts).process(bts_status)
  File "/home/vagrant/cloud/endagaweb/checkin.py", line 161, in process
    resp['config'] = self._optimize('config', self.gen_config())
  File "/home/vagrant/cloud/endagaweb/checkin.py", line 394, in gen_config
    channel='stable').order_by('-date')[0].version
  File "/home/vagrant/.virtualenvs/endaga/local/lib/python2.7/site-packages/django/db/models/query.py", line 296, in __getitem__
    return list(qs)[0]
IndexError: list index out of range
[21/Sep/2017 15:26:43] "POST /api/v1/checkin?id=68700135 HTTP/1.1" 500 130978

Any ideas?

@kheimerl
Copy link
Contributor

kheimerl commented Sep 22, 2017 via email

@kheimerl
Copy link
Contributor

kheimerl commented Sep 22, 2017 via email

@i-am-nut
Copy link
Author

i-am-nut commented Sep 25, 2017

Nice, you've got it!

It seems the issue is solved now, but before closing it i'm wondering what's the meaning in setting first a test db with some fictional data in order to use the application properly?
If you hadn't told me, i'd never realize it.
Sorry if i'm not getting the point.

@kheimerl
Copy link
Contributor

kheimerl commented Sep 25, 2017 via email

@marvdias
Copy link

Hi, I have a similar problem, I can see the tower status as "Active" however there is no telemetry data. The message on the monitor tab says "There are no monitoring stats available. This tower should send some data over soon.", I guess the check-in is not working as well. On web cloud I have messages like these:

[20/Oct/2017 18:45:35] "GET /api/v1/bts/register?vpn_ip=10.64.0.18&bts_uuid=346eae3b-e2d4-402e-aeed-f6c38cd73be9&federer_port=80&vpn_status=up HTTP/1.1" 200 112
[20/Oct/2017 18:45:37] "GET /api/v1/bts/register?vpn_ip=10.64.0.18&bts_uuid=346eae3b-e2d4-402e-aeed-f6c38cd73be9&vpn_status=up&federer_port=80 HTTP/1.1" 200 112
[20/Oct/2017 18:45:40] "GET /api/v1/bts/register?vpn_status=up&vpn_ip=10.64.0.18&federer_port=80&bts_uuid=346eae3b-e2d4-402e-aeed-f6c38cd73be9 HTTP/1.1" 200 112
[20/Oct/2017 18:45:44] "GET /api/v1/bts/register?federer_port=80&bts_uuid=346eae3b-e2d4-402e-aeed-f6c38cd73be9&vpn_ip=10.64.0.18&vpn_status=up HTTP/1.1" 200 112
[20/Oct/2017 18:45:55] "GET /api/v1/bts/register?federer_port=80&vpn_ip=10.64.0.18&bts_uuid=346eae3b-e2d4-402e-aeed-f6c38cd73be9&vpn_status=up HTTP/1.1" 200 112
[20/Oct/2017 18:45:58] "GET /api/v1/bts/register?vpn_ip=10.64.0.18&bts_uuid=346eae3b-e2d4-402e-aeed-f6c38cd73be9&federer_port=80&vpn_status=up HTTP/1.1" 200 112
[20/Oct/2017 18:46:02] "GET /api/v1/bts/register?vpn_ip=10.64.0.18&bts_uuid=346eae3b-e2d4-402e-aeed-f6c38cd73be9&vpn_status=up&federer_port=80 HTTP/1.1" 200 112
[20/Oct/2017 18:46:06] "GET /api/v1/bts/register?federer_port=80&vpn_status=up&bts_uuid=346eae3b-e2d4-402e-aeed-f6c38cd73be9&vpn_ip=10.64.0.18 HTTP/1.1" 200 112

The endaga log shows a critical error :

2017-10-20T18:49:00.409213+00:00 [CRITICAL] endaga: endagad:22:<module>: caught unhandled exception: 'python-gsm_version', Traceback (most recent call last):#012  File "/usr/local/lib/python3.4/dist-packages/core/db/kvstore.py", line 79, in __getitem__#012    (key, ))#012  File "/usr/local/lib/python3.4/dist-packages/core/db/connector.py", line 132, in exec_and_fetch_one#012    raise IndexError("no value generated by '%s'" % (stmt, ))#012IndexError: no value generated by 'SELECT value FROM endaga_config WHERE key = %s;'#012#012During handling of the above exception, another exception occurred:#012#012Traceback (most recent call last):#012  File "/usr/local/bin/endagad", line 19, in <module>#012    EndagaD().run()#012  File "/usr/local/lib/python3.4/dist-packages/core/endagad.py", line 112, in run#012    checkin_data = eapi.checkin(timeout=30)#012  File "/usr/local/lib/python3.4/dist-packages/core/interconnect.py", line 129, in checkin#012    status['versions'] = bts.get_versions()#012  File "/usr/local/lib/python3.4/dist-packages/core/bts/_osmocom.py", line 243, in get_versions#012    versions['python-osmocom'] = self.conf['python-gsm_version']#012  File "/usr/local/lib/python3.4/dist-packages/core/config_database.py", line 55, in __getitem__#012    return self._ducktype(super(ConfigDB, self).__getitem__(key))#012  File "/usr/local/lib/python3.4/dist-packages/core/db/kvstore.py", line 82, in __getitem__#012    raise KeyError(key)#012KeyError: 'python-gsm_version'

Is it normal? What should I do?

@kheimerl
Copy link
Contributor

kheimerl commented Oct 20, 2017 via email

@marvdias
Copy link

Yes, before it I ran "python manage.py setup_test_db"

@kheimerl
Copy link
Contributor

kheimerl commented Oct 20, 2017 via email

@marvdias
Copy link

Yes there is dummy data, for the other towers randomly created.
Does the client have ['python-gsm_version'] in the local database
('endaga_db_get')?
This message was shown after the command:

endaga_db_get python-gsm_version
no such key: python-gsm_version

@kheimerl
Copy link
Contributor

kheimerl commented Oct 20, 2017 via email

@marvdias
Copy link

Well, I ran update_installed_versions, and it only updates the following:

vagrant@endaga-client-osmocom:~$ sudo update_installed_versions
endaga:0.8.1
freeswitch:1.6.16~33~e6d643b-1~jessie+1
gsm:0.15.1.20171001

Is there anything else I can do?

@kheimerl
Copy link
Contributor

kheimerl commented Oct 20, 2017 via email

@marvdias
Copy link

marvdias commented Oct 20, 2017

Well, I don't know what could have gone wrong.. I just installed the endaga-osmocom package. I looked for the python3-osmocom and:

vagrant@endaga-client-osmocom:~$ sudo apt-cache search python3-osmocom
python3-osmocom - Osmocom VTY client
vagrant@endaga-client-osmocom:~$ sudo apt-get install python3-osmocom
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-osmocom is already the newest version.
python3-osmocom set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
20 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up libcurl3:amd64 (7.38.0-4+deb8u6) ...
Setting up libcurl3-gnutls:amd64 (7.38.0-4+deb8u6) ...
Setting up libosmocore8:amd64 (0.9.6.20171019) ...
Setting up libosmocodec0:amd64 (0.9.6.20171019) ...
Setting up libosmogsm7:amd64 (0.9.6.20171019) ...
Setting up libosmocoding0:amd64 (0.9.6.20171019) ...
Setting up libosmovty3:amd64 (0.9.6.20171019) ...
Setting up libosmoctrl0:amd64 (0.9.6.20171019) ...
Setting up libosmogb4:amd64 (0.9.6.20171019) ...
Setting up libsmpp0:amd64 (1.12.0.20171019) ...
Setting up curl (7.38.0-4+deb8u6) ...
Setting up libgtp1:amd64 (1.0.0.20171019) ...
Setting up libosmoabis5:amd64 (0.4.0.20171019) ...
Setting up libosmotrau1:amd64 (0.4.0.20171019) ...
Setting up libsmpp34-dev:amd64 (1.12.0.20171019) ...
Setting up osmo-bts-trx (0.6.0.20171019) ...
Setting up osmo-pcu (0.3.0.20171019) ...
Setting up osmo-sip-connector (1.20171019) ...
Setting up osmo-trx (0.1.9.20171019) ...
Setting up osmocom-sgsn (0.15.1.20171019) ...
Processing triggers for libc-bin (2.19-18+deb8u10) ...

@marvdias
Copy link

Is there anyway to proceed with another clean osmocom environment without destroying this one?

@kheimerl
Copy link
Contributor

kheimerl commented Oct 20, 2017 via email

@marvdias
Copy link

Alternatively you can just set it manually with endaga_db_set

I just did it and set to '1.0' version.
Now in the web cloud :

[20/Oct/2017 20:38:18] "GET /api/v1/bts/register?vpn_status=up&federer_port=80&vpn_ip=10.64.0.18&bts_uuid=346eae3b-e2d4-402e-aeed-f6c38cd73be9 HTTP/1.1" 200 112
[20/Oct/2017 20:38:19] INFO [checkin.py:80] got delta section 'config' from BTS 346eae3b-e2d4-402e-aeed-f6c38cd73be9: {u'alg': u'md5', u'sig': u'9ede14274f99566cea24c9bba126f31a'}
[20/Oct/2017 20:38:19] INFO [checkin.py:80] got delta section 'subscribers' from BTS 346eae3b-e2d4-402e-aeed-f6c38cd73be9: {u'alg': u'md5', u'sig': u'f5166904a4c424eb6f171dd8c66c03ea'}
[20/Oct/2017 20:38:20] INFO [connectionpool.py:257] Starting new HTTP connection (1): 192.168.40.10
[20/Oct/2017 20:38:20] "POST /sason/acquire/ HTTP/1.1" 200 9
[20/Oct/2017 20:38:20] "POST /api/v1/checkin?id=346eae3b HTTP/1.1" 200 282

and the endaga log :

2017-10-20T20:36:01.699820+00:00 [ERROR] endaga: checkin.py:63:process: Unexpected checkin section: sas
2017-10-20T20:36:01.727910+00:00 [ERROR] endaga: base.py:312:process_bts_settings: Failed to process openbts setting(): Control.LUR.OpenRegistration = ^90155

2017-10-20T20:37:09.017253+00:00 [ERROR] endaga: servicecontrol.py:106:startProcess: Supervisor: failed to start osmo-trx, fault code: 40


2017-10-20T20:37:12.821034+00:00 [ERROR] endaga: checkin.py:63:process: Unexpected checkin section: sas
2017-10-20T20:38:16.746848+00:00 [ERROR] endaga: servicecontrol.py:106:startProcess: Supervisor: failed to start osmo-trx, fault code: 40
2017-10-20T20:38:20.362358+00:00 [ERROR] endaga: checkin.py:63:process: Unexpected checkin section: sas
2017-10-20T20:39:28.085085+00:00 [ERROR] endaga: servicecontrol.py:106:startProcess: Supervisor: failed to start osmo-trx, fault code: 40
2017-10-20T20:39:30.065419+00:00 [ERROR] endaga: checkin.py:63:process: Unexpected checkin section: sas

@marvdias
Copy link

Just looked into the monitor tab, and it is showing some data!

@kheimerl
Copy link
Contributor

kheimerl commented Oct 20, 2017 via email

@marvdias
Copy link

No, I just cloned the master. Did the vagrant up (and provision) for the cloud. In the client used the apt-get to install the endaga-osmocom packages

@kheimerl
Copy link
Contributor

kheimerl commented Oct 20, 2017 via email

@marvdias
Copy link

Another thing that i had to do is in the cert cloud. I had to install the fpm manually (ssh to cert and use the folllowing command )- it does not skip this step otherwise.

sudo apt-get install ruby-dev gcc
sudo gem install fpm

@kheimerl
Copy link
Contributor

kheimerl commented Oct 20, 2017 via email

@9muir
Copy link
Contributor

9muir commented Jan 3, 2018

There's an Ansible role, setup_fpm, that should take care of installing all the dependencies for FPM. It seems to work for me, in particular the 'Build certifier package' step of the Ansible certifier_base role, which is what uses FPM, succeeded and the playbook was executed without errors.

@marvdias if this is still a problem for you please post the output of vagrant up cert in the cloud directory (after deleting the exist certifier VM), and reopen the task.

@9muir 9muir closed this as completed Jan 3, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants