New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Call/message usage reporting service for community edition #376

Closed
ivelin opened this Issue Aug 5, 2015 · 84 comments

Comments

@ivelin
Contributor

ivelin commented Aug 5, 2015

Related to #283

Related IRC discussion:

''
gvag: You mean the service that will collect the stats?
[10:25am] gvag: I think we have to go with Graylog for that, or a similar application
[10:26am] ivelin: yes, the cloud service
[10:27am] ivelin: Graylog is the tool, but I’m also thinking about the live service
[10:27am] gvag: I think Graylog, which is based on ElasticSearch but provides many nice features, is perfect for what we need
[10:27am] gvag: Can you elaborate on what you mean live service?
[10:28am] ivelin: something like stats.restcomm.com where we show a worldwide map of monthyl, weekly calls/messages served by restcomm
[10:28am] ivelin: anonymous global usage data
[10:30am] charles-r: ivelin, wouldn't some people object to us collecting data about their running Restcomm instance?
[10:31am] gvag: I am not sure about this service and what will take to build it. We can discuss it with Alex or Orestis for them to build something on AngularJS. Because I guess we don't want to have Graylog in public, unless if we can customize it completely and we provide our own logo etc
[10:31am] ivelin: charles-r , yes its optional
[10:32am] abhayani: we have more challenges here. Most of the enterprise doesn't like direct internet connection to their servers
[10:32am] abhayani: so may be an alternative could be to periodically print in logs
[10:33am] abhayani: and as and when net is available push only these lines from logs to central server
[10:33am] abhayani: just thinking about various possibilities
[10:33am] abhayani: as we want to have same for other legacy products too
[10:35am] charles-r: We could start by showing number of downloads and from how many countries in the world.
[10:35am] abhayani: gvag, I am thinking if this should be separate project which can reused by any product
[10:35am] gvag: abhayani, the monitoring services provided by Restcomm is a feature for enterprise monitoring, so customers can run their own elasticsearch or graylog or whatever monitoring server, to collect stats from Restcomm. On the side, we will collect anonymous call data for a statistics
[10:36am] gvag: abhayani, what do you have in mind for that? The monitoring service, checks for Restcomm live calls by receiving call events etc, this is something very specific to Restcomm.
[10:37am] gvag: These stats are exposed to any monitoring tool by the REST API, very Restcomm specific also
[10:37am] gvag: I don't see any other way that can be used in other projects
[10:38am] ivelin: the problem with downloads is that its hard to monitor nowadays
[10:38am] ivelin: between torrents and other sources, its hard to tell
[10:39am] ivelin: but call/message stats is something that I think will be more representative
[10:39am] ivelin: despite all the limitations you bring up.
[10:39am] ivelin: yes, some machines are behind VPN without any internat access
[10:39am] ivelin: but most are online
[10:39am] ivelin: we aren’t looking for a perfectly accurate number
[10:39am] ivelin: just a ballpark representation
[10:40am] ivelin: Do we handle tens or hundreds of millions of messages / calls per day ?
[10:41am] ivelin: let me open a new issue for 7.4
[10:41am] abhayani: for legacy we have like 100's of millions of SMS per day
[10:42am] ivelin: yes, so then overall do we handle 100’s of millions or billions?
''

@ivelin ivelin added this to the 7.4.0 milestone Aug 5, 2015

@gvagenas gvagenas modified the milestones: 7.4.0, 7.5.0 Oct 30, 2015

@gvagenas gvagenas added the Help Wanted label Jan 7, 2016

@gvagenas gvagenas modified the milestones: 7.6.0, 7.5.0 Jan 7, 2016

@gvagenas gvagenas modified the milestones: 7.7.0, 7.6.0 Feb 26, 2016

@deruelle deruelle assigned rlimonta and unassigned gvagenas May 3, 2016

@deruelle

This comment has been minimized.

Member

deruelle commented May 3, 2016

The goal is to design and implement a reusable library that can collect and send stats usage on SIP or SS7 calls and (SIP or SMPP or SS7) messages as well as charging/billing events (diameter) going through a JVM to a reliable hosted service (that you would design as well).
This library will eventually be part of all the main Restcomm projects

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented May 3, 2016

Awsome! I will start today!

I will study this item and return in case of questions.

Do you use a specific tool for the design?

Regards.

Ricardo Limonta

2016-05-03 9:14 GMT-03:00 Jean Deruelle notifications@github.com:

The goal is to design and implement a reusable library that can collect
and send stats usage on SIP or SS7 calls and (SIP or SMPP or SS7) messages
as well as charging/billing events (diameter) going through a JVM to a
reliable hosted service (that you would design as well).
This library will eventually be part of all the main Restcomm projects


You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
#376 (comment)

Ricardo Limonta

@deruelle

This comment has been minimized.

Member

deruelle commented May 3, 2016

We try to use PlantUML for design and Google Docs or asciidoc in general so far

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented May 5, 2016

Jean,

Currently the events from all stacks (SIP, SS7, SMPP, etc) are created via Log4j?

If so, I think that the collector could be a custom Log4J Appender, which receive these events and would send to the Stats Platform.

If we have different sources, I'll use a generic strategy for the collector.

@deruelle

This comment has been minimized.

Member

deruelle commented May 5, 2016

@rlimonta that's correct. that may be a good idea but 3 points to consider:

  1. in WARN or ERROR, messages are not logged at all so in production env you would get no stats
  2. Sending all events with the content of message may be overkill in terms of data sent, just counters at a configurable interval should be enough.
  3. the content of messages will not be anonymized which is a concern, we don't want to log users, IPs etc.
@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented May 5, 2016

Ok @deruelle , these points are very important!

We can create a specific log level (like a Level.STATS) and a specific appender (json format).

In this way you can decide in your stack which message goes to the central environment.

@deruelle

This comment has been minimized.

Member

deruelle commented May 5, 2016

@rlimonta do you have pseudo code of how it would like and what the various projects would have to add in the code ?

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented May 5, 2016

@deruelle follows two examples:

//String Message
logger.log(STATS, "string message");
result:
{
"logger":"org.mobicents.servlet.restcomm.sms.SmsService",
"timestamp":"1376681196470",
"level":"STATS",
"thread":"main",
"message":"string message"
}

//Object Message (Eg. instance of SmsMessage object)
//In this scenario it's possible to customize toString() method.
logger.log(STATS, smsMessage);

result:
{
"logger":"org.mobicents.servlet.restcomm.sms.SmsService",
"timestamp":"1376681196470",
"level":"STATS",
"thread":"main",
"message":"[sms.toString()]"
}

If this approach does not satisfy, I think of another strategy for the collector.

@deruelle

This comment has been minimized.

Member

deruelle commented May 5, 2016

@rlimonta This seems to be a clean, simple and effective approach to me although I have one question, some of our projects are reusing others, by example, JAIN SIP Stack is used by SIP Servlets and SIP Servlets is used by Restcomm-Connect. If each layer is using this, we would have recorded 3 times the same call or message when we reach Restcomm-Connect layer, how would you prevent that ?

Another comment, if we adopt that approach, we should probably define a number of Utility Stats classes in a common library (https://github.com/RestComm/commons) that implement the toString that you're suggesting and represent the main possibilities : Audio Calls (# of them, # of minutes, ..), Video Calls, Messages (SMS, IP), Billing, ... so not every project create a different convention and it becomes a mess on the server side ?

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented May 6, 2016

@deruelle I believe it is better to think of a different approach. the STATS not only host system logs, but also data billing, call quality, SMS messages and others. It should be more flexible in the data structure.

I will design the collector as a API that can take any object (like a java bean) and transforms it into JSON, persisting in the central system.

In this way we will have a flexible statistics system that will be able to be used by any stack.

For the back-end we should have a local environment to host the data that is able to replicate these data to the cloud, right?

I'll finish the solution design today.

@deruelle

This comment has been minimized.

Member

deruelle commented May 6, 2016

@rlimonta Thanks. Looking forward to the new design.

For the backend, I would like to have it hosted directly on cloud.
As a first step we can investigate using Google Analytics as a potential solution to report back to ?
We could then think of using platforms such as Openshift or AWS Lamba to host the backend as a second step or if Google Analytics is not possible/too restricting.

Feel free to share your thoughts on the backend as well.

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented May 6, 2016

@deruelle I was considering this point:

"[10:32am] abhayani: we have more challenges here. Most of the enterprise doesn't like direct internet connection to their servers."

So I considered primarily in having a local repository.

But if we consider the cloud as the first alternative, the solution is more simple.

@deruelle

This comment has been minimized.

Member

deruelle commented May 6, 2016

@rlimonta good catch. Let's still proceed with cloud solution first to iterate faster and not overdesign. We can think of a local proxy for onsite customers later.

@deruelle

This comment has been minimized.

Member

deruelle commented Jul 12, 2016

@rlimonta let me try to create my own environment on OpenShift to test the documentation.
Can you create a tutorial in asciidoc format push it to the statistics-service repo ?

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Jul 12, 2016

@deruelle Yes, I will work on documentation and @get methods.

@deruelle

This comment has been minimized.

Member

deruelle commented Jul 15, 2016

Thanks @rlimonta. Looking forward to set it up and start testing

@zahidMed

This comment has been minimized.

Collaborator

zahidMed commented Jul 18, 2016

@deruelle
I think it's too late for proposal. But I think that stats should be devided into differents layers (ObjectType) each ObjectType contains counters according to the nature of this layer (number of success requests, nbr of failure requests, latency ...). kpi4j was developed for that purpose and can be used in all the modules. the ObjectType activation is done through the config file and I can adapt it to your need.
https://github.com/zahidMed/kpi4j

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Jul 19, 2016

@deruelle I finished the implementation of @get methods and improved the dashboard graphics. I created a single chart by metric.

Now I need to finish the Installation Guide. I have all screenshots and need to finish the doc.

screen shot 2016-07-18 at 9 32 56 pm
screen shot 2016-07-18 at 9 33 19 pm

@deruelle

This comment has been minimized.

Member

deruelle commented Jul 21, 2016

@rlimonta looking better by the day. Looking forward to the doc to set it up !

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Jul 21, 2016

@deruelle In the last three days I had problems accessing OpenShift environment.

I do not know if this is true for all plans or only for free instances.

What do you think of using another alternative like Doker, AWS, etc ?

I received this message:
screenshot from 2016-07-20 19_20_09

@deruelle

This comment has been minimized.

Member

deruelle commented Jul 21, 2016

It seems to work on my side. But I'm fine to have a Docker guide to start with.

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Jul 21, 2016

Ok.

@deruelle

This comment has been minimized.

Member

deruelle commented Jul 28, 2016

@rlimonta any news on this yet ?

@deruelle

This comment has been minimized.

Member

deruelle commented Aug 4, 2016

@rlimonta were you able to wrap this up by any chance ?

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Aug 4, 2016

@deruelle I apologize for the delay, but I had to travel to work in the last two weeks. I need only finalize the document. I'll work on it and send you until the weekend. The OpenShift environment is running fine now!

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Aug 9, 2016

@deruelle I wrote the document README.md explaining the deployment process into OpenShift environment. If you have any comments or suggestions, please feel free to let me know.

@deruelle

This comment has been minimized.

Member

deruelle commented Aug 18, 2016

Thanks @rlimonta. I just returned from vacations. Let me check that out.

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Aug 18, 2016

@deruelle Ok Jean, feel free to any suggestion.

@deruelle

This comment has been minimized.

Member

deruelle commented Aug 23, 2016

Thanks @rlimonta I was able to get it up and running and push some test data to it but I see that when I try to change the date, the end date doesn't seem to be taken into account or something else

screenshot from 2016-08-23 11-42-05

Also I tried the GET API on http://statistics.restcomm.com/rest/counter/1471167956/1471945556/sip-invites but it didn't return anything

Shall we protect the GET API under the same username and pwd as the UI by the way ?

@deruelle

This comment has been minimized.

Member

deruelle commented Aug 23, 2016

Also restarting the OpenShift application led to loosing all the information. It seems counters are back to 0.

@deruelle

This comment has been minimized.

Member

deruelle commented Aug 24, 2016

@rlimonta

Also I tried the GET API on http://statistics.restcomm.com/rest/counter/1471167956/1471945556/sip-invites but it didn't return anything

actually I just noticed the timestamp is in seconds so using http://statistics.restcomm.com/rest/counter/1471167956000/1471945556000/sip-invites made it work.

Is there a Rest API for having a concatenated aggregated number of the counters over the period ?

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Aug 24, 2016

@deruelle Jean, I made a test on my environment and is working properly.

I have one suggestion for you simmulate quickly.

  1. Change your Remote Server on StatsCounterTest class, like bellow:

restcomm_rlimonta_openshift_countertest

I recovered the indicators through the dashboard and rest api:

restcomm_rlimonta_dashboard

restcomm_rlimonta_openshift_rest

I accessed my environment via ssh, and the data are saved on MongoDB.

restcomm_rlimonta_openshift

restcomm_rlimonta_openshift_query

I'm sending the address of my test environment:
http://rlimonta-restcommstats.rhcloud.com/
user: restcomm
pass: restcomm123

If you want, I can review your environment with you.

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Aug 24, 2016

@deruelle Jean, about your question:

"Is there a Rest API for having a concatenated aggregated number of the counters over the period ?"

The answer is no, but I can implement that. It's simple...

Best regards.

@deruelle

This comment has been minimized.

Member

deruelle commented Aug 24, 2016

Is there a Rest API for having a concatenated aggregated number of the counters over the period ?

RestComm/statistics-service#1 done ;)

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Aug 24, 2016

@deruelle

This comment has been minimized.

Member

deruelle commented Aug 24, 2016

@rlimonta on the dashboard, it works for the current day but if I try to see for a longer period of time the dashboard doesn't seem to display well for some reason. Did you try displaying values for counters inserted like a week before ?

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Aug 24, 2016

@deruelle This may be happening because of the default grouping in the chart is HH:mm. When you increase the period, it generates a lot of points to zero.

I will improve to support (month, day, hour, minute).

@deruelle

This comment has been minimized.

Member

deruelle commented Aug 25, 2016

@rlimonta also the UI seems broken under Firefox. Chrome doesn't display well as it does in your case either on Linux.

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Aug 25, 2016

@deruelle I'll check it out today.

@rlimonta

This comment has been minimized.

Collaborator

rlimonta commented Aug 31, 2016

@deruelle I implemented the following items:

  1. Firefox Issue: Fixed
  2. Dashbord Layout Improvement: Fixed
  3. Issue #1: Closed.
@deruelle

This comment has been minimized.

Member

deruelle commented Sep 2, 2016

Thanks @rlimonta !
Confirmed it works fine now !

@deruelle deruelle closed this Feb 12, 2017

@gvagenas gvagenas added this to the 8.1.0 milestone Apr 7, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment