Research drill-down analytics endpoint #2090

bajiat · 2017-02-08T11:31:13Z

Related to #2036

Test the data available from the API Umbrella 0.13 Admin API analytics endpoint. Select one of the testing deployments (nightly, staging) and apis to which there have been calls.

See documentation from about the Admin API from http://api-umbrella.readthedocs.io/en/latest/admin/api.html. Go to Analytics.

Definition of done

Make Postman calls to the analytics endpoint experimeting with the parameters.
- Check different options for prefix.
Document
- whether we will be able to get the same data structure through the API than from ElasticSearch. This is needed to know the amount of refactoring needed.
- Document whether there is missing data compared to what can be retrieved from Elastic search.

You can document your findings as part of this issue.

Add also the response objects as part of documentation.

Research

Parameters

Required Headers:
- X-Admin-Auth-Token:
- X-Api-Key:
Required Fields in request parameters:
- prefix=value**/** (it must contain a slash)
- start_at=yyyy-mm-dd
- end_at=yyyy-mm-dd
- interval=day (or month, week, day, hour, minute)
Request path:
- /api-umbrella/v1/analytics/drilldown.json (json is required)

If you have 500 error

Make sure you send all required fields and all headers are correct

If you have 406 error

Make sure your request path as .json in the end

The text was updated successfully, but these errors were encountered:

marla-singer · 2017-02-08T13:36:36Z

Tried to send request https://nightly.apinf.io:3002/api-umbrella/v1/analytics/drilldown?api_key=
The server response with 500 Error and text

I will try again tomorrow

brylie · 2017-02-09T09:27:24Z

Please avoid publishing your administrative credentials openly, e.g. on Github.

API Umbrella needs two credentials when calling the Admin API:

API Key
Admin Auth Token

The admin API is accessible via /api-umbrella/v1. In order to access this API, you must pass:

Your API key via one of the supported methods.
AND an admin token via the X-Admin-Auth-Token header.

To find the admin auth token for your admin account, login the web admin tool, and choose "My Account" under the top right gear menu. On that page, you should see your "Admin API Token" listed. Use this in conjunction with your normal API key to make requests to the admin APIs:

X-Api-Key: YOUR_API_KEY_HERE
X-Admin-Auth-Token: YOUR_ADMIN_TOKEN_HERE

marla-singer · 2017-02-09T09:28:29Z

@brylie I sent my credentials via headers and the response was the same

brylie · 2017-02-09T10:29:19Z

OK, I will try too.

brylie · 2017-02-09T10:39:34Z

What values did you use for the URL parameters:

prefix
start_at
end_at
granularity

marla-singer · 2017-02-09T10:39:45Z

@brylie I used REST client in my IDE.
I don't set the Request body, only Request Parameters

Parameters

HTTP method: GET
Host: https://nightly.apinf.io:3002
Path: /api-umbrella/v1/analytics/drilldown
Headers:

X-Api-Key: value
X-Admin-Auth-Token: value

Request Parameters:

prefix=alternative
start_at=2017-02-01
end_at=2017-02-09

marla-singer · 2017-02-09T10:40:42Z

brylie · 2017-02-09T10:44:04Z

And for granularity?

marla-singer · 2017-02-09T10:46:37Z

It's not required field. Anyway I added interval=day and it returns 406 error

brylie · 2017-02-09T10:48:48Z

It looks like the interval field (not granularity) is required:

brylie · 2017-02-09T10:49:10Z

Were you getting a 406 error before adding the interval field?

marla-singer · 2017-02-09T10:50:00Z

Nope.

If interval isn't set, I get 500 error
If interval is set, I get 406 error

brylie · 2017-02-09T10:50:52Z

In other words, we are probably getting closer now, since you have gone from a 500 error to a 406 error. Try tweaking the parameter values now, changing their formatting. If needed, open an upstream support request, so @GUI can help us figure this out.

marla-singer · 2017-02-10T09:32:07Z

Parameters

prefix is apinf-api/users/ 01.01 - 31.01 by week

Result

{
  "results":[

  ],
  "hits_over_time":{
    "cols":[
      {
        "id":"date",
        "label":"Date",
        "type":"datetime"
      }
    ],
    "rows":[
      {
        "c":[
          {
            "v":1482710400000,
            "f":"Dec 26, 2016 - Jan 1, 2017"
          }
        ]
      },
      {
        "c":[
          {
            "v":1483315200000,
            "f":"Jan 2, 2017 - Jan 8, 2017"
          }
        ]
      },
      {
        "c":[
          {
            "v":1483920000000,
            "f":"Jan 9, 2017 - Jan 15, 2017"
          }
        ]
      },
      {
        "c":[
          {
            "v":1484524800000,
            "f":"Jan 16, 2017 - Jan 22, 2017"
          }
        ]
      },
      {
        "c":[
          {
            "v":1485129600000,
            "f":"Jan 23, 2017 - Jan 29, 2017"
          }
        ]
      },
      {
        "c":[
          {
            "v":1485734400000,
            "f":"Jan 30, 2017 - Jan 31, 2017"
          }
        ]
      }
    ]
  },
  "breadcrumbs":[
    {
      "crumb":"All Hosts",
      "prefix":"0/"
    },
    {
      "crumb":"users",
      "prefix":"1/users/"
    }
  ]
}

marla-singer · 2017-02-10T10:11:37Z

Parameter

Sprecify prefix that doesn't exist. I used non-existing/ value
01.01 - 31.01 by week

Result

{
  "results":[

  ],
  "hits_over_time":{
    "cols":[
      {
        "id":"date",
        "label":"Date",
        "type":"datetime"
      }
    ],
    "rows":[
      {
        "c":[
          {
            "v":1482710400000,
            "f":"Dec 26, 2016 - Jan 1, 2017"
          }
        ]
      },
      {
        "c":[
          {
            "v":1483315200000,
            "f":"Jan 2, 2017 - Jan 8, 2017"
          }
        ]
      },
      {
        "c":[
          {
            "v":1483920000000,
            "f":"Jan 9, 2017 - Jan 15, 2017"
          }
        ]
      },
      {
        "c":[
          {
            "v":1484524800000,
            "f":"Jan 16, 2017 - Jan 22, 2017"
          }
        ]
      },
      {
        "c":[
          {
            "v":1485129600000,
            "f":"Jan 23, 2017 - Jan 29, 2017"
          }
        ]
      },
      {
        "c":[
          {
            "v":1485734400000,
            "f":"Jan 30, 2017 - Jan 31, 2017"
          }
        ]
      }
    ]
  },
  "breadcrumbs":[
    {
      "crumb":"All Hosts",
      "prefix":"0/"
    }
  ]
}

brylie · 2017-02-10T10:33:54Z

Although, this leaves us scratching our heads and asking "Where are the counts?".

"c":[
  {
    "v":1485734400000, // <-- unix timestamp?
    "f":"Jan 30, 2017 - Jan 31, 2017"// <-- date range
  }
]

My guess is the above example has no hits in that date range.

brylie · 2017-02-10T10:36:49Z

Lets see how these queries work on frontend prefixes with existing data.

marla-singer · 2017-02-10T10:38:32Z

@brylie

Lets see how these queries work on frontend prefixes with existing data.

Look at first report.

Now I'm going to test params as query (generates for fields to filter) and search (A raw elasticsearch query string)

marla-singer · 2017-02-10T11:12:30Z

I found issue NREL/api-umbrella#275
As I understood prefix is not the api frontend_prefix

marla-singer · 2017-02-10T11:13:37Z

Parameters

prefix is 0/
01.02-09.02 by day

Result

{  
   "results":[  
      {  
         "depth":0,
         "path":"nightly.apinf.io:3002/",
         "terminal":false,
         "descendent_prefix":"1/nightly.apinf.io:3002/",
         "hits":534
      },
      {  
         "depth":0,
         "path":"nightly.apinf.io:3002",
         "terminal":true,
         "descendent_prefix":"1/nightly.apinf.io:3002",
         "hits":7
      }
   ],
   "hits_over_time":{  
      "cols":[  
         {  
            "id":"date",
            "label":"Date",
            "type":"datetime"
         },
         {  
            "id":"0/nightly.apinf.io:3002/",
            "label":"nightly.apinf.io:3002/",
            "type":"number"
         },
         {  
            "id":"0/nightly.apinf.io:3002",
            "label":"nightly.apinf.io:3002",
            "type":"number"
         }
      ],
      "rows":[  
         {  
            "c":[  
               {  
                  "v":1485907200000,
                  "f":"Wed, Feb 1, 2017"
               },
               {  
                  "v":6,
                  "f":"6"
               },
               {  
                  "v":0,
                  "f":"0"
               }
            ]
         },
         {  
            "c":[  
               {  
                  "v":1485993600000,
                  "f":"Thu, Feb 2, 2017"
               },
               {  
                  "v":30,
                  "f":"30"
               },
               {  
                  "v":1,
                  "f":"1"
               }
            ]
         },
         {  
            "c":[  
               {  
                  "v":1486080000000,
                  "f":"Fri, Feb 3, 2017"
               },
               {  
                  "v":13,
                  "f":"13"
               },
               {  
                  "v":0,
                  "f":"0"
               }
            ]
         },
         {  
            "c":[  
               {  
                  "v":1486166400000,
                  "f":"Sat, Feb 4, 2017"
               },
               {  
                  "v":0,
                  "f":"0"
               },
               {  
                  "v":0,
                  "f":"0"
               }
            ]
         },
         {  
            "c":[  
               {  
                  "v":1486252800000,
                  "f":"Sun, Feb 5, 2017"
               },
               {  
                  "v":0,
                  "f":"0"
               },
               {  
                  "v":0,
                  "f":"0"
               }
            ]
         },
         {  
            "c":[  
               {  
                  "v":1486339200000,
                  "f":"Mon, Feb 6, 2017"
               },
               {  
                  "v":25,
                  "f":"25"
               },
               {  
                  "v":1,
                  "f":"1"
               }
            ]
         },
         {  
            "c":[  
               {  
                  "v":1486425600000,
                  "f":"Tue, Feb 7, 2017"
               },
               {  
                  "v":214,
                  "f":"214"
               },
               {  
                  "v":2,
                  "f":"2"
               }
            ]
         },
         {  
            "c":[  
               {  
                  "v":1486512000000,
                  "f":"Wed, Feb 8, 2017"
               },
               {  
                  "v":53,
                  "f":"53"
               },
               {  
                  "v":1,
                  "f":"1"
               }
            ]
         },
         {  
            "c":[  
               {  
                  "v":1486598400000,
                  "f":"Thu, Feb 9, 2017"
               },
               {  
                  "v":193,
                  "f":"193"
               },
               {  
                  "v":2,
                  "f":"2"
               }
            ]
         }
      ]
   },
   "breadcrumbs":[  
      {  
         "crumb":"All Hosts",
         "prefix":"0/"
      }
   ]
}

brylie · 2017-02-10T11:41:48Z

Cool, thanks. Strange looking structure!

marla-singer · 2017-02-10T14:26:46Z

I researched another thing. Not so useful but I'd like to notice
The result array is a projection of table in the api umbrella Analytics page.
When the prefix is 0/ we've got the array

 "results":[  
      {  
         "depth":0,
         "path":"nightly.apinf.io:3002/",
         "terminal":false,
         "descendent_prefix":"1/nightly.apinf.io:3002/",
         "hits":534
      },
      {  
         "depth":0,
         "path":"nightly.apinf.io:3002",
         "terminal":true,
         "descendent_prefix":"1/nightly.apinf.io:3002",
         "hits":7
      }
   ]

and table

When the prefix is1/ then it's some projection of this table
Array:

 "results":[  
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/api-umbrella/",
         "terminal":false,
         "descendent_prefix":"2/nightly.apinf.io:3002/api-umbrella/",
         "hits":305
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/admin",
         "terminal":true,
         "descendent_prefix":"2/nightly.apinf.io:3002/admin",
         "hits":43
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/assets/",
         "terminal":false,
         "descendent_prefix":"2/nightly.apinf.io:3002/assets/",
         "hits":37
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/web-assets/",
         "terminal":false,
         "descendent_prefix":"2/nightly.apinf.io:3002/web-assets/",
         "hits":34
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/admins/",
         "terminal":false,
         "descendent_prefix":"2/nightly.apinf.io:3002/admins/",
         "hits":26
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/admin/",
         "terminal":false,
         "descendent_prefix":"2/nightly.apinf.io:3002/admin/",
         "hits":16
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/favicon.ico",
         "terminal":true,
         "descendent_prefix":"2/nightly.apinf.io:3002/favicon.ico",
         "hits":14
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/index.ru.js",
         "terminal":true,
         "descendent_prefix":"2/nightly.apinf.io:3002/index.ru.js",
         "hits":10
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/api/",
         "terminal":false,
         "descendent_prefix":"2/nightly.apinf.io:3002/api/",
         "hits":8
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/index.css",
         "terminal":true,
         "descendent_prefix":"2/nightly.apinf.io:3002/index.css",
         "hits":8
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/index0.css",
         "terminal":true,
         "descendent_prefix":"2/nightly.apinf.io:3002/index0.css",
         "hits":8
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/proxy_base_path/",
         "terminal":false,
         "descendent_prefix":"2/nightly.apinf.io:3002/proxy_base_path/",
         "hits":8
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/proxy_baset_path/",
         "terminal":false,
         "descendent_prefix":"2/nightly.apinf.io:3002/proxy_baset_path/",
         "hits":4
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/docs",
         "terminal":true,
         "descendent_prefix":"2/nightly.apinf.io:3002/docs",
         "hits":3
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/signup",
         "terminal":true,
         "descendent_prefix":"2/nightly.apinf.io:3002/signup",
         "hits":3
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/ftu5iurutrf1jzoyga7ty961yx3qbrbsjrucnrvy",
         "terminal":true,
         "descendent_prefix":"2/nightly.apinf.io:3002/ftu5iurutrf1jzoyga7ty961yx3qbrbsjrucnrvy",
         "hits":2
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/newfieldstest",
         "terminal":true,
         "descendent_prefix":"2/nightly.apinf.io:3002/newfieldstest",
         "hits":2
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/proxy_base_path",
         "terminal":true,
         "descendent_prefix":"2/nightly.apinf.io:3002/proxy_base_path",
         "hits":2
      },
      {  
         "depth":1,
         "path":"nightly.apinf.io:3002/docs/",
         "terminal":false,
         "descendent_prefix":"2/nightly.apinf.io:3002/docs/",
         "hits":1
      }
   ]

Table:

And yes, the hits result are different.

marla-singer · 2017-02-10T15:14:28Z

You can filter using the search URL parameter if you want to pass in a raw Lucene-style query.

Tested some of ElasticSearch fields:

request_at: couldn't find the right format.
response_status: a format is a Number
response_time: a format is a Number
request_ip: format is an IP as 10.0.0.12
request_ip_country: format is RU, EN and the same
request_path: a format is \/frontend_prefix\/ (note that with the Lucene syntax forward slashes need to be escaped with backslashes)
user_id : test it via my apiUmbrella user id

For using a few search parameters is specify this parameter twice. For example, search=response_status:200&search=request_ip:10.0.0.12

You can filter using the query URL parameter if you want to use the query-builder JSON format the default admin tool uses.

Couldn't make to correct work this parameter so I wrote in the issue: NREL/api-umbrella#275 (comment)

bajiat · 2017-02-16T11:20:31Z

@marla-singer Thanks for the research.

Closing the issue.

marla-singer · 2017-02-16T12:00:59Z

@bajiat I'd like to add couple moments which I found during research for 2115

Using drill-down request

We can use fetched data only for API Request Timeline chart
To fetch data for specific frontend-prefix there should use query URL parameter with body as

{ "condition":"AND",
  "rules":[{
  "id":"request_path",
  "field":"request_path",
  "type":"string",
  "input":"text",
  "operator":"begins_with",
  "value":"/foo/bar"
  }]
}

(or couser, body must has encoded format)

Using response

If API-umrella doesn't has data for specific parameters then "results" array will be empty
The "rows" array from "hits_over_time" object has the coordinates for line charts.
- The first object in "c" array matches as X axis and has the date in timestamp format ( "v":1483228800000) and ordinary format ("f":"Jan 1, 2017 - Jan 31, 2017").
- The second object in "c" array matches with Y axis and has the value for date

{
        "c":[
          {
            "v":1483228800000,
            "f":"Jan 1, 2017 - Jan 31, 2017"
          },
          {
            "v":35,
            "f":"35"
          }
        ]
      },

bajiat added this to the Sprint 36 milestone Feb 8, 2017

marla-singer self-assigned this Feb 8, 2017

bajiat added the ready label Feb 9, 2017

bajiat mentioned this issue Feb 13, 2017

Research options for proper API Umbrella API backend prefix and analytics handling #2115

Closed

3 tasks

bajiat closed this as completed Feb 16, 2017

bajiat removed the ready label Feb 16, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Research drill-down analytics endpoint #2090

Research drill-down analytics endpoint #2090

bajiat commented Feb 8, 2017 •

edited

marla-singer commented Feb 8, 2017 •

edited

brylie commented Feb 9, 2017 •

edited

marla-singer commented Feb 9, 2017

brylie commented Feb 9, 2017

brylie commented Feb 9, 2017

marla-singer commented Feb 9, 2017 •

edited

marla-singer commented Feb 9, 2017

brylie commented Feb 9, 2017

marla-singer commented Feb 9, 2017 •

edited

brylie commented Feb 9, 2017

brylie commented Feb 9, 2017

marla-singer commented Feb 9, 2017

brylie commented Feb 9, 2017

marla-singer commented Feb 10, 2017

marla-singer commented Feb 10, 2017

brylie commented Feb 10, 2017 •

edited

brylie commented Feb 10, 2017

marla-singer commented Feb 10, 2017 •

edited

marla-singer commented Feb 10, 2017

marla-singer commented Feb 10, 2017

brylie commented Feb 10, 2017

marla-singer commented Feb 10, 2017

marla-singer commented Feb 10, 2017 •

edited

bajiat commented Feb 16, 2017

marla-singer commented Feb 16, 2017

Research drill-down analytics endpoint #2090

Research drill-down analytics endpoint #2090

Comments

bajiat commented Feb 8, 2017 • edited

Definition of done

Research

Parameters

If you have 500 error

If you have 406 error

marla-singer commented Feb 8, 2017 • edited

brylie commented Feb 9, 2017 • edited

marla-singer commented Feb 9, 2017

brylie commented Feb 9, 2017

brylie commented Feb 9, 2017

marla-singer commented Feb 9, 2017 • edited

Parameters

marla-singer commented Feb 9, 2017

brylie commented Feb 9, 2017

marla-singer commented Feb 9, 2017 • edited

brylie commented Feb 9, 2017

brylie commented Feb 9, 2017

marla-singer commented Feb 9, 2017

brylie commented Feb 9, 2017

marla-singer commented Feb 10, 2017

Parameters

Result

marla-singer commented Feb 10, 2017

Parameter

Result

brylie commented Feb 10, 2017 • edited

brylie commented Feb 10, 2017

marla-singer commented Feb 10, 2017 • edited

marla-singer commented Feb 10, 2017

marla-singer commented Feb 10, 2017

Parameters

Result

brylie commented Feb 10, 2017

marla-singer commented Feb 10, 2017

marla-singer commented Feb 10, 2017 • edited

bajiat commented Feb 16, 2017

marla-singer commented Feb 16, 2017

Using drill-down request

Using response

bajiat commented Feb 8, 2017 •

edited

marla-singer commented Feb 8, 2017 •

edited

brylie commented Feb 9, 2017 •

edited

marla-singer commented Feb 9, 2017 •

edited

marla-singer commented Feb 9, 2017 •

edited

brylie commented Feb 10, 2017 •

edited

marla-singer commented Feb 10, 2017 •

edited

marla-singer commented Feb 10, 2017 •

edited