Check how to get data about API key from Elastic

[bajiat]

We are interested to have analytics about the API key usage per API. This would for example be a first step towards API monetization.
API key is available in the API Umbrella drilldown dashboard, but does not appear to be available in the drilldown analytics results.

Check if this assumption is true or whether we are omitting API key in the requests we are making at the moment.

Definition of done

• Either make a small experiment or check from Elastic code.
• Document the result in this issue.

[bajiat]

@brylie Can you investigate this?

[bajiat]

@brylie Would it be ok, if @NNN would help you in this task?

[55]

Answering question stated in the issue - yes, it's possible.
Thanks to @frenchbread for help, we came up with this method to kinda demonstrate this.
To test it yourself, simply add it to /elasticsearch/server/methods.js

  getAPIKeys () {
    const esClient = new ElasticSearch.Client({ host: 'http://nightly.apinf.io:14002' });
    const params = {
      size: 10,
      body: {
        sort: [
          {
            request_at: {
              order: 'desc',
            },
          },
        ],
      },
    };
    esClient.search(params).then((res) => {
      console.log('Results:');
      console.log(res.hits.hits);
    }, (err) => {
      // Throw an error
      throw new Meteor.Error(err.message);
    });
  },

This structure for each call you get back:

I20170315-15:01:41.040(2)?   { _index: 'api-umbrella-logs-v1-2017-03',
I20170315-15:01:41.041(2)?     _type: 'log',
I20170315-15:01:41.041(2)?     _id: 'aqq7k887q9339spvrih0',
I20170315-15:01:41.041(2)?     _score: null,
I20170315-15:01:41.042(2)?     _source: 
I20170315-15:01:41.042(2)?      { api_key: '[actual user's API key]',
I20170315-15:01:41.049(2)?        backend_response_time: 161,
I20170315-15:01:41.050(2)?        gatekeeper_denied_code: null,
I20170315-15:01:41.050(2)?        internal_gatekeeper_time: 0,
I20170315-15:01:41.051(2)?        proxy_overhead: 7,
I20170315-15:01:41.051(2)?        request_accept: '*/*',
I20170315-15:01:41.051(2)?        request_accept_encoding: 'gzip, deflate, sdch, br',
I20170315-15:01:41.052(2)?        request_at: 1489580360900,
I20170315-15:01:41.052(2)?        request_basic_auth_username: null,
I20170315-15:01:41.053(2)?        request_connection: 'keep-alive',
I20170315-15:01:41.053(2)?        request_content_type: null,
I20170315-15:01:41.053(2)?        request_hierarchy: [Object],
I20170315-15:01:41.054(2)?        request_host: 'nightly.apinf.io:3002',
I20170315-15:01:41.054(2)?        request_ip: '109.204.245.84',
I20170315-15:01:41.056(2)?        request_ip_city: 'Tampere',
I20170315-15:01:41.057(2)?        request_ip_country: 'FI',
I20170315-15:01:41.057(2)?        request_ip_location: [Object],
I20170315-15:01:41.058(2)?        request_ip_region: '15',
I20170315-15:01:41.058(2)?        request_method: 'GET',
I20170315-15:01:41.059(2)?        request_origin: null,
I20170315-15:01:41.060(2)?        request_path: '/api-umbrella/v1/analytics/drilldown.json',
I20170315-15:01:41.060(2)?        request_query: [Object],
I20170315-15:01:41.060(2)?        request_referer: 'https://nightly.apinf.io:3002/admin/',
I20170315-15:01:41.060(2)?        request_scheme: 'https',
I20170315-15:01:41.061(2)?        request_size: 1326,
I20170315-15:01:41.061(2)?        request_url: 'https://nightly.apinf.io:3002/api-umbrella/v1/analytics/drilldown.json?tz=Europe%2FHelsinki&search=&start_at=2017-02-14&end_at=2017-03-15&query=%7B%22condition%22%3A%22AND%22%2C%22rules%22%3A%5B%7B%22field%22%3A%22gatekeeper_denied_code%22%2C%22id%22%3A%22gatekeeper_denied_code%22%2C%22input%22%3A%22select%22%2C%22operator%22%3A%22is_null%22%2C%22type%22%3A%22string%22%2C%22value%22%3Anull%7D%5D%7D&interval=day&prefix=0%2F&beta_analytics=false',
I20170315-15:01:41.061(2)?        request_user_agent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
I20170315-15:01:41.062(2)?        request_user_agent_family: 'Chrome',
I20170315-15:01:41.064(2)?        request_user_agent_type: 'Browser',
I20170315-15:01:41.065(2)?        response_age: 0,
I20170315-15:01:41.096(2)?        response_cache: 'MISS',
I20170315-15:01:41.097(2)?        response_content_encoding: 'gzip',
I20170315-15:01:41.097(2)?        response_content_length: null,
I20170315-15:01:41.099(2)?        response_content_type: 'application/json; charset=utf-8',
I20170315-15:01:41.100(2)?        response_server: 'openresty',
I20170315-15:01:41.100(2)?        response_size: 1439,
I20170315-15:01:41.101(2)?        response_status: 200,
I20170315-15:01:41.101(2)?        response_time: 168,
I20170315-15:01:41.102(2)?        response_transfer_encoding: 'chunked',
I20170315-15:01:41.102(2)?        user_email: 'web.admin.ajax@internal.apiumbrella',
I20170315-15:01:41.102(2)?        user_id: '70a3c56f-5764-4a69-9b2a-c87aff573735',
I20170315-15:01:41.103(2)?        user_registration_source: 'seed' },
I20170315-15:01:41.103(2)?     sort: [ 1489580360900 ] } ]

[brylie]

After discussing with @NNN, we agreed to create a new, or update an existing, document in apinf/docs with details about the Elasticsearch data structure. That way, we can use the data structure to guide the design of our dashboard charts, and determine the best data to meet end-user needs.