Logs Sample Data

[alexfrancoeur]

This is one of the data sets we'd like to add for sample data. This is my first take on the data set itself so open to any feedback.

We're using makelogs as the basis here but want to tweak the URL's and requests to be more Elastic-y.

Open to feedback on the dataset, dashboards, etc.

Also note that I did not add any tests. There were a bunch of sample flight data, I'm not sure if they're necessary for other sample data sets. I'll look for @nreese here for guidance and can add some in if necessary.

Screenshots

Sample document

{
  "_index": "kibana_sample_data_logs",
  "_type": "_doc",
  "_id": "hikLbGUBoflyAT6o14Fv",
  "_version": 1,
  "_score": null,
  "_source": {
    "referer": "http://www.elastic-elastic-elastic.com/success/kenneth-ham",
    "clientip": "109.157.83.197",
    "response": "200",
    "tags": [
      "success",
      "info"
    ],
    "message": "109.157.83.197 - - [2018-08-03T13:07:45.046Z] \"GET /apm-server/apm-server-6.3.2-windows-x86.zip HTTP/1.1\" 200 5782 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1\"",
    "request": "/apm-server/apm-server-6.3.2-windows-x86.zip",
    "geo": {
      "srcdest": "IN:AO",
      "src": "IN",
      "dest": "AO",
      "coordinates": {
        "lat": 39.21837556,
        "lon": -75.59642667
      }
    },
    "phpmemory": null,
    "extension": "zip",
    "index": "kibana_sample_data_logs",
    "url": "https://artifacts.elastic.co/downloads/apm-server/apm-server-6.3.2-windows-x86.zip",
    "memory": null,
    "host": "artifacts.elastic.co",
    "utc_time": "2018-08-03T13:07:45.046Z",
    "machine": {
      "ram": 8589934592,
      "os": "win 7"
    },
    "agent": "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1",
    "ip": "109.157.83.197",
    "bytes": 5782,
    "timestamp": "2018-08-24T13:07:45.046Z"
  },
  "fields": {
    "utc_time": [
      "2018-08-03T13:07:45.046Z"
    ],
    "hour_of_day": [
      13
    ],
    "timestamp": [
      "2018-08-24T13:07:45.046Z"
    ]
  },
  "sort": [
    1535116065046
  ]
}

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[jamiesmith]

Nice job! eCommerce next?

[alexfrancoeur]

Updated PR so that it now has a working index pattern. New screenshot and sample doc above.

@jamiesmith @asawariS would love some feedback on the visualization types shown here, metrics to track, etc. Any thoughts on a color schema would be great as well. I'd still like to add annotations to the TSVB chart and better metrics in the table. There needs to be some cleanup, but this is actually a dashboard I use for demo's pretty frequently. More to show off functionality, not necessarily tell a story.

@jamiesmith I know we talked about adding "Elasticy" hosts, requests, etc. Any input here from PMM would be great, I'd really like to make this data fun. I know that you were also thinking of anomalies to add in. Note that I made a few changes in the Sample Data Clean Up doc and removed @ from a few of the field names. I also updated the index to be kibana_sample_data_logs.

@nreese open question here around tests. Do I need to add them in if there are already some for flight sample data? I'm guessing I do, but thought I'd ask before I write any 😄

[nreese]

open question here around tests. Do I need to add them in if there are already some for flight sample data? I'm guessing I do, but thought I'd ask before I write any

I think we should add a test that installs the sample data set, opens the dashboard and makes sure the expected number of panels are opened, and then uninstalls the sample data set. I can work on this and give you a PR with the tests.

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[alexfrancoeur]

@nreese I should be able to add them, will ping with questions. Thanks for the input!

[alexfrancoeur]

@nreese I'm assuming I can pretty much copy what you did with flights here

[nreese]

@alexfrancoeur yes, just don't copy the skipped tests. Simply opening the dashboard and verifying the panel count is good enough for now.

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[alexfrancoeur]

Still iterating on the data set a bit with @jamiesmith, updated screenshot and doc in the summary. We're close!

[alexfrancoeur]

@nreese how do those tests look?

[nreese]

@alexfrancoeur tests look great. Nice work adding another data set

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[nreese]

This test should not be skipped. It is failing because PageObjects.home.launchSampleDataSet('logs') needs to be run from from the kibana home page tutorial_directory/sampleData. You had it right by changing after to afterEach.

[alexfrancoeur]

:doh:

[nreese]

Remove the check for the toast notifications. Those are flaky.

[alexfrancoeur]

++ removed, testing locally (because I just figured out how to do that 😄 )

[nreese]

Remove the check for the toast notifications. Those are flaky.

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[nreese]

Looks great. Nice to have another sample data set with a great looking dashboard

lgtm
code review, installed sample data set and viewed dashboard.

[alexfrancoeur]

nice! thank you @nreese 😄

[AlonaNadler]

@alexfrancoeur @nreese Great job on the sample web logs!!
Few suggestions:

• Can you add trend logs to the TSVB table, it's a features users don't know exists, even if it won't make sense it demonstrates a capability we have
  

• Why do you have - in the custom label, it looks like there is an error when it's presented in the dashboard

• I prefer to have panels titles, users are dropped to this dashboard, any additional context can help, especially when it comes to logs when your metrics might inspire new ideas in them
• Change the prefix you give to all objects from [logs] to [sample logs] since it might get mixed with any future log users will ingest
• Change the title of Sample log data to Sample web logs these 2 panels looks identical, it will be easier to differentiate when just quickly scanning them
  

[nreese]

@alexfrancoeur @nreese Great job on the sample web logs!!

It was all @alexfrancoeur.

[alexfrancoeur]

@nreese stop being modest - I copied all your code 😄

@AlonaNadler great feedback! Updated and committed. Once green, I'll merge

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[nreese]

jenkins, test this

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request