Importing Data

okay edited this page Jun 13, 2017 · 3 revisions

Importing Data

Continuous Data

HTTP

the simplest way of importing data is to send it over HTTP. Notice that samples field is a string representing an array of dictionaries.

$.post("http://your.snorkel.host/data/import", {
  dataset: "mysql",
  subset: "slow_queries",
  samples: JSON.stringify([ // that's right, you send a JSON stringified array of samples.
    {
      integer: {
        'query_duration': 500,
        'query_count' : 10,
        // this sample is actually emitted on 1 / 1000 requests,
        // so we give it a 'weight' of 1000 or sample_rate of 1000
        'weight' : 1000, 
      },
      string: {
        'table' : 'a_mysql_table',
        'query_str' : 'select * from <TABLE>;',
        'host' : 'mysql001'
      },
      set: {
        'flags' : [
          'foo', 'bar', 'baz'
        ]
      },
    }
  ])});

UDP

If HTTP protocol is too expensive, UDP is also viable. in snorkel's config, a UDP port can be supplied that snorkel will listen for JSON samples on.

TODO: provide example UDP logging

MQ

TODO: explanation and method

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.