Build endpoints to access API metadata

[arowla]

Requested fields to get unique values for are:

• data_source
• notice_type

Unique notice_types can be gotten via this aggregation:

  "aggs": { "notice_types": { "terms": {
    "field": "notice_type",
    "size": 10
  }  }  }

[arowla]

To this I'd like to add date aggregation, to show populated dates or ranges for the data that we have. Could be a separate endpoint or parameter, but the work should be similar to get here.

[adelevie]

Should the data_sources be hardcoded into the express route, or is there a (preferred) dynamic way to do this?

[arowla]

Not quite sure I understand the question. Could you give an example of the route (and perhaps a quick mockup of what it might return) that you have in mind?

[adelevie]

To achieve a "a new API endpoint to hit and get all available data_sources":

api/app.js (approximately):

app.get('/v0/data_sources', function(req, res) {
  res.json({
    results: [
       'fbo.gov', 'grants.gov', 'dodsbir.net', 'bids.state.gov'
    ]
  });
});

Is this fine? or is there a dynamic way to populate the results array?

[arowla]

Gotcha. These should be dynamically populated. See the Elasticsearch sample aggregation code at the top of the Issue.

Also we may want to name the route a bit differently, but I will need to think on it and we should discuss with the group.

[adelevie]

Great, very clarifying. Is there any code we have that already does something similar? I'm guessing that the aggregation code gets passed to (es) client.search()? (still getting used to ES w/ Node)

[arowla]

Yes, JSON can be passed directly to client.search({body: [insert query here]}), but since we're using Elastic.js (API docs) as a DSL to help build up queries elsewhere in the API, you should first look into whether that can do what we need without too much trouble, before resorting to raw JSON.

[adelevie]

Looks like this is the answer. * studies docs * ...

[adelevie]

So far

app.get('/v0/data_sources', function(req, res) {
  client.search({
    index: config.elasticsearch.index,
    type: 'opp',
    body: ejs.TermsAggregation({name: 'data_source'})
  }, function(err, body) {
    res.json(body)
  });
});

outputs:

{"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[zTzPNIcxTuSlaiGgZ2ncmg][fbopen0][0]: RemoteTransportException[[Nebulos][inet[/172.23.198.98:9300]][search/phase/query]]; nested: SearchParseException[[fbopen0][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"[object Object]":{"terms":{}}}]]]; nested: SearchParseException[[fbopen0][0]: from[-1],size[-1]: Parse Failure [No parser for element [[object Object]]]]; }{[d56lqUq7S56wszWZiDuyYw][fbopen0][1]: SearchParseException[[fbopen0][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"[object Object]":{"terms":{}}}]]]; nested: SearchParseException[[fbopen0][1]: from[-1],size[-1]: Parse Failure [No parser for element [[object Object]]]]; }{[d56lqUq7S56wszWZiDuyYw][fbopen0][4]: SearchParseException[[fbopen0][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"[object Object]":{"terms":{}}}]]]; nested: SearchParseException[[fbopen0][4]: from[-1],size[-1]: Parse Failure [No parser for element [[object Object]]]]; }{[zTzPNIcxTuSlaiGgZ2ncmg][fbopen0][2]: RemoteTransportException[[Nebulos][inet[/172.23.198.98:9300]][search/phase/query]]; nested: SearchParseException[[fbopen0][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"[object Object]":{"terms":{}}}]]]; nested: SearchParseException[[fbopen0][2]: from[-1],size[-1]: Parse Failure [No parser for element [[object Object]]]]; }{[zTzPNIcxTuSlaiGgZ2ncmg][fbopen0][3]: RemoteTransportException[[Nebulos][inet[/172.23.198.98:9300]][search/phase/query]]; nested: SearchParseException[[fbopen0][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"[object Object]":{"terms":{}}}]]]; nested: SearchParseException[[fbopen0][3]: from[-1],size[-1]: Parse Failure [No parser for element [[object Object]]]]; }]","status":400}

[adelevie]

#151 implements the data_source option