Install dependencies with opam install --deps-only .
Build with make
esgg
takes as an input an ES mapping (schema description)
and actual query (with syntax for variables, described below). Often additional information is needed to map ES fields into proper OCaml types,
this is achieved by attaching _meta
annotation object to the affected field (ES only supports _meta
at root level, so these annotations make it
impossible to store extended mapping back into ES which is a pity), as follows:
"counts": {
"_meta": {
"optional": true
},
"properties": {
"hash": {
"type": "long",
"_meta": { "repr": "int64" }
},
"value": {
"type": "long"
}
}
},
Supported _meta
attributes:
{"list":true}
- property is an array (mapped tolist
){"list":"sometimes"}
- property is either an array or single element (mapped to json with custom ocaml module wrap that will need to be provided in scope){"optional":<true|false>}
- property may be missing (mapped tooption
){"ignore":true}
- skip property altogether{"fields_default_optional":true}
- any subfield may be missing (can be overriden by per-fieldoptional:false
){"repr":"int64"}
- override EStype
, currently the only possible value is"int64"
to ensure no bits are lost (by defaultlong
is mapped to OCamlint
)
Generated code allows to use application types for any fields. This is achieved by referencing specific type for each field in generated
code, instead of the primitive type from the mapping, allowing consumer of the code to map it onto custom type etc. For example the field
hash
in example above will have type Counts.Hash.t
in generated code. In order to compile the generated code this type must be present
in scope and mapped to something useful. Default mapping (which just maps everything to corresponding primitive types) can be generated
with esgg reflect <mapping name> <mapping.json>
, e.g.:
esgg reflect hello_world src/mappings/hello_world.json >> src/mapping.ml
will generate the following, which should be edited manually as needed, e.g. by making Hash
a module with an abstract type
module Counts = struct
module Hash = Id_(Int64_)
module Value = Id_(Long_)
end
Syntax for variables in template json files is as follows:
$var
for regular required variable$var?
for optional variable (minimal surrounding scope is conditionally expunged)- full form
$(var:hint)
wherehint
can be eitherlist
orlist?
currently
To reuse shared definitions using the -shared <file.atd>
option, the atd
file must have the <esgg from="...">
annotation at the top of the file.
The value of the annotation must correspond to the OCaml module containing the shared definitions.
Example:
# file.atd
<esgg from="Your_ocaml_module_name">
...atd type definitions...
TODO document what is supported
Some notes follow:
The following aggregation types are supported:
- named
- anonymous
- dynamic (i.e. a variable)
- partial dynamic (i.e. containing variables)
- other_bucket and other_bucket_key
- other_bucket with anonymous filters (ignored, user is responsible to treat last element of result specially)
Dynamic (defined at runtime) filters are supported, as follows { "filters": { "filters": $x } }
.
In this case corresponding part of output will be quite untyped. $x
is assumed to be a dictionary and result will be represented with
dictionaries. For anonymous filters (ie array of filters) use $(x:list)
.
key_as_string
is returned in output only when format
is
explicitly specified,
to discourage fragile code.
Keyed aggregation expects explicit key
for each range. from
/to
fields in response are not extracted.
same as for range aggregation
- min
- max
- avg
- sum
- value_count
- cardinality
- terms
- significant_terms
- significant_text
- histogram
- top_hits
- nested
- reverse_nested
- bucket_sort
Specifying aggregation as variable ($var
) will lead to an untyped json in place of aggregation output, this can be used as temporary
workaround for unsupported aggregation types or for truly dynamic usecase (aggregation built at run-time).
Scripts are opaque, ie no type information is extracted and result is json.
- exclude and include
- wildcards
- dynamic (i.e. a variable) NB not implemented for get and mget
make test
runs regression tests in test/ verifying
that input and output atd generated from query stays unchanged.
Once there is an expected change in generated query - it should be committed.
Tests are easy to add and fast to run.
TODO tests to verify that:
- code generated for query application of input variables does actually compile and produce correct query when run
- atd description of output (generated from query) can indeed unserialize ES output from that actual query
Copyright (c) 2018 Ahrefs github@ahrefs.com
This project is distributed under the terms of GPL Version 2. See LICENSE file for full license text.
NB the output of esgg, i.e. the generated code, is all yours of course :)
2019-10-29