metadata documentation

[petrelharp]

In getting pyslim to use the new metadata machinery I'm coming across some things to be documented, so I'm opening this to make a list of those (and to check if I'm doing things right):

• to check if metadata is being decoded, you can check if len(metadata_schema.schema) > 0 (but there's probably a better way)

• the C functions to set metadata schemas want valid json strings (and note that this looks almost like a python dictionary but not quite: e.g. true not True)

• the top-level (tree sequence) metadata can be directly assigned to, e.g. tables.metadata = { 'a': 'valid_dict' } (and, does it get validated?)

• if you change the metadata schema, it does not (?) go through and validate all your metadata (even the top-level metadata)

• different things can co-exist in the top-level metadata, so client code should usually create their own top-level key and put stuff under that, leaving alone whatever is there already in both the schema and the metadata itself

• to allow metadata to be "this thing or else nothing", set "type": ["object", "null"] (and, carry on with properties, they'll only apply if the metadata is not null)

• to add keys to the top-level metadata schema, (a) get the dictionary that is the metadata-schema.schema property, e.g., schema = tables.metadata_schema.schema, and then edit it, e.g. schema['objects']['properties']['SLiM'] = { some new dict }, and then reassign it (e.g. tables.metadata_schema = tskit.MetadataSchema(schema)) Beware that doing some of these operations on a tree sequence, or even a table collection, silently does nothing)

• things are a bit confusing because (IIUC) you can modify the top-level metadata in place, as a dict, but you cannot modify the top-level metadata_schema.schema in place, even though it's also a dict

Apologies for the unordered brain dump.

[petrelharp]

Two more things for the documentation:

• a list of required entries in provenance (or perhaps that's already in the docs? I didn't see it on a quick skim)

• an example function to get the required environment (eg a copy of msprime.get_environment())

[benjeffery]

Thanks for these @petrelharp - I'll roll them into the doc improvements at #741

[petrelharp]

• To change the top-level metadata and metadata schema, I think you need to (a) set the schema to the new schema and use the fact that it doesn't re-validate when doing this, and then (b) set the new metadata.

[petrelharp]

• Metadata schema equality depends on ordering and whitespace handling of the text representation (see metadata schema equality and hence table equality is not well-defined #763).

[petrelharp]

Maybe this is too minor to mention, but:

• numpy integers don't validate as integers:

>>> jsonschema.validate(3, {"type": "integer"})
# but
>>> jsonschema.validate(np.int32(3), {"type": "integer"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/peter/.local/lib/python3.7/site-packages/jsonschema/validators.py", line 899, in validate
    raise error
jsonschema.exceptions.ValidationError: 3 is not of type 'integer'

Failed validating 'type' in schema:
    {'type': 'integer'}

On instance:
    3

... which I found since that's what you get if you use the attr.asdict() method to get your dicts.

[benjeffery]

Unless we also fix the codecs to encode numpy ints that is probably the right behaviour:

>>> json.dumps(np.int32(3))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type int32 is not JSON serializable

[jeromekelleher]

Yeah, this is a tedious but common problem. I don't think we want to change from the default Python encoder, tbh.

[petrelharp]

Since this is a "random things about metadata" issue here, let me also report: the tskit parsing of metadata is a bit slower, but not terribly, than what I had in pyslim. This is good! There's some extra things that tskit's metadata is doing to be more general, and do error checking, etcetera, so I thought I'd check. Here's the results: using a tree sequence with 29396 nodes, 140975 mutations, and 10000 individuals, and this script:

#!/bin/bash

SETUP="
import tskit, pyslim
ts = tskit.load('benchmark/test.trees')
tables = ts.tables
tables.individuals.metadata_schema = tskit.MetadataSchema(None)
tables.nodes.metadata_schema = tskit.MetadataSchema(None)
tables.mutations.metadata_schema = tskit.MetadataSchema(None)
nts = tables.tree_sequence()

def tsk_fn(ts):
    pid = [ind.metadata['pedigree_id'] for ind in ts.individuals()]
    sid = [n.metadata['slim_id'] for n in ts.nodes()]
    s = [m.metadata['mutation_list'][0]['selection_coeff'] for m in ts.mutations()]
    return 0

def py_fn(ts):
    pid = [pyslim.decode_individual(ind.metadata).pedigree_id for ind in nts.individuals()]
    sid = [pyslim.decode_node(n.metadata).slim_id for n in nts.nodes()]
    s = [pyslim.decode_mutation(m.metadata)[0].selection_coeff for m in nts.mutations()]
    return 0
"

echo "tskit:"
python3 -m timeit -s "$SETUP" "tsk_fn(ts)"

echo "pyslim:"
python3 -m timeit -s "$SETUP" "py_fn(nts)"

the difference is about 50%:

tskit:
1 loop, best of 5: 1.58 sec per loop
pyslim:
1 loop, best of 5: 925 msec per loop

... which seems reasonable.

[jeromekelleher]

That's excellent, thanks for checking this out @petrelharp! I'm sure we can speed this up a bit over time, but this perf is certainly good enough to start with. Well done @benjeffery !

[benjeffery]

Yep perf is not as good as it could be, thanks for the data point @petrelharp. #614 suggests the low-hanging fruit.

[jeromekelleher]

Can we close this @benjeffery?

[benjeffery]

Closed in #741