Is struct code sufficiently rich for SLiM and fwdpy11?

[jeromekelleher]

The struct codec was merged in #601, and there's a number of ways we could extend it. We'd like to ship the initial version in 0.3.0 that is sufficiently rich to encode metadata from SLiM and fwdpy11. Other features can be considered later.

@bhaller/@petrelharp and @molpopgen, would it be possible for you to try out the current struct codec, and see if it'll work for you? The docs need some work, so if you see places for improvement, please shout out.

https://tskit.readthedocs.io/en/latest/metadata.html

[benjeffery]

Please get in touch if I can help at all, the issue for docs suggestions is #603

[benjeffery]

Skeleton schemas for SLiM are at https://github.com/tskit-dev/tskit/blob/master/python/tests/test_metadata.py#L1247 but should be augmented with descriptions etc.

[molpopgen]

This is working great! I have notes on the docs as I'm reading them. Will add them to #603 once they're ready.

[benjeffery]

I'm not sure @bhaller got notified in the PR description due to the "/" so making sure here!

[bhaller]

I'm not sure @bhaller got notified in the PR description due to the "/" so making sure here!

I wasn't notified, but now I am, thanks! I'll have a look at this soon. I'm probably too Python-incompetent to try it out there, but I'll have a read through the docs, at least.

[molpopgen]

due to the "/" so making sure here!

So weird!

[bhaller]

Reviewed the doc at #603. As far as I can tell it should work for SLiM metadata just fine, but since I'm Python-illiterate it would be good for @petrelharp to have a look over it. Speaking of @petrelharp and SLiM: will pyslim's way of interpreting/vending metadata remain the same, at least from the user's perspective, or is it going to change because of these tskit changes?

[jeromekelleher]

Thanks for the input @bhaller, this is very helpful. My expectation re pyslim is that nothing will change from a user's perspective, but pyslim will be simplified quite a bit as most of the decoding will happen automatically from tskit. (There will still be a little bit of work to do if pyslim decodes to objects rather than dicts - we can't do that yet in tskit.)

[petrelharp]

It looks great; actually testing this out will mean I have to get around to updating the version of tskit that SLiM carries around with it (which would be good to do!). I'll try to get to that this week, but bug me if this is holding you up.

[bhaller]

I can take a PR any time, @petrelharp. I wasn't planning on actually rolling a new release of SLiM for a fairly long time; will the existing SLiM and pyslim continue to work when this new tskit rolls out, or do we need to plan a synchronized release so that nobody gets broken?

[petrelharp]

will the existing SLiM and pyslim continue to work when this new tskit rolls out

We'll continue to be backwards-compatible; no worries there.

[jeromekelleher]

It looks great; actually testing this out will mean I have to get around to updating the version of tskit that SLiM carries around with it (which would be good to do!).

Would you mind waiting until C 0.99.3 is released @petrelharp? This is the next milestone, which we're hoping to hit in < 2 weeks.

https://github.com/tskit-dev/tskit/milestone/8

Changes to the schema, etc, will happen at the Python level so there'll be room to sort things out before Python 0.3.0 is shipped (https://github.com/tskit-dev/tskit/milestone/7).

[petrelharp]

Would you mind waiting until C 0.99.3 is released @petrelharp?

Sounds good. But: I can actually test out the schemas at the python level now; I've had a start at it already. It's just the C code that we won't be able to test yet.

[petrelharp]

@benjeffery, I need some help - in population metadata we actually encode how long the trailing array is. How does one specify this? Like, how's the alternative to noLengthEncodingExhaustBuffer work?