Support migrations in simplify

[jeromekelleher]

Simplify current ignores migration records. We need to update simplify so that migrations are also processed. This should be simple to do; we just use the node ID map to translate the node IDs in the old table into new node IDs. If the old node ID maps to -1, we leave the migration out.

[jeromekelleher]

#567 changed the behaviour to raise an error when attempting to simplify a set of tables with migrations.

[jeromekelleher]

When tackling this, add a check to the simulate_from_demography statistical check to see if the number of migrations is conserved when we have simulate_from. See #597 and #596 for background.

Another aspect of migration simplification is that we should take references to populations from migrations into account too.

[hyanwong]

Ping @Chris1221 who was interested in taking a look at this. I'm not sure how complex an issue this would be for someone unfamiliar with the codebase, but Jerome's comment at the top gives me hope that it's not too bad

[jeromekelleher]

Not the ideal place to start, as simplify is pretty hairy and multi-layered. But, would be a great way to learn the codebase and how the top-to-bottom development process works.

[Chris1221]

Perfect, thank you @hyanwong! Yes, probably won't start here, but I'll keep it on my radar. This feature would be particularly useful for the smcsmc conversion to tskit, which is why I was interested in it. But you're right @jeromekelleher, it is probably pretty hairy.

[jeromekelleher]

It's not so bad @Chris1221 --- I'd be happy to help you get up to speed. It'd be great to get smcsmc converted to tskit!

[Chris1221]

Yea I think so too. I was talking to @hyanwong a little about it, but we've implemented a very hacky conversion to the basic tables in text format -- it needs quite a bit of refining but the major issue was that we treat migration events as nodes, which is different than tskit. I'm going to look at the tsconvert work done so far and see if I can do it in a better way. I think it's pretty crucial to get everything in comparable formats, then we can compare apples to apples for the inference methods.

[jeromekelleher]

I think it's pretty crucial to get everything in comparable formats, then we can compare apples to apples for the inference methods.

Amen brother! I think it'd be well worth while spending a few days with us at the BDI then, I'm sure we can iron out the details between us.

[Chris1221]

That sounds great! I'm going home for Christmas soon, so realistically I don't think I'll have enough time in the next week or so, but we should try to find some time early in the new year. It's pretty high on my to-do list for smcsmc. Having it tied into the tskit ecosystem will also make it more easy for users to approach the output (and better integrated into stdpopsim).

[Chris1221]

I'm going to take a look at this in the next few days. If anyone else has been working on it, let me know!

[jeromekelleher]

Nope, this is all yours @Chris1221! I'm happy to have a quick chat to get you up to speed - do send me an email/slack if you have any questions.

[hyanwong]

This seems to have dropped off the radar. It would be nice to have for one of the tuts. Any takers to finish it off? @Chris1221 I assume you never found time to do it?

[Chris1221]

So sorry @hyanwong, I started working on it but time in my PhD is running out so I never got around to finishing it. My apologies. :(

[benjeffery]

I've added this to C 1.0 as we should think if it has any effect on the simplify API. (I don't think so but worth some more thought)

[benjeffery]

Moving to C upcoming as doesn't have an impact on the public API.

[hyanwong]

Just to record that we should remember to update the doc strings for the TSK_SIMPLIFY_NO_UPDATE_SAMPLE_FLAGS (C) and update_sample_flags (Python) API.