New way of generating replication diff

[joto]

This was asked for in #154 . I implemented it in https://github.com/openstreetmap/osmdbt and it is basically finished.

To be resolved:

• Maybe change some internals
• Installable package (see issue Ubuntu 20.20 PPA build) Is this really what we need? Please chime in on that issue with your wishes and ideas.

To test we need a machine that is sufficiently similar to the production DB machine where we can install the software and run it through its paces with more or less real work data.

Once that works we can install it on the production machine but keep the old diff generation to make sure a) that it doesn't put undue load on the machine or has any other side effects and b) that the diffs it generates are the same as the old method. Then we can switch to the new method keeping the old one around for a while.

Operations people: Do you need anything else from me? If you see anything missing in the software or documentation feel free to open issues on osmdbt or put it in this issue here. Can your provide a machine where I can do some testing?

[tomhughes]

Well obviously I can build from source but an installable package certainly makes life easier.

[mmd-osm]

I just finished a first round of testing and code review, and wanted to share a few impressions.

Given that there's no formal specification on how the process is supposed to work in detail, I tested osmdbt alongside the existing osmosis, and tried to identify differences in functionality/performance/xml output... Out of the 8 open issues about half of them discuss some deviations from what we have right now.

I would propose to leave ample time for testing, given how complex the overall process is. Some of the issues become only obvious in case there's concurrent changeset uploads going on. Performance topics will become even more relevant when moving the osmdbt-* tools away from the database server to another machine with additional network latency.

Over time, I learned more about how osmosis works, but that's probably still only scratching the surface. I can only recommend to continue reviewing relevant parts of that coding, and think if assumptions there are also relevant for osmdbt.

What I couldn't test are scenarios like data centre fail over. I've read that there are some additional steps to be taken, but there's no way for me to try this out.

[pnorman]

Operations people: Do you need anything else from me? If you see anything missing in the software or documentation feel free to open issues on osmdbt or put it in this issue here. Can your provide a machine where I can do some testing?

Just reopening this question - it's not clear what the next steps are.

[tomhughes]

I did review this, and a related ticket, last week as I thought they were waiting on me but I reached the conclusion they weren't so left them pending more information from @joto or @mmd-osm on what they are looking for.

[mmd-osm]

I've been somewhat involved in local testing and code review, as this stuff is complex and an additional pair of eyeballs doesn't hurt in general. Some of the issues I raised have already been addressed by coding fixes, a few others are still pending.

I don't know exactly what joto has planned in terms of test cases in a production-like environment. Obviously, we would need to run the old and new world side-by-side at some point and make sure results are eventually identical. Most likely we won't be able to do a 1:1 file comparison, as the extraction process execution timestamps and extraction methods vary slightly.

[joto]

I believe I have addressed all issues. osmdbt is now ready for some more real-world testing.

[pnorman]

Now in production. Any problems can be ticked as new issues.