Exploring wal2json

[mmd-osm]

As we were having lots of discussions on the plugin topic recently, and more challenges ahead of us when it comes to packaging the plugin for different Postgresql versions, I started exploring the option to use an existing plugin, that is part of the Postgresql repository. wal2json seemed like a good option for this exercise:

This extension is supported on those platforms that PostgreSQL is. The installation steps depend on your operating system. PostgreSQL yum repository and PostgreSQL apt repository provide wal2json packages.

There's one downside I see at the moment: they still have to figure out the same issue with excessive BEGIN/COMMIT entries, we've already solved (see eulerto/wal2json#106). Apart from that, I believe this plugin would be feature complete for our requirements.

I prepared a small demo in https://github.com/mmd-osm/osmdbt/tree/json as a proof of concept. It moves the mapping that was previously done in osm-plugin to a JSON parser in osmdbt-get-log.

In the long run this might be cheaper for us, as can leave all those Postgresql plugins details to others.

By the way, https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html also lists wal2json as supported plugin, albeit in a somewhat older version. So this all seems like production ready code.

[joto]

Generating and parsing the verbose JSON seems like a huge and unnecessary overhead to me. We are moving quite a bit of data here and should avoid any overhead we can.

Lets move ahead with what we have now instead of getting sidetracked by shinier things somewhere else.

[mmd-osm]

Just for the record, osmdbt-get-log w/ catch-up for 1 mio changes takes about 41s on wal2json, and 17s on osm-logical. Both values are still well within our worst case volume for minutely replication (which would be around 500k changes). So yes, there's overhead (which was expected), but it's not totally out of proportion.