GME & the future?

[roynasser]

Hi,

with the announcement that rivers are being discontinued in 2.0, and are discouraged from 1.5 on, will this affect Go-Mysql-Elasticsearch?

Will you be pursuing other alternatives? We are looking into ways of syncing using the recommended method which best suits us (in our case we dont want to start from zero, so looking at Logstash)... Are you/will you be considering GME as an input to logstash perhaps? or is there too much overlap in code alreadt in your opinion?

Logstash doesnt have a mysql input layer, and probably wont ever have something binlog capable... on the otherhand, logstash is already bulking requests and inserting them using the correct apis on ES, so a bridge between GME and logstash could be a natural progression?

I'd like to hear the dev's opinions...

[siddontang]

Hi @RVN-BR

I know that ES will discard river feature, but this doesn't affect GME, because I don't use river feature at all.

GME is a service acting as a MySQL slave to sync MySQL binlog and then update the data changes into ES with ES http bulk API.

Btw, if you want to sync MySQL binlog and write the changes into logstash, you can use my another project github.com/siddontang/go-mysql, you can use the canal package to sync binlog and write the json format changes in a place which logstash can get.

[roynasser]

@siddontang Makes perfect sense! I didnt realize GME was using bulk api and not river api... I guess thats future-proof then :)

Thanks

[siddontang]

Hi @RVN-BR

Thank you reminding me, maybe we can build another repository named go-mysql-logstash, it will sync the mysql data and then write the json format in a place.

But I haven't used logstash, so I don't know the proper format the logstash needs.

[roynasser]

@siddontang It appears it shouldnt be an issue to configure from the format files in GME to get the appropriate mappings...

I'm now been investigating another interesting and promising project called Crate.io, built on top of ES, but which is more SQL-esque. It allows me to use ES but also to query some data in a structured fashion where it makes more sense. I'm looking into making a driver for GME for crate. It shouldnt be too difficult... crate has a bulk api too, it differs a bit in format from ES, but I'm going to poke around a bit of GO and see if I can eventually try something...

In theory, one can insert straight to ES, but Crate does its own internal type, id, etc mapping which may be difficult or impossible to reproduce statically, so it may be best to try to do it straight to the crate api... https://crate.io/docs/stable/sql/rest.html#bulk-operations

anyways, I am thinking crate may be useful... I'm thinking abt using it as a "bridging gap" for when some things need to be quickly shifted from mysql into a more search-esque backend :) GME might be a great tool to keep the data in sync. At the same time we can still use logstash and other great tools on ES, as well as having a running ES cluster... win win 👍

[siddontang]

Thanks @RVN-BR, I will look it later. :-)