Compatibility with 0.90.3

[gkarlikatwork]

Hi,

due to changes on elasticsearch side in 0.90.3 plugin does not build anymore.

[mfussenegger]

Hi thanks for the report.

We're aware that the recent update broke some of our plugin. Our plan was to finish up some other work before we start updating the plugins.

For this plugin I've already created a pull request here #36. It builds again and tests pass. But I will have to follow up with a real world test (hopefully tomorrow) before I can merge it.

So we should have a new release out soon enough.

[gkarlikatwork]

Hi,

thanks for your checkin. I tried it and start restoring my test full dump. I got immediately FullGCs in Elasticseach. I had no problems with same data in 0.90.1. I use default elasticsearch GC Settings with 1GB Xmx. I tried with 1.5 Gb Xmx but it does not solve this problem. My test dump has 12 millions docs with 600 MB of json files ...

[mfussenegger]

I've added another commit to the 0.90.3 branch that should help reduce the GCs see https://github.com/crate/elasticsearch-inout-plugin/tree/0.90.3

Can you try that and see if it helps? How exactly are the GCs affecting you? Is a performance difference noticeable between 0.90.1 and 0.90.3?

[gkarlikatwork]

Hi,

the performance is really bad with 0.90.3. But i think, it's because of FullGCs ... I will try you checkin and post a result.

[gkarlikatwork]

I've tried your checkin, but unfortunately, the same problem still exists.

My GC Log looks like this:
2013-08-29T11:30:19.740+0200: 44,469: [Full GC 44,469: [CMS2013-08-29T11:30:19.769+0200: 44,498: [CMS-concurrent-mark: 0,040/0,042 secs] [Times: user=0,15 sys=0,01, real=0,04 secs](concurrent mode failure): 699068K->699071K(699072K), 0,2804530 secs] 1013627K->802848K(1013632K), [CMS Perm : 35901K->35901K(59884K)], 0,2805270 secs] [Times: user=0,32 sys=0,00, real=0,28 secs]
2013-08-29T11:30:20.029+0200: 44,758: [GC [1 CMS-initial-mark: 699071K(699072K)] 825226K(1013632K), 0,0080890 secs] [Times: user=0,01 sys=0,00, real=0,01 secs]
2013-08-29T11:30:20.038+0200: 44,767: [CMS-concurrent-mark-start]
2013-08-29T11:30:20.109+0200: 44,838: [Full GC 44,838: [CMS2013-08-29T11:30:20.113+0200: 44,842: [CMS-concurrent-mark: 0,069/0,076 secs] [Times: user=0,53 sys=0,00, real=0,08 secs](concurrent mode failure): 699071K->699044K(699072K), 0,2318230 secs] 1013609K->867754K(1013632K), [CMS Perm : 35904K->35904K(59884K)], 0,2318940 secs] [Times: user=0,24 sys=0,00, real=0,24 secs]
2013-08-29T11:30:20.394+0200: 45,123: [Full GC 45,123: [CMS: 699044K->699043K(699072K), 0,2267800 secs] 1013587K->907125K(1013632K), [CMS Perm : 35907K->35907K(59884K)], 0,2268510 secs] [Times: user=0,25 sys=0,00, real=0,23 secs]
2013-08-29T11:30:20.622+0200: 45,351: [GC [1 CMS-initial-mark: 699043K(699072K)] 909525K(1013632K), 0,0070420 secs] [Times: user=0,01 sys=0,00, real=0,00 secs]
2013-08-29T11:30:20.629+0200: 45,358: [CMS-concurrent-mark-start]
2013-08-29T11:30:20.684+0200: 45,413: [Full GC 45,413: [CMS2013-08-29T11:30:20.689+0200: 45,418: [CMS-concurrent-mark: 0,057/0,060 secs] [Times: user=0,41 sys=0,00, real=0,06 secs](concurrent mode failure): 699050K->699050K(699072K), 0,1611840 secs] 1013610K->936892K(1013632K), [CMS Perm : 35908K->35908K(59884K)], 0,1612560 secs] [Times: user=0,17 sys=0,00, real=0,16 secs]
2013-08-29T11:30:20.858+0200: 45,587: [GC [1 CMS-initial-mark: 699050K(699072K)] 984412K(1013632K), 0,0157220 secs] [Times: user=0,02 sys=0,00, real=0,02 secs]
2013-08-29T11:30:20.874+0200: 45,603: [CMS-concurrent-mark-start]
2013-08-29T11:30:20.905+0200: 45,634: [Full GC 45,634: [CMS2013-08-29T11:30:20.929+0200: 45,658: [CMS-concurrent-mark: 0,049/0,055 secs] [Times: user=0,26 sys=0,01, real=0,05 secs](concurrent mode failure): 699050K->699041K(699072K), 0,2526990 secs] 1013581K->964362K(1013632K), [CMS Perm : 35913K->35913K(59884K)], 0,2527690 secs] [Times: user=0,29 sys=0,00, real=0,25 secs]
2013-08-29T11:30:21.190+0200: 45,919: [Full GC 45,919: [CMS: 699041K->699041K(699072K), 0,1645230 secs] 1013592K->983925K(1013632K), [CMS Perm : 35916K->35916K(59884K)], 0,1645950 secs] [Times: user=0,17 sys=0,00, real=0,17 secs]
2013-08-29T11:30:21.355+0200: 46,084: [GC [1 CMS-initial-mark: 699041K(699072K)] 983985K(1013632K), 0,0078270 secs] [Times: user=0,01 sys=0,00, real=0,01 secs]
2013-08-29T11:30:21.363+0200: 46,092: [CMS-concurrent-mark-start]

[gkarlikatwork]

This issue could be a problem: elastic/elasticsearch#3638

[gkarlikatwork]

Plugin works fine with 0.90.4

[gkarlikatwork]

Plugin does not build with 0.90.4, 0.90.5. I tried it as a binary from 0.90.3 before and import works well.

[apatrida]

See other tickets, I ported to 0.90.5, 0.90.6 and 1.0.0.beta1 as pull requests.

If you apply the pull requests you can help test.