support for ES 7 (fixes #155) #161

jameslamb · 2019-08-22T04:51:26Z

This pull requests aims to add support in uptasticsearch for Elasticsearch 7.x. See the linked issue and changes to NEWS.md for details on what has changed in ES7.x.

This PR's scope is limited to "get uptasticsearch code working with Es7.x". It does not include taking advantage of any ES7-specific features like new types of aggregations.

Opening this as a draft PR as it currently only addresses the R side. The Python side needs to be updated n this PR as well.

jameslamb · 2019-08-22T04:53:49Z

test_data/sample_es7.json

@@ -0,0 +1,96 @@
+{"index":{"_index":"shakespeare","_id":2}}
+{"line_id":3,"play_name":"Henry IV","speech_number":"","line_number":"","speaker":"","text_entry":"Enter KING HENRY, LORD JOHN OF LANCASTER, the EARL of WESTMORELAND, SIR WALTER BLUNT, and others"}
+{"index":{"_index":"shakespeare","_id":3}}


it was necessary to make this new file because _type was removed in ES7. Having documents with multiple types within one index is now not allowed and attempts to set _type will raise errors complaining about multiple document types (unless you explicitly include a default document type in your mapping).

It's possible this file could be irrelevant and that es7_mapping.json could just have some argument added to it that specifics the default document type to line, but I couldn't figure out how to do that in 5 minutes and this worked. Definitely would like to come back to it.

nice, want to make an issue for that after this is merged?

created #167

austin3dickey · 2019-08-22T15:06:40Z

NEWS.md

+
+## Features
+
+### Full support for ES7.x


Maybe just "Support" without "Full"? Since new features aren't accounted for yet?

yeah good call

still have to do this I think

ugh you're right. I want Bitbucket tasks back :(

austin3dickey · 2019-08-22T15:13:03Z

r-pkg/tests/testthat/test-integration.R

        expect_true(data.table::is.data.table(outDT))
-        expect_true(nrow(outDT) == 4)
+        expect_true(nrow(outDT) == 3)


Looks like this is failing checks?

Ha yeah I kind of thought it might! This is a weird inconsistency between ES <7 and Es 7. I think that what's happening is that previously you were able to have multiple document types in one index and that + something weird in the way we write test data was causing duplicate entries for some indices. I'll have to figure that out to get this merged.

nope I was totally wrong. It's a different thing. In the ES7.x one, I'm using the keyword type (first introduced in ES6.x) for field speaker. That tells Elasticsearch "don't pass the values through a tokenizer, treat the full text as a single level of a categorical".

For ES6 I'm using the text type and for ES5 I'm using the string type. Those default to breaking down their inputs into tokens with a whitespace tokenizer.

So basically for the ES7 test it thinks there are three unique speakers: henry iv, king, and westmoreland. But for all earlier versions, the same terms agg gives you four levels:

thing doc_count 1: henry 34 2: iv 34 3: king 34 4: westmoreland 13

I chose the keyword type for this field in the ES7 mapping because Elasticsearch removed the use of fielddata = true to say "make it possible to do a terms agg".

For now, I'm going to go with the approach of added an explicit check on the version around this test. It's gross so I'll open a "come back and make this less gross" bug, but I feel like it's a thing that will:

Give us confidence that this PR doesn't break backwards compatibility of our library with all earlier Elasticsearch versions

Give us confidence that uptasticsearch can process the result of a terms agg from ES7.x correctly

austin3dickey · 2019-08-22T15:13:44Z

I couldn't find Travis checking 7.3.0; is that expected?

austin3dickey · 2019-08-22T15:14:41Z

TIL the "work in progress" feature! Haven't seen that before

jameslamb · 2019-08-22T22:15:11Z

TIL the "work in progress" feature! Haven't seen that before

yep it's a new-ish feature! I of course appreciate the review, but I did open it in WIP so you'd know you didn't have to review yet

jameslamb · 2019-08-22T22:15:36Z

I couldn't find Travis checking 7.3.0; is that expected?

Nope that's an omission, thank you!

jameslamb · 2019-09-04T01:51:50Z

Ok I rebased to catch the changes in #163 , so the diff for .travis.yml looks a lot different now. I also commented out all but one 6.2.x build and one 7.3.x build for each language, to speed up the feedback cycle on this PR. I'll uncomment them all whenever I feel ready to move this from draft to official open PR

codecov-io · 2019-09-04T03:18:14Z

Codecov Report

Merging #161 into master will increase coverage by 0.3%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master    #161     +/-   ##
========================================
+ Coverage    92.8%   93.1%   +0.3%     
========================================
  Files           8       8             
  Lines         556     595     +39     
========================================
+ Hits          516     554     +38     
- Misses         40      41      +1

Impacted Files	Coverage Δ
R/es_search.R	`87.87% <0%> (-0.25%)`	⬇️
R/get_fields.R	`94.69% <0%> (+2.48%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a6b39b...19ae702. Read the comment docs.

jameslamb · 2019-09-04T03:22:26Z

Ok I rebased to catch the changes in #163 , so the diff for .travis.yml looks a lot different now. I also commented out all but one 6.2.x build and one 7.3.x build for each language, to speed up the feedback cycle on this PR. I'll uncomment them all whenever I feel ready to move this from draft to official open PR

This is working! Going to add ALL of the versions back and make this an official PR.

@austin3dickey take a look whenever you have time.

austin3dickey · 2019-09-04T14:55:58Z

py-pkg/uptasticsearch/clients.py

+        respose to a  ``POST /_search`` request, return the total
+        number of docs matching the query
+        """
+        return response_json['hits']['total']['value']


austin3dickey

LGTM! Just that small NEWS tweak

jameslamb · 2019-09-04T20:20:07Z

Ok one more review por favor (I also miss the bitbucket "leave tasks plus approve" thing)

austin3dickey

nice!!

jameslamb commented Aug 22, 2019

View reviewed changes

jameslamb mentioned this pull request Aug 22, 2019

add support for Elasticsearch 7 #155

Closed

austin3dickey reviewed Aug 22, 2019

View reviewed changes

jameslamb mentioned this pull request Aug 30, 2019

simplified CI setup #163

Merged

jameslamb force-pushed the es7 branch from 5d16a17 to b9c53bd Compare September 4, 2019 01:50

This was referenced Sep 4, 2019

added python package stuff to gitignore #166

Merged

test_data includes a file only used for ES 7 and up #167

Open

jameslamb marked this pull request as ready for review September 4, 2019 03:22

austin3dickey reviewed Sep 4, 2019

View reviewed changes

austin3dickey requested changes Sep 4, 2019

View reviewed changes

jameslamb added 12 commits September 4, 2019 15:17

updated R package to support ES 7 (fixes uptake#155)

d44df57

added ES7 support on the Python side

3fcdd1d

fixed tests

163fa5e

fixed R unit tests to handle ES7

6be2982

updated strategy for running ES 7 in CI

4f3174a

simplified CI code for ES7

a4799e8

updated installations because caching on travis is weird

8675a44

try to get around permissions issues

b953e44

trying ubuntu things I only somewhat understand

3bff78c

trying to fix permissions issues

8021676

trying to get more informative error

b3ced6d

fixed data writing code in CI

3e40617

jameslamb added 3 commits September 4, 2019 15:17

trying more CI things

a1284e5

added ES7 to CI

91e7cf2

clarified item in NEWS

e3abdee

jameslamb requested a review from austin3dickey September 4, 2019 20:19

jameslamb force-pushed the es7 branch from 19ae702 to e3abdee Compare September 4, 2019 20:19

austin3dickey approved these changes Sep 4, 2019

View reviewed changes

jameslamb merged commit 3283bca into uptake:master Sep 4, 2019

jameslamb deleted the es7 branch September 4, 2019 21:52

This was referenced Sep 4, 2019

update ES versions we test against #169

Merged

release v0.4.0 #170

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support for ES 7 (fixes #155) #161

support for ES 7 (fixes #155) #161

jameslamb commented Aug 22, 2019

jameslamb Aug 22, 2019

austin3dickey Aug 22, 2019

jameslamb Sep 4, 2019

austin3dickey Aug 22, 2019

jameslamb Aug 22, 2019

austin3dickey Sep 4, 2019

jameslamb Sep 4, 2019

austin3dickey Aug 22, 2019

jameslamb Aug 23, 2019

jameslamb Aug 31, 2019

austin3dickey commented Aug 22, 2019

austin3dickey commented Aug 22, 2019

jameslamb commented Aug 22, 2019

jameslamb commented Aug 22, 2019

jameslamb commented Sep 4, 2019

codecov-io commented Sep 4, 2019 •

edited

Loading

jameslamb commented Sep 4, 2019

austin3dickey Sep 4, 2019

austin3dickey left a comment

jameslamb commented Sep 4, 2019 •

edited

Loading

austin3dickey left a comment

support for ES 7 (fixes #155) #161

support for ES 7 (fixes #155) #161

Conversation

jameslamb commented Aug 22, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

austin3dickey commented Aug 22, 2019

austin3dickey commented Aug 22, 2019

jameslamb commented Aug 22, 2019

jameslamb commented Aug 22, 2019

jameslamb commented Sep 4, 2019

codecov-io commented Sep 4, 2019 • edited Loading

Codecov Report

jameslamb commented Sep 4, 2019

Choose a reason for hiding this comment

austin3dickey left a comment

Choose a reason for hiding this comment

jameslamb commented Sep 4, 2019 • edited Loading

austin3dickey left a comment

Choose a reason for hiding this comment

codecov-io commented Sep 4, 2019 •

edited

Loading

jameslamb commented Sep 4, 2019 •

edited

Loading