New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ES upgrade to 2.4.1 #808

Open
wants to merge 19 commits into
from

Conversation

Projects
None yet
3 participants
@ChandraAddala
Contributor

ChandraAddala commented May 11, 2017

Upgrading ES to 2.x
 - Upgrading ES to 2.4.4 and lucene to 5.5.2
 - Changes to mapping document: Change from index_analyzer to analyzer
 - Changing max limit from 100,000 to 10,000; Externalized this limit.
 - Adding ESIntegTestCase for integration tests
 - Changes to ingest ES client to ElasticIO and ElasticTokensIO
 - Remove references to shaded libraries of old ES
 - Removed old test library, elasticsearch-test
 - Common logic for ES setup is placed in HttpESIntegrationBase

Chandra Addala added some commits Apr 14, 2017

Chandra Addala
Upgrading ES to 2.x
 - Upgrading ES to 2.4.4 and lucene to 5.5.2
 - Changes to mapping document: Change from index_analyzer to analyzer
 - Changing max limit from 100,000 to 10,000; Externalized this limit.
 - Adding ESIntegTestCase for integration tests
 - Changes to ingest ES client to ElasticIO and ElasticTokensIO
 - Remove references to shaded libraries of old ES
Chandra Addala
Upgrading ES to 2.x
 - Removed old test library, elasticsearch-test
 - Common logic for ES setup is placed in HttpESIntegrationBase
@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 11, 2017

Coverage Status

Coverage decreased (-20.5%) to 54.613% when pulling 1b2649f on ChandraAddala:es-upgrade into be582d1 on rackerlabs:master.

Coverage Status

Coverage decreased (-20.5%) to 54.613% when pulling 1b2649f on ChandraAddala:es-upgrade into be582d1 on rackerlabs:master.

@@ -122,6 +122,7 @@
<!-- do not require zookeeper for these tests. -->
<argLine>-DZOOKEEPER_CLUSTER=NONE</argLine>
<argLine>-Dlog4j.configuration=file://${basedir}/src/test/resources/log4j-test.properties ${jacoco.agent.it.argLine} -Xmx1024m -XX:MaxPermSize=256m</argLine>
<argLine>-Duser.language=en</argLine>

This comment has been minimized.

@shintasmith

shintasmith May 12, 2017

Contributor

What's wrong with French or Korean??? 😂

@shintasmith

shintasmith May 12, 2017

Contributor

What's wrong with French or Korean??? 😂

@@ -37,7 +37,7 @@
private static final char[] STRING_SEEDS = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890_".toCharArray();
protected static final Random RAND = new Random(System.currentTimeMillis());
protected static final ConcurrentHashMap<Locator, String> locatorToUnitMap = new ConcurrentHashMap<Locator, String>();
public static final ConcurrentHashMap<Locator, String> locatorToUnitMap = new ConcurrentHashMap<Locator, String>();

This comment has been minimized.

@shintasmith

shintasmith May 12, 2017

Contributor

Just curious, why are these methods and the above variable all made public?

@shintasmith

shintasmith May 12, 2017

Contributor

Just curious, why are these methods and the above variable all made public?

This comment has been minimized.

@ChandraAddala

ChandraAddala May 17, 2017

Contributor

This class is in blueflood-core. It was earlier extended in blueflood-http test classes. Now instead of extending they contain this class (has-a relationship). I have to make them public to be able to call them.

@ChandraAddala

ChandraAddala May 17, 2017

Contributor

This class is in blueflood-core. It was earlier extended in blueflood-http test classes. Now instead of extending they contain this class (has-a relationship). I have to make them public to be able to call them.

Show outdated Hide outdated blueflood-elasticsearch/pom.xml
@@ -33,8 +33,101 @@
<build>
<testResources>
<testResource>
<directory>src/integration-test/java</directory>

This comment has been minimized.

@shintasmith

shintasmith May 12, 2017

Contributor

Why is this directory specified as a testResource? If there are some files in that directory needed to be copied to classpath, then those files should be moved to src/integration-test/resources directory.

@shintasmith

shintasmith May 12, 2017

Contributor

Why is this directory specified as a testResource? If there are some files in that directory needed to be copied to classpath, then those files should be moved to src/integration-test/resources directory.

<id>add-integration-test-sources</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>

This comment has been minimized.

@shintasmith

shintasmith May 12, 2017

Contributor

Do we need to do this? I know we did this in blueflood-core but I'm wondering if it's just left over cruft there. Adding test sources should be done within the failsafe-plugin or surefire-plugin. There is a

        <testSourceDirectory>src/integration-test/java</testSourceDirectory>

configuration element. Does that not work?

@shintasmith

shintasmith May 12, 2017

Contributor

Do we need to do this? I know we did this in blueflood-core but I'm wondering if it's just left over cruft there. Adding test sources should be done within the failsafe-plugin or surefire-plugin. There is a

        <testSourceDirectory>src/integration-test/java</testSourceDirectory>

configuration element. Does that not work?

This comment has been minimized.

@ChandraAddala

ChandraAddala May 17, 2017

Contributor

This is not available in the version we are using 2.19. It is available from 2.2 onwards.

@ChandraAddala

ChandraAddala May 17, 2017

Contributor

This is not available in the version we are using 2.19. It is available from 2.2 onwards.

public class ElasticIOIntegrationTest extends BaseElasticTest {
protected ElasticIO elasticIO;
protected ElasticTokensIO elasticTokensIO;

This comment has been minimized.

@shintasmith

shintasmith May 12, 2017

Contributor

So is this test changing from testing the Token indexes to just general ElasticSearch test?

@shintasmith

shintasmith May 12, 2017

Contributor

So is this test changing from testing the Token indexes to just general ElasticSearch test?

This comment has been minimized.

@ChandraAddala

ChandraAddala May 17, 2017

Contributor

testing token indexes is in its own class ElasticTokensIOIntegrationTest

@ChandraAddala

ChandraAddala May 17, 2017

Contributor

testing token indexes is in its own class ElasticTokensIOIntegrationTest

* -Dtests.security.manager=false (https://github.com/elastic/elasticsearch/issues/16459)
*
*/
public class ElasticTokensIOIntegrationTest extends BaseElasticTest {

This comment has been minimized.

@shintasmith

shintasmith May 12, 2017

Contributor

👍 for separating this

@shintasmith

shintasmith May 12, 2017

Contributor

👍 for separating this

Configuration.getInstance().setProperty(CoreConfig.DISCOVERY_MODULES.name(), "com.rackspacecloud.blueflood.io.ElasticIO");
Configuration.getInstance().setProperty(CoreConfig.EVENTS_MODULES.name(), "com.rackspacecloud.blueflood.io.EventElasticSearchIO");
Configuration.getInstance().setProperty(CoreConfig.TOKEN_DISCOVERY_MODULES.name(), "com.rackspacecloud.blueflood.io.ElasticTokensIO");
Configuration.getInstance().setProperty(CoreConfig.ENABLE_TOKEN_SEARCH_IMPROVEMENTS.name(), "true");

This comment has been minimized.

@shintasmith

shintasmith May 12, 2017

Contributor

Do we not load a blueflood config for these tests? Should we be putting these in the config file?

@shintasmith

shintasmith May 12, 2017

Contributor

Do we not load a blueflood config for these tests? Should we be putting these in the config file?

This comment has been minimized.

@ChandraAddala

ChandraAddala May 17, 2017

Contributor

Thats correct. We dont load a separate config file. I just used the way it was being done before. It mostly takes defaults for other properties except a few of them.

@ChandraAddala

ChandraAddala May 17, 2017

Contributor

Thats correct. We dont load a separate config file. I just used the way it was being done before. It mostly takes defaults for other properties except a few of them.

@shintasmith

This comment has been minimized.

Show comment
Hide comment
@shintasmith

shintasmith May 12, 2017

Contributor

@ChandraAddala do these changes work with ElasticSearch 1.x ?

Once we merge this PR, we will essentially move our tests to ElasticSearch 2.x, while our Prod is still 1.x. I'm wondering what should be the order of things?

Contributor

shintasmith commented May 12, 2017

@ChandraAddala do these changes work with ElasticSearch 1.x ?

Once we merge this PR, we will essentially move our tests to ElasticSearch 2.x, while our Prod is still 1.x. I'm wondering what should be the order of things?

@shintasmith

This comment has been minimized.

Show comment
Hide comment
@shintasmith

shintasmith May 12, 2017

Contributor

Also, coverage seems to decrease 20% with this PR. Can you look to see if that's just a bug in coverall? Or was there something big we missed testing?

Contributor

shintasmith commented May 12, 2017

Also, coverage seems to decrease 20% with this PR. Can you look to see if that's just a bug in coverall? Or was there something big we missed testing?

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 23, 2017

Coverage Status

Coverage decreased (-20.4%) to 54.675% when pulling da6bbf4 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

Coverage Status

Coverage decreased (-20.4%) to 54.675% when pulling da6bbf4 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 24, 2017

Coverage Status

Coverage decreased (-20.3%) to 54.736% when pulling d8c88a7 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

Coverage Status

Coverage decreased (-20.3%) to 54.736% when pulling d8c88a7 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 25, 2017

Coverage Status

Coverage decreased (-20.4%) to 54.655% when pulling 9e8b15c on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

Coverage Status

Coverage decreased (-20.4%) to 54.655% when pulling 9e8b15c on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 25, 2017

Coverage Status

Coverage decreased (-20.4%) to 54.675% when pulling b83a659 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

Coverage Status

Coverage decreased (-20.4%) to 54.675% when pulling b83a659 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

@ChandraAddala ChandraAddala changed the title from ES upgrade to 2.4.4 to ES upgrade to 2.4.1 May 25, 2017

@ChandraAddala ChandraAddala reopened this May 25, 2017

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 25, 2017

Coverage Status

Coverage decreased (-20.4%) to 54.655% when pulling b7e5f5b on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

Coverage Status

Coverage decreased (-20.4%) to 54.655% when pulling b7e5f5b on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 25, 2017

Coverage Status

Coverage decreased (-20.5%) to 54.614% when pulling 9d7c34e on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

Coverage Status

Coverage decreased (-20.5%) to 54.614% when pulling 9d7c34e on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 25, 2017

Coverage Status

Coverage decreased (-20.5%) to 54.614% when pulling 6fea98e on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

Coverage Status

Coverage decreased (-20.5%) to 54.614% when pulling 6fea98e on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls May 26, 2017

Coverage Status

Coverage decreased (-20.5%) to 54.604% when pulling 8ca95d2 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

Coverage Status

Coverage decreased (-20.5%) to 54.604% when pulling 8ca95d2 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment