Prepare custom build of Elasticsearch 1.4.1

[lukas-vlcek]

After we upgrade to Elasticsearch 1.4.1 (see #194) we shall try to use custom build of Elasticsearch 1.4.1 with specific issues fixed. This ticket describes how to prepare the custom build of Elasticsearch 1.4.1

Abstract

The goal is to prepare Elasticsearch build that includes two pull requests to fix issues with Search Template API. More specifically we want to apply commits from elastic/elasticsearch#8393 lukas-vlcek/elasticsearch@1874efa and elastic/elasticsearch#8255. It should be noted that we will drop Elasticsearch custom build once the Search Template API issues are solved in official Elasticsearch release.

At high-level we will do the following:

• checkout Elasticsearch 1.4.1 into new branch
• pull or cherry-pick needed commits on top of 1.4.1 release commit (there should be no merge conflicts!)
• run Elasticsearch tests
• build custom Elasticsearch artifact and upload the artifact into Maven repository

Git commands

The following is CLI script:

# Clone fresh elasticsearch repo
git clone https://github.com/elasticsearch/elasticsearch.git
cd elasticsearch

# Make sure you are in the master branch
git branch master

# Checkout specific Elasticsearch tag
git checkout v1.4.1

# Create a new branch with (future) custom version based on v1.4.1 code and switch to it
git checkout -b custom1.4.1

# Add lukas-fix remote
git remote add lukas-fix https://github.com/lukas-vlcek/elasticsearch.git

# Assume the following setup of remote repositories
git remote -v

# lukas-fix https://github.com/lukas-vlcek/elasticsearch.git (fetch)
# lukas-fix https://github.com/lukas-vlcek/elasticsearch.git (push)
# origin    https://github.com/elasticsearch/elasticsearch.git (fetch)
# origin    https://github.com/elasticsearch/elasticsearch.git (push)

# Create a new branch with fix 1
git fetch lukas-fix 8308-v1.4.1:fix1

# ...
# From https://github.com/lukas-vlcek/elasticsearch
#  * [new branch]      8308-v1.4.1 -> fix1

# Create a new branch with fix 2
git fetch origin pull/8255/head:fix2

# ...
# From https://github.com/elasticsearch/elasticsearch
#  * [new ref]         refs/pull/8255/head -> fix2

# Create a new branch with fix 3
git fetch lukas-fix pom-v1.4.1:fix3

# Make sure you are in the 'custom1.4.1' branch and there are other
# branches with fixes: 'fix1', 'fix2' and 'fix3'
git branch

# * custom1.4.1
#   fix1
#   fix2
#   fix3
#   master

# Pull from branch 'fix1'
git pull . fix1

# cherry-pick commit from 'fix2'
git cherry-pick d601ba6

# cherry-pick commit from 'fix3'
git cherry-pick 0f498e9

# Verify we have needed commits on top of official 1.4.1 release commit
git log --oneline -n 4 

# 90faa36 Add required pom elements.
# bd559e8 Fix SearchRequest.templateParams so that it is a Map<String, Object> so that it can take more data-types than just strings, to support Arrays.
# 1874efa Search Template - parse template if it is provided as a single escaped string. Closes #8308
# 89d3241 Update Documentation Feature Flags [1.4.1]

At this point the git branch custom1.4.1 contains code that will be used for building custom Elasticsearch 1.4.1.

Testing

You should verify all Elasticsearch tests are passing. As a minimum the following tests MUST pass:

• mvn clean test -Dtests.class=org.elasticsearch.test.rest.ElasticsearchRestTests
• mvn clean test -Dtests.class=org.elasticsearch.index.query.TemplateQueryTest
• mvn clean test -Dtests.class=org.elasticsearch.index.query.TemplateQueryParserTest

Building custom maven artifact and install it into Nexus

In order to deploy final artifact into Nexus Maven repo the pom.xml needs to comply with all requirements. See https://developer.jboss.org/wiki/WildflyExtrasRepository, section "2.2 Valid pom.xml" for details. These updates were subject to 'fix3'. So we should be ready to start maven build and install the artifact into Nexus now.

mvn clean deploy -DskipTests -Dmaven.javadoc.failOnError=false

Using custom Elasticsearch build

Custom build is available in JBoss maven repository. It can be referenced as follows:

    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-patched</artifactId>
        <version>1.4.1</version>
        <exclusions>
        ...
        </exclusions>
    </dependency>

It is also recommended to explicitly exclude elasticsearch dependency from other dependencies that are dependent of elasticsearch to prevent downloading the official elasticsearch jar.

For example:

    <dependency>
        <groupId>org.jboss.elasticsearch</groupId>
        <artifactId>structured-content-tools</artifactId>
        <version>${structured-content-tools.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.elasticsearch</groupId>
                <artifactId>elasticsearch</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Add the following repository to pom.xml:

    <repository>
        <id>jboss-org-dev</id>
        <name>JBoss Public Maven Repository with experimental artifacts</name>
        <url>https://repository.jboss.org/nexus/content/repositories/jbossorg-dev/</url>
        <layout>default</layout>
        <releases>
            <enabled>true</enabled>
            <updatePolicy>never</updatePolicy>
        </releases>
        <snapshots>
            <enabled>true</enabled>
            <updatePolicy>never</updatePolicy>
        </snapshots>
    </repository>

Generate patch file

git checkout custom1.4.1
git format-patch v1.4.1 --stdout > fix_elasticsearch_v1.4.1.patch

Resulting file fix_elasticsearch_v1.4.1.patch can be used when building RPMs from official Elasticsearch sources.

Building release bundle

If we need to build a release bundles then keep in mind that currently custom named Elasticsearch artifact is not included. To make it included you need to update src/main/assemblies/common-bin.xml. Find the following:

        <includes>
            <include>org.elasticsearch:elasticsearch</include>
        </includes> 

and update it to:

        <includes>
            <include>org.elasticsearch:elasticsearch</include>
            <include>org.elasticsearch:elasticsearch-patched</include>
        </includes>

This will include patched Elasticsearch into release bundles when mvn clean package -DskipTests is executed. However, some supporting scripts my still assume there is elasticsearch-1.4.1.jar under lib folder inside the bundle. You might want to rename elasticsearch-patched-1.4.1.jar to elasticsearch-1.4.1.jar after bundle is installed on target machine.