Skip to content

Commit

Permalink
Merge branch 'master' into thread-pool-info-migration
Browse files Browse the repository at this point in the history
* master: (110 commits)
  Remove undocumented action.master.force_local setting (elastic#29351)
  Enhance error for out of bounds byte size settings (elastic#29338)
  Fix QueryAnalyzerTests.
  Fix HasChildQueryBuilderTests to not use the `classic` similarity.
  [Docs] Correct javadoc of GetIndexRequest (elastic#29364)
  Make TransportRankEvalAction members final
  Add awaits fix for a query analyzer test
  Check presence of multi-types before validating new mapping (elastic#29316)
  Add awaits fix for HasChildQueryBuilderTests
  Remove silent batch mode from install plugin (elastic#29359)
  Align cat thread pool info to thread pool config (elastic#29195)
  Track Lucene operations in engine explicitly (elastic#29357)
  Build: Fix Java9 MR build (elastic#29312)
  Reindex: Fix error in delete-by-query rest spec (elastic#29318)
  Improve similarity integration. (elastic#29187)
  Fix some query extraction bugs. (elastic#29283)
  [Docs] Correct experimental note formatting
  Move Nullable into core (elastic#29341)
  [Docs] Update getting-started.asciidoc (elastic#29294)
  Elasticsearch 6.3.0 is now on Lucene 7.3.
  ...
  • Loading branch information
jasontedor committed Apr 5, 2018
2 parents c5ab1b3 + 25d411e commit c7dae3c
Show file tree
Hide file tree
Showing 631 changed files with 12,119 additions and 6,025 deletions.
8 changes: 8 additions & 0 deletions .ci/java-versions.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# This file is used with all of the non-matrix tests in Jenkins.

# This .properties file defines the versions of Java with which to
# build and test Elasticsearch for this branch. Valid Java versions
# are 'java' or 'openjdk' followed by the major release number.

ES_BUILD_JAVA=java10
ES_RUNTIME_JAVA=java8
9 changes: 9 additions & 0 deletions .ci/matrix-build-javas.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# This file is used as part of a matrix build in Jenkins where the
# values below are included as an axis of the matrix.

# This axis of the build matrix represents the versions of Java with
# which Elasticsearch will be built. Valid Java versions are 'java'
# or 'openjdk' followed by the major release number.

ES_BUILD_JAVA:
- java10
14 changes: 14 additions & 0 deletions .ci/matrix-java-exclusions.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# This file is used as part of a matrix build in Jenkins where the
# values below are excluded from the test matrix.

# The yaml mapping below represents a single intersection on the build
# matrix where a test *should not* be run. The value of the exclude
# key is a list of maps.

# In this example all of the combinations defined in the matrix will
# run except for the test that builds with java10 and runs with java8.
# exclude:
# - ES_BUILD_JAVA: java10
# ES_RUNTIME_JAVA: java8

exclude:
10 changes: 10 additions & 0 deletions .ci/matrix-runtime-javas.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# This file is used as part of a matrix build in Jenkins where the
# values below are included as an axis of the matrix.

# This axis of the build matrix represents the versions of Java on
# which Elasticsearch will be tested. Valid Java versions are 'java'
# or 'openjdk' followed by the major release number.

ES_RUNTIME_JAVA:
- java8
- java10
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,11 +92,11 @@ Contributing to the Elasticsearch codebase

**Repository:** [https://github.com/elastic/elasticsearch](https://github.com/elastic/elasticsearch)

JDK 9 is required to build Elasticsearch. You must have a JDK 9 installation
JDK 10 is required to build Elasticsearch. You must have a JDK 10 installation
with the environment variable `JAVA_HOME` referencing the path to Java home for
your JDK 9 installation. By default, tests use the same runtime as `JAVA_HOME`.
your JDK 10 installation. By default, tests use the same runtime as `JAVA_HOME`.
However, since Elasticsearch, supports JDK 8 the build supports compiling with
JDK 9 and testing on a JDK 8 runtime; to do this, set `RUNTIME_JAVA_HOME`
JDK 10 and testing on a JDK 8 runtime; to do this, set `RUNTIME_JAVA_HOME`
pointing to the Java home of a JDK 8 installation. Note that this mechanism can
be used to test against other JDKs as well, this is not only limited to JDK 8.

Expand Down
21 changes: 0 additions & 21 deletions README.textile
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ Elasticsearch is a distributed RESTful search engine built for the cloud. Featur
** All the power of Lucene easily exposed through simple configuration / plugins.
* Per operation consistency
** Single document level operations are atomic, consistent, isolated and durable.
* Open Source under the Apache License, version 2 ("ALv2")

h2. Getting Started

Expand Down Expand Up @@ -217,23 +216,3 @@ Elasticsearch (1.x), it is required to perform a full cluster restart. Please
see the "setup reference":
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
for more details on the upgrade process.

h1. License

<pre>
This software is licensed under the Apache License, version 2 ("ALv2"), quoted below.

Copyright 2009-2016 Elasticsearch <https://www.elastic.co>

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.
</pre>
46 changes: 17 additions & 29 deletions TESTING.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,6 @@ e.g. -Dtests.rest.suite=index,get,create/10_with_id
* `tests.rest.blacklist`: comma separated globs that identify tests that are
blacklisted and need to be skipped
e.g. -Dtests.rest.blacklist=index/*/Index document,get/10_basic/*
* `tests.rest.spec`: REST spec path (default /rest-api-spec/api)

Note that the REST tests, like all the integration tests, can be run against an external
cluster by specifying the `tests.cluster` property, which if present needs to contain a
Expand Down Expand Up @@ -414,16 +413,16 @@ and in another window:

----------------------------------------------------
vagrant up centos-7 --provider virtualbox && vagrant ssh centos-7
cd $BATS_ARCHIVES
cd $PACKAGING_ARCHIVES
sudo -E bats $BATS_TESTS/*rpm*.bats
----------------------------------------------------

If you wanted to retest all the release artifacts on a single VM you could:

-------------------------------------------------
./gradlew setupBats
./gradlew setupPackagingTest
cd qa/vagrant; vagrant up ubuntu-1404 --provider virtualbox && vagrant ssh ubuntu-1404
cd $BATS_ARCHIVES
cd $PACKAGING_ARCHIVES
sudo -E bats $BATS_TESTS/*.bats
-------------------------------------------------

Expand Down Expand Up @@ -477,12 +476,12 @@ branch. Finally, on a release branch, it will test against the most recent relea
=== BWC Testing against a specific remote/branch

Sometimes a backward compatibility change spans two versions. A common case is a new functionality
that needs a BWC bridge in and an unreleased versioned of a release branch (for example, 5.x).
that needs a BWC bridge in an unreleased versioned of a release branch (for example, 5.x).
To test the changes, you can instruct Gradle to build the BWC version from a another remote/branch combination instead of
pulling the release branch from GitHub. You do so using the `tests.bwc.remote` and `tests.bwc.refspec` system properties:
pulling the release branch from GitHub. You do so using the `tests.bwc.remote` and `tests.bwc.refspec.BRANCH` system properties:

-------------------------------------------------
./gradlew check -Dtests.bwc.remote=${remote} -Dtests.bwc.refspec=index_req_bwc_5.x
./gradlew check -Dtests.bwc.remote=${remote} -Dtests.bwc.refspec.5.x=index_req_bwc_5.x
-------------------------------------------------

The branch needs to be available on the remote that the BWC makes of the
Expand All @@ -497,34 +496,23 @@ will need to:
will contain your change.
. Create a branch called `index_req_bwc_5.x` off `5.x`. This will contain your bwc layer.
. Push both branches to your remote repository.
. Run the tests with `./gradlew check -Dtests.bwc.remote=${remote} -Dtests.bwc.refspec=index_req_bwc_5.x`.
. Run the tests with `./gradlew check -Dtests.bwc.remote=${remote} -Dtests.bwc.refspec.5.x=index_req_bwc_5.x`.

== Coverage analysis
== Test coverage analysis

Tests can be run instrumented with jacoco to produce a coverage report in
`target/site/jacoco/`.
Generating test coverage reports for Elasticsearch is currently not possible through Gradle.
However, it _is_ possible to gain insight in code coverage using IntelliJ's built-in coverage
analysis tool that can measure coverage upon executing specific tests. Eclipse may also be able
to do the same using the EclEmma plugin.

Unit test coverage:

---------------------------------------------------------------------------
mvn -Dtests.coverage test jacoco:report
---------------------------------------------------------------------------

Integration test coverage:

---------------------------------------------------------------------------
mvn -Dtests.coverage -Dskip.unit.tests verify jacoco:report
---------------------------------------------------------------------------

Combined (Unit+Integration) coverage:

---------------------------------------------------------------------------
mvn -Dtests.coverage verify jacoco:report
---------------------------------------------------------------------------
Test coverage reporting used to be possible with JaCoCo when Elasticsearch was using Maven
as its build system. Since the switch to Gradle though, this is no longer possible, seeing as
the code currently used to build Elasticsearch does not allow JaCoCo to recognize its tests.
For more information on this, see the discussion in https://github.com/elastic/elasticsearch/issues/28867[issue #28867].

== Launching and debugging from an IDE

If you want to run elasticsearch from your IDE, the `./gradlew run` task
If you want to run Elasticsearch from your IDE, the `./gradlew run` task
supports a remote debugging option:

---------------------------------------------------------------------------
Expand Down
8 changes: 4 additions & 4 deletions Vagrantfile
Original file line number Diff line number Diff line change
Expand Up @@ -334,9 +334,9 @@ export TAR=/elasticsearch/distribution/tar/build/distributions
export RPM=/elasticsearch/distribution/rpm/build/distributions
export DEB=/elasticsearch/distribution/deb/build/distributions
export BATS=/project/build/bats
export BATS_UTILS=/project/build/bats/utils
export BATS_TESTS=/project/build/bats/tests
export BATS_ARCHIVES=/project/build/bats/archives
export BATS_UTILS=/project/build/packaging/bats/utils
export BATS_TESTS=/project/build/packaging/bats/tests
export PACKAGING_ARCHIVES=/project/build/packaging/archives
VARS
cat \<\<SUDOERS_VARS > /etc/sudoers.d/elasticsearch_vars
Defaults env_keep += "ZIP"
Expand All @@ -346,7 +346,7 @@ Defaults env_keep += "DEB"
Defaults env_keep += "BATS"
Defaults env_keep += "BATS_UTILS"
Defaults env_keep += "BATS_TESTS"
Defaults env_keep += "BATS_ARCHIVES"
Defaults env_keep += "PACKAGING_ARCHIVES"
SUDOERS_VARS
chmod 0440 /etc/sudoers.d/elasticsearch_vars
SHELL
Expand Down
1 change: 1 addition & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ subprojects {
"org.elasticsearch:elasticsearch-cli:${version}": ':server:cli',
"org.elasticsearch:elasticsearch-core:${version}": ':libs:elasticsearch-core',
"org.elasticsearch:elasticsearch-nio:${version}": ':libs:elasticsearch-nio',
"org.elasticsearch:elasticsearch-x-content:${version}": ':libs:x-content',
"org.elasticsearch:elasticsearch-secure-sm:${version}": ':libs:secure-sm',
"org.elasticsearch.client:elasticsearch-rest-client:${version}": ':client:rest',
"org.elasticsearch.client:elasticsearch-rest-client-sniffer:${version}": ':client:sniffer',
Expand Down
2 changes: 1 addition & 1 deletion buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ dependencies {
compile 'com.netflix.nebula:gradle-info-plugin:3.0.3'
compile 'org.eclipse.jgit:org.eclipse.jgit:3.2.0.201312181205-r'
compile 'com.perforce:p4java:2012.3.551082' // THIS IS SUPPOSED TO BE OPTIONAL IN THE FUTURE....
compile 'de.thetaphi:forbiddenapis:2.4.1'
compile 'de.thetaphi:forbiddenapis:2.5'
compile 'org.apache.rat:apache-rat:0.11'
compile "org.elasticsearch:jna:4.5.1"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ import java.time.ZonedDateTime
class BuildPlugin implements Plugin<Project> {

static final JavaVersion minimumRuntimeVersion = JavaVersion.VERSION_1_8
static final JavaVersion minimumCompilerVersion = JavaVersion.VERSION_1_9
static final JavaVersion minimumCompilerVersion = JavaVersion.VERSION_1_10

@Override
void apply(Project project) {
Expand Down Expand Up @@ -311,8 +311,8 @@ class BuildPlugin implements Plugin<Project> {
/** Adds repositories used by ES dependencies */
static void configureRepositories(Project project) {
RepositoryHandler repos = project.repositories
if (System.getProperty("repos.mavenlocal") != null) {
// with -Drepos.mavenlocal=true we can force checking the local .m2 repo which is
if (System.getProperty("repos.mavenLocal") != null) {
// with -Drepos.mavenLocal=true we can force checking the local .m2 repo which is
// useful for development ie. bwc tests where we install stuff in the local repository
// such that we don't have to pass hardcoded files to gradle
repos.mavenLocal()
Expand Down Expand Up @@ -551,7 +551,7 @@ class BuildPlugin implements Plugin<Project> {
if (project.licenseFile == null || project.noticeFile == null) {
throw new GradleException("Must specify license and notice file for project ${project.path}")
}
jarTask.into('META-INF') {
jarTask.metaInf {
from(project.licenseFile.parent) {
include project.licenseFile.name
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ public class PluginBuildPlugin extends BuildPlugin {
// we "upgrade" these optional deps to provided for plugins, since they will run
// with a full elasticsearch server that includes optional deps
compileOnly "org.locationtech.spatial4j:spatial4j:${project.versions.spatial4j}"
compileOnly "com.vividsolutions:jts:${project.versions.jts}"
compileOnly "org.locationtech.jts:jts-core:${project.versions.jts}"
compileOnly "org.apache.logging.log4j:log4j-api:${project.versions.log4j}"
compileOnly "org.apache.logging.log4j:log4j-core:${project.versions.log4j}"
compileOnly "org.elasticsearch:jna:${project.versions.jna}"
Expand Down Expand Up @@ -168,12 +168,10 @@ public class PluginBuildPlugin extends BuildPlugin {
Files.copy(jarFile.resolveSibling(sourcesFileName), jarFile.resolveSibling(clientSourcesFileName),
StandardCopyOption.REPLACE_EXISTING)

if (project.compilerJavaVersion < JavaVersion.VERSION_1_10) {
String javadocFileName = jarFile.fileName.toString().replace('.jar', '-javadoc.jar')
String clientJavadocFileName = clientFileName.replace('.jar', '-javadoc.jar')
Files.copy(jarFile.resolveSibling(javadocFileName), jarFile.resolveSibling(clientJavadocFileName),
StandardCopyOption.REPLACE_EXISTING)
}
String javadocFileName = jarFile.fileName.toString().replace('.jar', '-javadoc.jar')
String clientJavadocFileName = clientFileName.replace('.jar', '-javadoc.jar')
Files.copy(jarFile.resolveSibling(javadocFileName), jarFile.resolveSibling(clientJavadocFileName),
StandardCopyOption.REPLACE_EXISTING)
}
project.assemble.dependsOn(clientJar)
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.elasticsearch.gradle.precommit

import org.gradle.api.DefaultTask
import org.gradle.api.GradleException
import org.gradle.api.file.FileCollection
import org.gradle.api.tasks.InputFiles
import org.gradle.api.tasks.OutputFile
import org.gradle.api.tasks.SourceSet
import org.gradle.api.tasks.TaskAction
import org.gradle.api.tasks.util.PatternSet
import org.gradle.api.tasks.util.PatternFilterable
import org.apache.tools.ant.taskdefs.condition.Os

import java.nio.file.Files
import java.nio.file.attribute.PosixFilePermission
import java.nio.file.attribute.PosixFileAttributeView

import static java.nio.file.attribute.PosixFilePermission.OTHERS_EXECUTE
import static java.nio.file.attribute.PosixFilePermission.GROUP_EXECUTE
import static java.nio.file.attribute.PosixFilePermission.OWNER_EXECUTE

/**
* Checks source files for correct file permissions.
*/
public class FilePermissionsTask extends DefaultTask {

/** A pattern set of which files should be checked. */
private PatternFilterable filesFilter = new PatternSet()

@OutputFile
File outputMarker = new File(project.buildDir, 'markers/filePermissions')

FilePermissionsTask() {
onlyIf { !Os.isFamily(Os.FAMILY_WINDOWS) }
description = "Checks java source files for correct file permissions"
// we always include all source files, and exclude what should not be checked
filesFilter.include('**')
// exclude sh files that might have the executable bit set
filesFilter.exclude('**/*.sh')
}

/** Returns the files this task will check */
@InputFiles
FileCollection files() {
List<FileCollection> collections = new ArrayList<>()
for (SourceSet sourceSet : project.sourceSets) {
collections.add(sourceSet.allSource.matching(filesFilter))
}
return project.files(collections.toArray())
}

@TaskAction
void checkInvalidPermissions() {
List<String> failures = new ArrayList<>()
for (File f : files()) {
PosixFileAttributeView fileAttributeView = Files.getFileAttributeView(f.toPath(), PosixFileAttributeView.class)
Set<PosixFilePermission> permissions = fileAttributeView.readAttributes().permissions()
if (permissions.contains(OTHERS_EXECUTE) || permissions.contains(OWNER_EXECUTE) ||
permissions.contains(GROUP_EXECUTE)) {
failures.add("Source file is executable: " + f)
}
}
if (failures.isEmpty() == false) {
throw new GradleException('Found invalid file permissions:\n' + failures.join('\n'))
}
outputMarker.setText('done', 'UTF-8')
}

}
Loading

0 comments on commit c7dae3c

Please sign in to comment.