Unable to run mvn test, failing for randomizedtesting

[meghalidhoble]

Elasticsearch version: 2.3

JVM version: "1.8.0_91"

OS version: Rhel 7.2

Description of the problem including expected versus actual behavior:
mvn clean package and mvn test command are failing with
"[ERROR] Failed to execute goal com.carrotsearch.randomizedtesting:junit4-maven-plugin:2.3.2:junit4 (tests) on project elasticsearch"

Steps to reproduce:

1. Install maven 3.3.3 from tar ball link : http://archive.apache.org/dist/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz and install Java 8 (java-1.8.0-openjdk-devel
2. Set desired PATH to have maven and java usable
3. Copy the source for Elastic-search
4. Then run maven commands inside the elasticsearch directory; "mvn clean package/ mvn test"

Relevant Logs:

Tests with failures:

• org.elasticsearch.plugins.PluginManagerPermissionTests.testThatUnaccessiblePluginConfigDirectoryAbortsPluginInstallation
• org.elasticsearch.plugins.PluginManagerPermissionTests.testThatUnaccessibleBinDirectoryAbortsPluginInstallation
• org.elasticsearch.plugins.PluginManagerPermissionTests.testThatNonWritablePluginsDirectoryLeavesNoLeftOver

[INFO] JVM J0: 0.98 .. 95.07 = 94.10s
[INFO] JVM J1: 0.98 .. 95.29 = 94.31s
[INFO] JVM J2: 0.97 .. 95.18 = 94.21s
[INFO] Execution time total: 1 minute 35 seconds
[INFO] Tests summary: 516 suites, 2943 tests, 3 failures, 10 ignored (10 assumptions)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Build Tools and Resources .......................... SUCCESS [02:28 min]
[INFO] Rest API Specification ............................. SUCCESS [ 0.465 s]
[INFO] Elasticsearch: Parent POM .......................... SUCCESS [02:35 min]
[INFO] Elasticsearch: Core ................................ FAILURE [05:04 min]
[INFO] Distribution: Parent POM ........................... SKIPPED
[INFO] Distribution: TAR .................................. SKIPPED
[INFO] Distribution: ZIP .................................. SKIPPED
[INFO] Distribution: Deb .................................. SKIPPED
[INFO] Plugin: Parent POM ................................. SKIPPED
[INFO] Plugin: Analysis: Japanese (kuromoji) .............. SKIPPED
[INFO] Plugin: Analysis: Smart Chinese (smartcn) .......... SKIPPED
[INFO] Plugin: Analysis: Polish (stempel) ................. SKIPPED
[INFO] Plugin: Analysis: Phonetic ......................... SKIPPED
[INFO] Plugin: Analysis: ICU .............................. SKIPPED
[INFO] Plugin: Cloud: Google Compute Engine ............... SKIPPED
[INFO] Plugin: Cloud: Azure ............................... SKIPPED
[INFO] Plugin: Cloud: AWS ................................. SKIPPED
[INFO] Plugin: Delete By Query ............................ SKIPPED
[INFO] Plugin: Discovery: Multicast ....................... SKIPPED
[INFO] Plugin: Language: Python ........................... SKIPPED
[INFO] Plugin: Language: JavaScript ....................... SKIPPED
[INFO] Plugin: Mapper: Murmur3 ............................ SKIPPED
[INFO] Plugin: Mapper: Size ............................... SKIPPED
[INFO] Plugin: JVM example ................................ SKIPPED
[INFO] Plugin: Example site ............................... SKIPPED
[INFO] QA: Parent POM ..................................... SKIPPED
[INFO] QA: Smoke Test Plugins ............................. SKIPPED
[INFO] QA: Smoke Test Multi-Node IT ....................... SKIPPED
[INFO] QA: Smoke Test Client .............................. SKIPPED
[INFO] QA: Smoke Test Command Line Params ................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:03 min
[INFO] Finished at: 2016-05-16T12:16:20+00:00
[INFO] Final Memory: 39M/435M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.carrotsearch.randomizedtesting:junit4-maven-plugin:2.1.16:junit4 (tests) on project elasticsearch: There were test failures: 516 suites, 2943 tests, 3 failures, 10 ignored (10 assumptions) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :elasticsearch

Additional Info:
I think this is similar issue as that of #3674.
Can someone please guide me on this, any pointers would be useful.

[s1monw]

which branch are you trying to build? Our tests are green on the 2.x end?

[meghalidhoble]

This is with branch 2.3.
Also I missed to mention that I am observing this on the powerPC platform.

[s1monw]

This is with branch 2.3.
Also I missed to mention that I am observing this on the powerPC platform.

ok can you provide the actual failures you are seeing?

[meghalidhoble]

Here are the details of failure log:

at java.lang.Thread.run(Thread.java:745)
2> REPRODUCE WITH: mvn test -Pdev -pl org.elasticsearch:elasticsearch -Dtests.seed=3E0503E7B91CDFFF -Dtests.class=org.elasticsearch.plugins.PluginManagerPermissionTests -Dtests.method="testThatNonWritablePluginsDirectoryLeavesNoLeftOver" -Des.logger.level=ERROR -Dtests.assertion.disabled=false -Dtests.security.manager=true -Dtests.heap.size=512m -Dtests.locale=lt -Dtests.timezone=Europe/Ljubljana
FAILURE 0.65s J1 | PluginManagerPermissionTests.testThatNonWritablePluginsDirectoryLeavesNoLeftOver <<<
Throwable #1: java.lang.AssertionError: Expected IOException due to read-only plugins/ directory
at __randomizedtesting.SeedInfo.seed([3E0503E7B91CDFFF:C0C0DB6EA268E674]:0)
at org.elasticsearch.plugins.PluginManagerPermissionTests.testThatNonWritablePluginsDirectoryLeavesNoLeftOver(PluginManagerPermissionTests.java:172)
at java.lang.Thread.run(Thread.java:745)
2> REPRODUCE WITH: mvn test -Pdev -pl org.elasticsearch:elasticsearch -Dtests.seed=3E0503E7B91CDFFF -Dtests.class=org.elasticsearch.plugins.PluginManagerPermissionTests -Dtests.method="testThatUnaccessibleBinDirectoryAbortsPluginInstallation" -Des.logger.level=ERROR -Dtests.assertion.disabled=false -Dtests.security.manager=true -Dtests.heap.size=512m -Dtests.locale=lt -Dtests.timezone=Europe/Ljubljana
FAILURE 0.71s J1 | PluginManagerPermissionTests.testThatUnaccessibleBinDirectoryAbortsPluginInstallation <<<
Throwable #1: java.lang.AssertionError: Expected IOException but did not happen
at __randomizedtesting.SeedInfo.seed([3E0503E7B91CDFFF:B9027C256E2A9041]:0)
at org.elasticsearch.plugins.PluginManagerPermissionTests.testThatUnaccessibleBinDirectoryAbortsPluginInstallation(PluginManagerPermissionTests.java:89)
at java.lang.Thread.run(Thread.java:745)
2> NOTE: leaving temporary files on disk at: /root/meghali/elastic-search-port/elasticsearch/core/target/J1/temp/org.elasticsearch.plugins.PluginManagerPermissionTests_3E0503E7B91CDFFF-001
2> NOTE: test params are: codec=Asserting(Lucene54): {}, docValues:{}, sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=lt, timezone=Europe/Ljubljana
2> NOTE: Linux 3.10.0-210.ael7a.ppc64le ppc64le/Oracle Corporation 1.8.0_91 (64-bit)/cpus=4,threads=1,free=337761568,total=528482304
2> NOTE: All tests run in this JVM: [PropertiesSettingsLoaderTests, BytesReferenceTests, NettySizeHeaderFrameDecoderTests, GeoDistanceTests, ChannelsTests, SimpleNumericTests, BlobStoreTests, PrioritizedExecutorsTests, EsExecutorsTests, SpecialPermissionTests, FieldStatsRequestTests, JavaVersionTests, DoubleIndexingDocTests, MultiPercolatorRequestTests, TranslogVersionTests, IndicesServiceTests, CompoundAnalysisTests, XContentBuilderTests, SameShardRoutingTests, SortedSetDVStringFieldDataTests, RoutingFieldTypeTests, NetworkModuleTests, DiskThresholdDeciderTests, JsonSettingsLoaderTests, InternalEngineSettingsTests, ASCIIFoldingTokenFilterFactoryTests, CopyOnWriteHashSetTests, RetryTests, SingleShardOneReplicaRoutingTests, HppcMapsTests, DefaultSourceMappingTests, IndexStoreBWCTests, ShingleTokenFilterFactoryTests, PutWarmerRequestTests, SimpleStringMappingTests, RestUtilsTests, ExpectedShardSizeAllocationTests, UpdateMappingTests, WildcardExpressionResolverTests, PriorityComparatorTests, StemmerTokenFilterFactoryTests, SnowballAnalyzerTests, FieldNamesFieldMapperTests, PrimaryElectionRoutingTests, VersionsTests, IndexFieldTypeTests, TimestampMappingTests, CodecTests, XContentHelperTests, BytesRefHashTests, ClusterStateToStringTests, CountDownTests, InternalSearchHitTests, ESExceptionTests, SimpleLuceneTests, Log4jESLoggerTests, NullValueTests, FieldDataCacheTests, RoutingServiceTests, DeflateCompressedStreamTests, SnapshotRequestsTests, BytesRestResponseTests, DeadNodesAllocationTests, SloppyMathTests, NettyTransportTests, JsonPathTests, PipelineAggregationHelperTests, DistanceUnitTests, JvmStatsTests, RestTableTests, RestTestParserTests, ClusterStateDiffPublishingTests, BigArraysTests, ThreadPoolTypeSettingsValidatorTests, CancellableThreadsTests, WordDelimiterTokenFilterFactoryTests, TenShardsOneReplicaRoutingTests, SimilarityTests, KeyedLockTests, NettyScheduledPingTests, TimestampFieldTypeTests, GatewayMetaStateTests, MetaDataIndexUpgradeServiceTests, DeflateXContentTests, IndexSearcherWrapperTests, NetworkServiceTests, CustomSeparatorBreakIteratorTests, BalanceUnbalancedClusterTests, CustomPassageFormatterTests, SimpleExternalMappingTests, GeoHashGridParserTests, JNANativesTests, NettyTransportMultiPortTests, IndexCacheModuleTests, FailedShardsRoutingTests, ShortFieldTypeTests, IndexStoreTests, SlicedInputStreamTests, UidFieldTypeTests, BulkShardRequestTests, AllFieldTypeTests, SizeValueTests, CommitPointsTests, IndexCacheableQueryTests, VectorHighlighterTests, ExtensionPointTests, RecoverySettingsTests, ClusterBlockTests, TerminalTests, BlendedTermQueryTests, FunctionScoreEquivalenceTests, HeadersAndContextCopyClientTests, AwarenessAllocationTests, ByteSizeValueTests, TransportBroadcastByNodeActionTests, ShadowEngineTests, PathTests, PostingsFormatTests, NettyHttpPublishPortTests, IndicesQueryCacheTests, SortParserTests, TribeServiceTests, ClusterChangedEventTests, ClusterRebalanceRoutingTests, AnalysisFactoryTests, JavaMultiFieldMergeTests, CountRequestBuilderTests, MultiDataPathUpgraderTests, RecoveriesCollectionTests, TemplateQueryBuilderTests, MetaStateServiceTests, CamelCaseFieldNameTests, BroadcastReplicationTests, RatioValueTests, CborXContentTests, NodeVersionAllocationDeciderTests, DateBackwardsCompatibilityTests, FsProbeTests, TransportReplicationActionTests, ESPolicyTests, GeoShapeQueryTests, PluginManagerPermissionTests]
Completed [423/558 (1!)] on J1 in 6.85s, 9 tests, 3 failures <<< FAILURES!

I have attached the complete log file.
log.txt

[s1monw]

weird, what filesystem are you running on?

[meghalidhoble]

It's running on xfs.

[s1monw]

we also run on XFS and Rhel 7.2 I am not sure what's going on.. I also can't reproduce this problem sorry

[wenpos]

hi, did you solve the problem? I also met the problem, thinks~

[meghalidhoble]

Hi @wenpos,
Yes, I could resolve this problem, though I am sure of the root-cause of it. There were some persmission issues when I reached to the test-case failure source code and found that its not expecting root user.
Hence I did all the steps all over again, with non-root user and it worked fine for me.
See if this helps you as well.

[wenpos]

@meghalidhoble
thank you for your advice O(∩_∩)O

[danielmitterdorfer]

Closing as is was not reproducible on our end and @meghalidhoble could solve the problem locally. Please feel free to reopen if this should appear again.

[liuxiansong-ones]

LSB Version: :core-4.1-aarch64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.9.2009 (AltArch)
Release: 7.9.2009
Codename: AltArch

Error Info:

Suite: org.elasticsearch.plugins.PluginManagerPermissionTests
2> REPRODUCE WITH: mvn test -Pdev -pl org.elasticsearch:elasticsearch -Dtests.seed=1EB42FF3162725B2 -Dtests.class=org.elasticsearch.plugins.PluginManagerPermissionTests -Dtests.method="testThatUnaccessibleBinDirectoryAbortsPluginInstallation" -Des.logger.level=ERROR -Dtests.assertion.disabled=false -Dtests.security.manager=true -Dtests.heap.size=512m -Dtests.locale=ja-JP -Dtests.timezone=America/Rosario
FAILURE 0.59s J2 | PluginManagerPermissionTests.testThatUnaccessibleBinDirectoryAbortsPluginInstallation <<<

Throwable #1: java.lang.AssertionError: Expected IOException but did not happen
at __randomizedtesting.SeedInfo.seed([1EB42FF3162725B2:99B35031C1116A0C]:0)
at org.elasticsearch.plugins.PluginManagerPermissionTests.testThatUnaccessibleBinDirectoryAbortsPluginInstallation(PluginManagerPermissionTests.java:89)
at java.lang.Thread.run(Thread.java:748)
2> REPRODUCE WITH: mvn test -Pdev -pl org.elasticsearch:elasticsearch -Dtests.seed=1EB42FF3162725B2 -Dtests.class=org.elasticsearch.plugins.PluginManagerPermissionTests -Dtests.method="testThatNonWritablePluginsDirectoryLeavesNoLeftOver" -Des.logger.level=ERROR -Dtests.assertion.disabled=false -Dtests.security.manager=true -Dtests.heap.size=512m -Dtests.locale=ja-JP -Dtests.timezone=America/Rosario
FAILURE 0.52s J2 | PluginManagerPermissionTests.testThatNonWritablePluginsDirectoryLeavesNoLeftOver <<<
Throwable #1: java.lang.AssertionError: Expected IOException due to read-only plugins/ directory
at __randomizedtesting.SeedInfo.seed([1EB42FF3162725B2:E071F77A0D531C39]:0)
at org.elasticsearch.plugins.PluginManagerPermissionTests.testThatNonWritablePluginsDirectoryLeavesNoLeftOver(PluginManagerPermissionTests.java:172)
at java.lang.Thread.run(Thread.java:748)
2> REPRODUCE WITH: mvn test -Pdev -pl org.elasticsearch:elasticsearch -Dtests.seed=1EB42FF3162725B2 -Dtests.class=org.elasticsearch.plugins.PluginManagerPermissionTests -Dtests.method="testThatUnaccessiblePluginConfigDirectoryAbortsPluginInstallation" -Des.logger.level=ERROR -Dtests.assertion.disabled=false -Dtests.security.manager=true -Dtests.heap.size=512m -Dtests.locale=ja-JP -Dtests.timezone=America/Rosario
FAILURE 0.46s J2 | PluginManagerPermissionTests.testThatUnaccessiblePluginConfigDirectoryAbortsPluginInstallation <<<
Throwable #1: java.lang.AssertionError: Expected IOException but did not happen, terminal output was [Plugins directory [/opt/es1105/elasticsearch-2.4.1/core/target/J2/temp/org.elasticsearch.plugins.PluginManagerPermissionTests_1EB42FF3162725B2-001/tempDir-021/plugins] does not exist. Creating...
, Trying file:/opt/es1105/elasticsearch-2.4.1/core/target/J2/temp/org.elasticsearch.plugins.PluginManagerPermissionTests_1EB42FF3162725B2-001/tempDir-024/fake-plugin.zip ...
, Verifying file:/opt/es1105/elasticsearch-2.4.1/core/target/J2/temp/org.elasticsearch.plugins.PluginManagerPermissionTests_1EB42FF3162725B2-001/tempDir-024/fake-plugin.zip checksums if available ...
, NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
, Installed my-plugin into /opt/es1105/elasticsearch-2.4.1/core/target/J2/temp/org.elasticsearch.plugins.PluginManagerPermissionTests_1EB42FF3162725B2-001/tempDir-021/plugins/my-plugin
]
at __randomizedtesting.SeedInfo.seed([1EB42FF3162725B2:D2C618F9913FF3EE]:0)
at org.elasticsearch.plugins.PluginManagerPermissionTests.testThatUnaccessiblePluginConfigDirectoryAbortsPluginInstallation(PluginManagerPermissionTests.java:120)
at java.lang.Thread.run(Thread.java:748)
2> NOTE: leaving temporary files on disk at: /opt/es1105/elasticsearch-2.4.1/core/target/J2/temp/org.elasticsearch.plugins.PluginManagerPermissionTests_1EB42FF3162725B2-001
2> NOTE: test params are: codec=Asserting(Lucene54): {}, docValues:{}, sim=DefaultSimilarity, locale=ja-JP, timezone=America/Rosario
2> NOTE: Linux 4.18.0-80.7.2.el7.aarch64 aarch64/Red Hat, Inc. 1.8.0_312 (64-bit)/cpus=4,threads=1,free=183446728,total=531103744
2> NOTE: All tests run in this JVM: [AllocationCommandsTests, DynamicMappingDisabledTests, NumericAnalyzerTests, BoundTransportAddressTests, BinaryMappingTests, RecoveryStateTests, PathTests, LongFieldDataTests, DiskUsageTests, BigArraysTests, VersionTypeTests, VersionTests, ScriptModesTests, FloatFieldTypeTests, TransportBroadcastByNodeActionTests, NetworkModuleTests, ConcurrentRebalanceRoutingTests, JsonVsCborTests, DiscoveryModuleTests, SizeValueTests, CountDownTests, SearchSourceBuilderTests, AnalysisModuleTests, StartedShardsRoutingTests, ScriptValuesTests, IndicesQueryCacheTests, SnapshotDeletionPolicyTests, TransportClientTests, RestApiParserFailingTests, UUIDTests, ESExceptionTests, StoredNumericValuesTests, MinDocQueryTests, OriginalIndicesTests, CompletionTokenStreamTests, StemmerTokenFilterFactoryTests, ASCIIFoldingTokenFilterFactoryTests, LoggingListenerTests, ESFileStoreTests, FuzzinessTests, RandomAllocationDeciderTests, JsonFilteringGeneratorTests, CopyOnWriteHashSetTests, PluginInfoTests, YamlXContentTests, ByteSizeValueTests, SortedSetDVStringFieldDataTests, LZFCompressedStreamTests, TranslogTests, PipelineAggregationHelperTests, MultiFieldTests, NewPathForShardTests, CamelCaseFieldNameTests, DateMathParserTests, TestSectionParserTests, SimpleAllMapperTests, ESDirectoryReaderTests, TransportClientHeadersTests, WordDelimiterTokenFilterFactoryTests, PreBuiltCharFilterFactoryFactoryTests, NodeVersionAllocationDeciderTests, AnalysisServiceTests, UidFieldTypeTests, PutWarmerRequestTests, ConcurrentRecyclerTests, DocumentParserTests, MultiPhrasePrefixQueryTests, CountRequestBuilderTests, IndicesLifecycleListenerSingleNodeTests, SyncedFlushUnitTests, ScriptParameterParserTests, AllFieldTypeTests, AggregationCollectorTests, PathMatchDynamicTemplateTests, XMoreLikeThisTests, TestMergeMapperTests, SortParserTests, BitSetFilterCacheTests, IndicesStatsTests, DoubleIndexingDocTests, BalanceConfigurationTests, StopAnalyzerTests, MetaDataMappingServiceTests, RecoveriesCollectionTests, MultiPercolatorRequestTests, KeepTypesFilterFactoryTests, ShadowEngineTests, ZenDiscoveryUnitTests, BufferedTranslogTests, OsProbeTests, VersionUtilsTests, GeoFieldDataTests, LongObjectHashMapTests, CountRequestTests, ThreadPoolTypeSettingsValidatorTests, CustomPostingsHighlighterTests, DateFieldTypeTests, CustomSeparatorBreakIteratorTests, GatewayTests, ScanContextTests, ParseMappingTypeLevelTests, VersionsTests, HunspellTokenFilterFactoryTests, IdMappingTests, ShardCoreKeyMapTests, MultiGetShardRequestTests, SignificanceHeuristicTests, DiskThresholdDeciderTests, UidTests, MurmurHash3Tests, CommitStatsTests, IndicesModuleTests, StopTokenFilterTests, CompletionPostingsFormatTests, StringFieldTypeTests, XContentMapValuesTests, DiffableTests, ClusterHealthResponsesTests, SettingsTests, ProcessProbeTests, EnableAllocationTests, QueueRecyclerTests, ElectMasterServiceTests, FieldNamesFieldMapperTests, GeoHashGridParserTests, DeflateXContentTests, XContentHelperTests, IndexingSlowLogTests, RestAnalyzeActionTests, IndexStoreTests, MissingValuesTests, BroadcastReplicationTests, LongNestedSortingTests, SearchSourceCompressTests, AllocationPriorityTests, InternalSearchHitTests, MessageDigestsTests, TermVectorsUnitTests, FileSystemUtilsTests, ClusterStateRequestTests, URIPatternTests, ScoreTypeTests, IndicesShardStoreResponseTests, YamlSettingsLoaderTests, SimpleIndexQueryParserTests, PluginManagerPermissionTests]
Completed [433/571 (1!)] on J2 in 5.32s, 9 tests, 3 failures <<< FAILURES!