Permalink
Commits on Sep 7, 2015
  1. Project stopped

    dadoonet committed Sep 7, 2015
Commits on Jun 11, 2015
  1. Create branch es-1.6

    dadoonet committed Jun 11, 2015
Commits on May 26, 2015
Commits on May 22, 2015
  1. fix license headers

    rmuir committed May 22, 2015
  2. remove logging properties

    rmuir committed May 22, 2015
  3. remove tabs

    rmuir committed May 22, 2015
  4. switch to plugin pom

    rmuir committed May 22, 2015
Commits on May 21, 2015
  1. remove duplicate test config

    rmuir committed May 21, 2015
  2. engage forbidden apis

    rmuir committed May 21, 2015
Commits on May 14, 2015
Commits on May 12, 2015
  1. enable security manager in tests

    rmuir committed May 12, 2015
Commits on Mar 31, 2015
  1. Deprecate river

    **Rivers are [deprecated](https://www.elastic.co/blog/deprecating_rivers) and will be removed in the future.**
    Have a look at [stream2es](https://github.com/elastic/stream2es#wikipedia).
    
    Closes #48.
    dadoonet committed Mar 31, 2015
  2. Move to elastic owner

    dadoonet committed Mar 31, 2015
Commits on Mar 16, 2015
  1. create `es-1.5` branch

    dadoonet committed Mar 16, 2015
Commits on Feb 27, 2015
  1. Remove thread leak when stopping river

    Caught by integration tests, we could have thread leaks when closing river:
    
    ```
       >    2) Thread[id=228, name=elasticsearch[node_s3][wikipedia_slurper][T#1], state=RUNNABLE, group=TGRP-WikipediaRiverTest]
       >         at java.net.SocketInputStream.socketRead0(Native Method)
       >         at java.net.SocketInputStream.read(SocketInputStream.java:152)
       >         at java.net.SocketInputStream.read(SocketInputStream.java:122)
       >         at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
       >         at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
       >         at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
       >         at sun.net.www.MeteredStream.read(MeteredStream.java:134)
       >         at java.io.FilterInputStream.read(FilterInputStream.java:133)
       >         at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3053)
       >         at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3047)
       >         at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3035)
       >         at org.elasticsearch.river.wikipedia.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:693)
       >         at org.elasticsearch.river.wikipedia.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:282)
       >         at org.elasticsearch.river.wikipedia.bzip2.CBZip2InputStream.setupNoRandPartA(CBZip2InputStream.java:839)
       >         at org.elasticsearch.river.wikipedia.bzip2.CBZip2InputStream.setupNoRandPartB(CBZip2InputStream.java:888)
       >         at org.elasticsearch.river.wikipedia.bzip2.CBZip2InputStream.read0(CBZip2InputStream.java:205)
       >         at org.elasticsearch.river.wikipedia.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:173)
       >         at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
       >         at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
       >         at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
       >         at java.io.InputStreamReader.read(InputStreamReader.java:184)
       >         at java.io.BufferedReader.read1(BufferedReader.java:203)
       >         at java.io.BufferedReader.read(BufferedReader.java:279)
       >         at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)
       >         at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1426)
       >         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2807)
       >         at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
       >         at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
       >         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
       >         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
       >         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
       >         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
       >         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
       >         at org.elasticsearch.river.wikipedia.support.WikiXMLSAXParser.parse(WikiXMLSAXParser.java:68)
       >         at org.elasticsearch.river.wikipedia.WikipediaRiver$Parser.run(WikipediaRiver.java:192)
       >         at java.lang.Thread.run(Thread.java:745)
    ```
    
    BufferedReader is never closed.
    dadoonet committed Feb 26, 2015
Commits on Feb 18, 2015
  1. [Test] make sure internet is working before running

    It could happen that we get a network issue while running tests:
    
    ```
    java.io.IOException: Server returned HTTP response code: 503 for URL: http://dumps.wikimedia.org/frwiki/latest/frwiki-latest-pages-articles.xml.bz2
    ```
    
    We should try to limit those failures by detecting if the URL is responding before trying to create the river.
    
    Closes #44.
    dadoonet committed Feb 18, 2015
Commits on Feb 16, 2015
  1. [Test] do not use anymore maven target dir for tests

    The test framework we use creates its own data dir. We should not use anymore any maven `target` data dir.
    
    Closes #43.
    (cherry picked from commit f15799b)
    (cherry picked from commit 60240db)
    (cherry picked from commit 1338151)
    dadoonet committed Feb 16, 2015
Commits on Jan 2, 2015
Commits on Dec 14, 2014
  1. Depend on elasticsearch-parent

    To simplify plugins maintenance and provide more value in the future, we are starting to build an `elasticsearch-parent` project.
    This commit is the first step for this plugin to depend on this new `pom` maven project.
    dadoonet committed Dec 14, 2014
Commits on Dec 2, 2014
Commits on Nov 27, 2014
  1. Tests: Fix test fail

     Failed to load class setting [gateway.type] with value [none]
    
    Related elastic/elasticsearch@4540884
    johtani committed Nov 27, 2014
Commits on Nov 24, 2014
Commits on Nov 10, 2014
Commits on Nov 5, 2014
  1. upgrade to lucene 5 snapshot

    rmuir committed Nov 5, 2014
Commits on Oct 30, 2014
  1. Tests: Fix randomizedtest fail

    Closes #39
    johtani committed Oct 30, 2014
  2. Upgrade to Lucene 4.10.2

    Closes #40
    mikemccand committed with mikemccand Oct 30, 2014
Commits on Oct 9, 2014
  1. Tests: wait for bulk before closing the test

    When closing a test, our test infra removes created indices.
    
    But as the river is still processing documents, the bulk processor contains remaining document which have not been indexed yet.
    
    When the river stops, it fires a `close()` call on the bulk processor.
    This call creates again the index. Test framework throw an exception as the mock directory wrapper can not close due to write operations in progress.
    
    Logs:
    
    ```
      1> [2014-10-09 11:59:40,965][INFO ][cluster.metadata         ] [node_s0] [_river] deleting index
      1> [2014-10-09 11:59:40,982][INFO ][test.store               ] [node_s1] [_river][5] Shard state before potentially flushing is STARTED
      1> [2014-10-09 11:59:41,084][INFO ][test.store               ] [node_s3] [_river][0] Shard state before potentially flushing is STARTED
      1> [2014-10-09 11:59:41,115][INFO ][test.store               ] [node_s2] [_river][6] Shard state before potentially flushing is STARTED
      1> [2014-10-09 11:59:41,943][INFO ][river.wikipedia          ] [node_s1] [wikipedia][wikipedia] closing wikipedia river
      1> [2014-10-09 11:59:42,693][WARN ][river.wikipedia          ] [node_s1] [wikipedia][wikipedia] river was closing while processing wikipedia page [39]/[Aube (département)
      1>     ]. Operation skipped.
      1> [2014-10-09 11:59:42,768][INFO ][river.wikipedia          ] [WikipediaRiverTest#testWikipediaRiverFrench]: cleaning up after test
      1> [2014-10-09 11:59:42,769][INFO ][cluster.metadata         ] [node_s0] [wikipedia] deleting index
      1> [2014-10-09 11:59:43,587][WARN ][river.wikipedia          ] There was failures while executing bulk
      1> [2014-10-09 11:59:44,971][INFO ][cluster.metadata         ] [node_s0] [wikipedia] creating index, cause [auto(bulk api)], shards [9]/[2], mappings [_default_]
      1> [2014-10-09 11:59:45,805][INFO ][cluster.metadata         ] [node_s0] [wikipedia] update_mapping [page] (dynamic)
      1> [2014-10-09 12:00:00,466][INFO ][test.store               ] [node_s2] [wikipedia][0] MockDirectoryWrapper#close() threw exception
      1> java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still open locks: {write.lock=java.lang.RuntimeException: lock "write.lock" was not released}
      1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:709)
      1> 	at org.elasticsearch.test.store.MockDirectoryHelper$ElasticsearchMockDirectoryWrapper.close(MockDirectoryHelper.java:140)
      1> 	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAllFilesClosed(ElasticsearchAssertions.java:667)
      1> 	at org.elasticsearch.test.TestCluster.assertAfterTest(TestCluster.java:76)
      1> 	at org.elasticsearch.test.ElasticsearchIntegrationTest.afterInternal(ElasticsearchIntegrationTest.java:603)
      1> 	at org.elasticsearch.test.ElasticsearchIntegrationTest.after(ElasticsearchIntegrationTest.java:1722)
    ```
    
    Closes #37.
    (cherry picked from commit d2f869c)
    dadoonet committed Oct 9, 2014
  2. Tests: _river should have only one shard one replica

    By the way, though a `_river` index should have only one shard and one replica, our test infra builds a random number of shards. We should fix it by forcing the number of shards and replicas before starting the test. Seed: `D5BE6F4EED5380F7` for example.
    
    ```sh
    mvn clean test -Dtests.seed=D5BE6F4EED5380F7
    ```
    
    Closes #38.
    (cherry picked from commit 9945ea3)
    dadoonet committed Oct 9, 2014
  3. Update to elasticsearch 1.4.0.Beta1

    Related to #27.
    
    (cherry picked from commit 6e6a2d5)
    dadoonet committed Oct 9, 2014
  4. Tests: remove _river index before exiting test

    It could happen that when closing a test, some Lucene files are still opened.
    
    ```
    ERROR   20.6s | WikipediaRiverTest.testWikipediaRiverFrench <<<
       > Throwable #1: java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still open files: {_0.cfs=1, _1.cfs=1}
       >    at __randomizedtesting.SeedInfo.seed([BA6DF9B07D6F6CD0:845570F06FCD3422]:0)
       >    at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:701)
       >    at org.elasticsearch.test.store.MockDirectoryHelper$ElasticsearchMockDirectoryWrapper.close(MockDirectoryHelper.java:140)
       >    at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAllFilesClosed(ElasticsearchAssertions.java:664)
       >    at org.elasticsearch.test.TestCluster.assertAfterTest(TestCluster.java:76)
       >    at org.elasticsearch.test.ElasticsearchIntegrationTest.afterInternal(ElasticsearchIntegrationTest.java:602)
       >    at org.elasticsearch.test.ElasticsearchIntegrationTest.after(ElasticsearchIntegrationTest.java:1720)
    ```
    
    Cleaning the `_river` index should fix that.
    
    Closes #36.
    
    (cherry picked from commit 77dcdcf)
    dadoonet committed Oct 9, 2014