Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync loggly fork with current master #2

Merged
merged 10,000 commits into from May 12, 2016
Merged

Sync loggly fork with current master #2

merged 10,000 commits into from May 12, 2016

Conversation

@zzbennett
Copy link

@zzbennett zzbennett commented May 12, 2016

No description provided.

gmoskovicz and others added 30 commits Apr 28, 2016
Allow import of all projects in Eclipse on Windows
Gradle has a "shortcut" which omits the target and source compatibility
we set when it thinks it is not necessary (eg gradle is running on the
same version as the target compat). However, the way we compile against
java 9 is to set javac to use java 9, while gradle still runs on java 8.
This change makes -source and -target explicit for now, until these
"optimizations" can be removed from gradle.

closes elastic#18039
Build: Explicitly set target and source compatibility for javac
fix exists method for list settings when using numbered setting format
This change fixes the generation of plugin properties files when the
version changes. Before, it would not regenerate, and running integTest
would fail with an incompatibile version error.
Build: Fix plugin properties generation when version changes
add ability to disable ability to override values of existing fields in set processor
This commit addresses a checkstyle line-length violation in
SetProcessor.java.
…y dsl.

Fix a limitation that prevent from hierarchical inner hits be defined in query dsl.

Removed the nested_path, parent_child_type and query options from inner hits dsl. These options are only set by ES
upon parsing the has_child, has_parent and nested queries are using their respective query builders.

These options are still used internally, when these options are set a new private copy is created based on the
provided InnerHitBuilder and configuring either nested_path or parent_child_type and the inner query of the query builder
being used.

Closes elastic#11118
Today we use a sliced lock strategy for acquiring locks to prevent
concurrent updates to the same document. The number of sliced locks is
computed as a linear function of the number of logical
processors. Unfortunately, the probability of a collision against a
sliced lock is prone to the birthday problem and grows faster than
expected. In fact, the mathematics works out such that for a fixed
target probability of collision, the number of lock slices should grow
like the square of the number of logical processors. This is
less-than-ideal, and we can do better anyway. This commit introduces a
strategy for avoiding lock contention within the internal
engine. Ideally, we would only have lock contention if there were
concurrent updates to the same document. We can get close to this ideal
world by associating a lock with the ID of each document. This
association can be held in a concurrent hash map. Now, the JDK
ConcurrentHashMap also uses a sliced lock internally, but it has several
strategies for avoiding taking the locks and these locks are only held
for a very short period of time. This implementation associates a
reference count with the lock that is associated with a document ID and
automatically removes the document ID from the concurrent hash map when
the reference count reaches zero.

Relates elastic#18060
Add new IPv6 types to docs where it's supported
This commit modifes the EsThreadPoolTestCase#info helper method to
return null when info for the thread pool can not be found. This really
should only happen for the "same" thread pool, and so we also assert
that we only get to a place where there is no info if the thread pool
that info was requested for is in fact the "same" thread pool. Not
returning null here and instead throwing an exception would fail tests
that tried to lookup info on the "same" thread pool.
name and uuid because the cluster state may contain an active index of
the same name (but different uuid).

Closes elastic#18058
Closes elastic#18054
The source code example for the initial example was missing the correct JSON object formatting and syntax.  That has been fixed with my change.
… terms

When the termslookup (mocked in this case) doesn't return any terms, the
query used to rewrite to an empty boolean query. Now it rewrites to a
MatchNoDocsQuery. This changes the test expectation accordingly.

Closes elastic#18071
This commit removes a racy but unnecessary assertion in scaling thread
pool idle test. Namely, the main test thread can reach the removed
assertion before the last few threads in the thread pool have completed
their tasks and caused the completed tasks count on the underlying
executor to be updated. But this assertion is unnecessary. The main test
thread already waits on a latch that is only decremented immediately
before a task completes. This ensures that it was in fact the case that
every submitted task was executed.

Closes elastic#18072
Ali Beyad and others added 12 commits May 11, 2016
request host, as the request is not a cross origin.

Relates elastic#18256
This commit ensures that if CORS is enabled, then Origin headers are
checked regardless of whether the request came from a browser or not.
In the past, we only proceeded with CORS checks if the User-Agent was a
browser.
…sh get, etc) for painless.

but i have no clue how to test it, it seems this feature never worked via REST?

Should we drop the feature instead?
This commit fixes the Debian package requires clause for bash. For the
RPM it is okay to specify the binary, but for the Debian package the
package name must be specified.
painless: remove input, support params instead
painless: add debugging tool to tests
@s1monw

This comment has been minimized.

should we add dest.newPrintWriter() to forbidden APIs? I think it should even be forbidden upstream?

This comment has been minimized.

Copy link
Contributor Author

@ywelsch ywelsch replied May 12, 2016

This is code in buildSrc, which is not checked by forbiddenApis at the moment. I'll have to investigate how much effort it is to enable forbiddenApis there.

This comment has been minimized.

Copy link
Contributor

@s1monw s1monw replied May 12, 2016

oh right nevermind then

This comment has been minimized.

Copy link
Contributor

@nik9000 nik9000 replied May 12, 2016

I think #18196 makes it possible. I think this is a groovy API which might make it harder but it should be possible.

This comment has been minimized.

Copy link
Member

@jasontedor jasontedor replied May 12, 2016

It's tricky because forbidden APIs looks at compiled byte code. But since Groovy is dynamic, the compiled byte code does not look like a regular method invocation. Instead, it prepares the call site:

     493: ldc_w         #316                // int 68
     496: ldc_w         #452                // String newPrintWriter

and at the call site:

     291: aload_2
     292: ldc_w         #316                // int 68
     295: aaload
     296: aload         4
     298: invokeinterface #196,  2          // InterfaceMethod org/codehaus/groovy/runtime/callsite/CallSite.call:(Ljava/lang/Object;)Ljava/lang/Object;
     303: astore        7
     305: aload         7
     307: ldc_w         #277                // class java/io/PrintWriter
     310: invokestatic  #202                // Method org/codehaus/groovy/runtime/ScriptBytecodeAdapter.castToType:(Ljava/lang/Object;Ljava/lang/Class;)Ljava/lang/Object;
     313: checkcast     #277                // class java/io/PrintWriter
jdconrad and others added 13 commits May 12, 2016
This commit adds bats tests that the RPM and Debian packages depend on
bash.

Relates elastic#18292
Rearranges the FingerprintAnalyzer so that AsciiFolding comes earlier in the chain (after lowercasing, before stop removal, for maximum deduping power)

Closes elastic#18266
This should help the tests pass more consistently. Should.

This also removes from `?pretty` from the docs. It isn't a thing
with `// CONSOLE`.

Relates to
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-os-compatibility/os=oraclelinux/399/console
As most of our log messages are not sentences and do not end with
periods, this commit removes a period from the end of the min master
node bootstrap check log message.
Make Line Number Available in Painless
Pretty sure we're not making a brand new `/var/logs` directory when everything else goes into `/var/log`

Relates elastic#18305
This commit modifies the init.d scripts to use bash now that bash is a
required dependency.

Relates elastic#18308
@zzbennett zzbennett merged commit b7ab4a0 into loggly:master May 12, 2016
zzbennett pushed a commit that referenced this pull request May 12, 2016
This change adds a new "filter_path" parameter that can be used to filter and reduce the responses returned by the REST API of elasticsearch.

For example, returning only the shards that failed to be optimized:
```
curl -XPOST 'localhost:9200/beer/_optimize?filter_path=_shards.failed'
{"_shards":{"failed":0}}%
```

It supports multiple filters (separated by a comma):
```
curl -XGET 'localhost:9200/_mapping?pretty&filter_path=*.mappings.*.properties.name,*.mappings.*.properties.title'
```

It also supports the YAML response format. Here it returns only the `_id` field of a newly indexed document:
```
curl -XPOST 'localhost:9200/library/book?filter_path=_id' -d '---hello:\n  world: 1\n'
---
_id: "AU0j64-b-stVfkvus5-A"
```

It also supports wildcards. Here it returns only the host name of every nodes in the cluster:
```
curl -XGET 'http://localhost:9200/_nodes/stats?filter_path=nodes.*.host*'
{"nodes":{"lvJHed8uQQu4brS-SXKsNA":{"host":"portable"}}}
```

And "**" can be used to include sub fields without knowing the exact path. Here it returns only the Lucene version of every segment:
```
curl 'http://localhost:9200/_segments?pretty&filter_path=indices.**.version'
{
  "indices" : {
    "beer" : {
      "shards" : {
        "0" : [ {
          "segments" : {
            "_0" : {
              "version" : "5.2.0"
            },
            "_1" : {
              "version" : "5.2.0"
            }
          }
        } ]
      }
    }
  }
}
```

Note that elasticsearch sometimes returns directly the raw value of a field, like the _source field. If you want to filter _source fields, you should consider combining the already existing _source parameter (see Get API for more details) with the filter_path parameter like this:

```
curl -XGET 'localhost:9200/_search?pretty&filter_path=hits.hits._source&_source=title'
{
  "hits" : {
    "hits" : [ {
      "_source":{"title":"Book #2"}
    }, {
      "_source":{"title":"Book #1"}
    }, {
      "_source":{"title":"Book elastic#3"}
    } ]
  }
}
```
@hakanai

This comment has been minimized.

This method was removed without updating the "breaking changes" document to describe the replacement...

This comment has been minimized.

Copy link
Member

@javanna javanna replied Jan 20, 2017

I am not sure that this was considered public api. we do not fill a new breaking change whenever we remove a method. Maybe this comes from the fact that java api users have to depend on the whole elasticsearch and end up using methods that we think they shouldn't be using?

This comment has been minimized.

Copy link

@hakanai hakanai replied Jan 20, 2017

The class was public and the method was public, and the class is not in a package named "internal", so I consider this a "breaking API change that did not even get documented or properly deprecated to warn the user of the API."

Also, you didn't remove the toDelimitedString method, so now it's an inconsistent API - you can convert from Settings to String, but how do you get back to a Settings? What if someone was using this to serialise a Settings object? (We were.)

This comment has been minimized.

Copy link
Member

@javanna javanna replied Jan 20, 2017

The class was public and the method was public, and the class is not in a package named "internal", so I consider this a "breaking API change that did not even get documented or properly deprecated to warn the user of the API."

That's the problem. Elasticsearch is a server, not a library. Unfortunately java api users need to depend on the whole elasticsearch and may end up using classes that we don't expect them to use, but we commit only to document breaking changes on the java client api (we may have missed something there too and that we have to fix for the future). All the rest, no matter what package or visibility, is internal to us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.