Stats API: moved detector count call outside transport layer and make asynchronous #108

jmazanec15 · 2020-05-05T18:58:20Z

Issue #, if available:
#107

Description of changes:
Previously, getting the detector count for the stats API was leading to timeouts. The detector count was retrieved by making a timed blocking SearchRequest in the transport layer and tallying up the number of results.

This PR changes the way in which the detector count is retrieved. It moves the logic to get the detector count to a nonblocking IndicesStatsRequest in the Rest layer. Now, the rest layer makes two asynchronous requests:

IndicesStatsRequest to get detector count (only if needed)
ADStatsRequest to get the node level stats

On response from those requests, it combines the node stats and the cluster stats together and sends the response back to the user.

In order to test, I modified existing tests and added a new one. I also manually tested on a local cluster using the command ./gradlew run -PnumNodes=3

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

wnbts · 2020-05-05T20:40:49Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/stats/ADStatsResponse.java

+
+    @Override
+    public void merge(Mergeable other) {
+        if (this == other || other == null || getClass() != other.getClass()) {


Minor. other instanceof ADStatsResponse seems better.

Good point will fix

We don't want to use instanceof since this makes a subclass relationship returns true. We want to have exact same class.

That makes sense, will change back.

wnbts · 2020-05-05T20:43:35Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/stats/ADStatsResponse.java

+
+        ADStatsResponse otherResponse = (ADStatsResponse) other;
+
+        if (otherResponse.adStatsNodesResponse != null) {


Minor. Some edge cases might be handled more clearly, such as when both have node responses/cluster stats, throw an exception if unexpected.

For this object, the expected behavior if other has both node response and cluster stats is to merge them both into the object.

wnbts · 2020-05-05T20:55:28Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/rest/RestStatsAnomalyDetectorAction.java

+        Set<String> statsToBeRetrieved = adStatsRequest.getStatsToBeRetrieved();
+        for (String statName : adStats.getClusterStats().keySet()) {
+            if (statsToBeRetrieved.contains(statName)) {
+                clusterStats.put(statName, adStats.getStats().get(statName).getValue());


Minor. Seems more consistent to use adStats.getClusterStats() as in the for loop.

a different way might be iterating the entries using map::entrySet

Good suggestion

wnbts · 2020-05-05T22:42:32Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/transport/ADStatsNodesAction.java


-    public static final ADStatsAction INSTANCE = new ADStatsAction();
+    public static final ADStatsNodesAction INSTANCE = new ADStatsNodesAction();
    public static final String NAME = "cluster:admin/ad_stats_action";


minor. should ad_stats_action be renamed to ad_stats_node_action ?

Yes it should good catch.

all of our transport actions are using prefix "cluster:admin/ad" like:

"cluster:admin/ad/result"
"cluster:admin/ad/cron"
"cluster:admin/ad/model/delete"

It would be good to have the same style.

Good call out. Will update.

wnbts · 2020-05-05T22:47:13Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/rest/RestStatsAnomalyDetectorAction.java

+     * @param channel Channel to send response
+     * @param adStatsRequest Request containing stats to be retrieved
+     */
+    public void getStats(Client client, RestChannel channel, ADStatsRequest adStatsRequest) {


minor. The get methods should be private because they are implementation not contract. Tests are done based on contract not implementation. Because implementation is not client-facing, the full documentation is optional.

Good point. I will update them.

wnbts · 2020-05-05T22:52:09Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/rest/RestStatsAnomalyDetectorAction.java

+        Set<String> statsToBeRetrieved = adStatsRequest.getStatsToBeRetrieved();
+        for (String statName : adStats.getClusterStats().keySet()) {
+            if (statsToBeRetrieved.contains(statName)) {
+                clusterStats.put(statName, adStats.getStats().get(statName).getValue());


a different way might be iterating the entries using map::entrySet

kaituo · 2020-05-05T21:30:53Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/rest/RestStatsAnomalyDetectorAction.java

+            "Unable to return AD Stats"
+        );
+
+        onGetClusterStats(client, delegateListener, adStatsRequest);


on is more of a name for listener. You can change to getClusterStats or sth similar. Same applies below.

Good point. Will update both.

kaituo · 2020-05-05T21:41:16Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/rest/RestStatsAnomalyDetectorAction.java

+        ADStatsResponse adStatsResponse = new ADStatsResponse();
+        if (adStatsRequest.getStatsToBeRetrieved().contains(StatNames.DETECTOR_COUNT.getName())) {
+            if (clusterService.state().getRoutingTable().hasIndex(AnomalyDetector.ANOMALY_DETECTORS_INDEX)) {
+                IndicesStatsRequest indicesStatsRequest = new IndicesStatsRequest();


You only need docs statistics. .setDocs(true) seems what you need.

Right that is correct. Will update.

kaituo · 2020-05-05T21:44:16Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/rest/RestStatsAnomalyDetectorAction.java

+                        .setValue(indicesStatsResponse.getIndex(AnomalyDetector.ANOMALY_DETECTORS_INDEX).getPrimaries().docs.getCount());
+                    adStatsResponse.setClusterStats(getClusterStats(adStatsRequest));
+                    listener.onResponse(adStatsResponse);
+                }, e -> listener.onFailure(new RuntimeException("Failed to get AD cluster stats: " + e))));


new RuntimeException("Failed to get AD cluster stats", e) is more common.

Got it. Will update.

kaituo · 2020-05-05T23:17:28Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/rest/RestStatsAnomalyDetectorAction.java

+                listener.onResponse(adStatsResponse);
+            }
+        } else {
+            adStatsResponse.setClusterStats(getClusterStats(adStatsRequest));


Since you didn't set a value here, would it be possible that we pick some value set previously?

This else block is only entered if the user does not want to get the detector count stat. So, it will not be retrieved.

Got it. Is SettableSupplier's value a singleton shared by all of the requests? Would we have concurrent issue like one thread is writing to it while another is reading it?

Yes SettableSupplier's value is a singleton. SettableSupplier uses AtomicLong for thread safety.

I believe this is the scenario of concern:

async request(req_1) to get detector count is made (detector count = 3)

detector is created (detector count = 4)

async request(req_2) to get detector count is made (detector count = 4)

req_2 returns and sets detector count to 4

req_1 returns and sets detector count to 3

stats are returned for req_1 with detector count = 3

stats are returned for req_2 with detector count = 3

I think this is a minor issue for a couple of reasons. (1) On each stats request, the detector count is updated so it will not put the system in a permanent state of inconsistency (2) The scenario above is low probability because detector count is not expected to be that volatile and also stats api calls are not expected to be in that tight of a window.

A workaround could be to pass the detector count to getClusterStatsMap(ADStatsRequest adStatsRequest) and have a special case for DetectorCount stat. However, I am not sure this is best practice. Do you have any other suggestions?

Agree it is a minor issue since document count is only used by humans, right?

Can you create a github issue for this?

Yes only humans will read.

#112

kaituo · 2020-05-06T03:19:07Z

src/main/java/com/amazon/opendistroforelasticsearch/ad/rest/RestStatsAnomalyDetectorAction.java

+        // Use MultiResponsesDelegateActionListener to execute 2 async requests and create the response once they finish
+        MultiResponsesDelegateActionListener<ADStatsResponse> delegateListener = new MultiResponsesDelegateActionListener<>(
+            getRestStatsListener(channel),
+            2,


Is it always 2? If customers don't want to get detector count or only want to get detector count, this value should be 1.

Yes it is always 2. We still call the listeners onResponse method even if the calls are not made. For example, here

… asynchronous (opendistro-for-elasticsearch#108)

moved detector count call outside transport layer

f10ec35

jmazanec15 requested review from kaituo and wnbts May 5, 2020 18:58

jmazanec15 changed the title ~~moved detector count call outside transport layer and make asynchronous~~ Stats API: moved detector count call outside transport layer and make asynchronous May 5, 2020

wnbts approved these changes May 5, 2020

View reviewed changes

wnbts reviewed May 5, 2020

View reviewed changes

kaituo reviewed May 5, 2020

View reviewed changes

fixed minor issues with styling

8943661

kaituo reviewed May 6, 2020

View reviewed changes

minor style fixes

568dc07

kaituo approved these changes May 6, 2020

View reviewed changes

jmazanec15 merged commit 5bc0ccd into opendistro-for-elasticsearch:master May 6, 2020

jmazanec15 added a commit to jmazanec15/anomaly-detection that referenced this pull request May 6, 2020

Stats API: moved detector count call outside transport layer and make…

7a128d9

… asynchronous (opendistro-for-elasticsearch#108)

jmazanec15 mentioned this pull request May 6, 2020

Stats API: moved detector count call outside transport layer and make… #114

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stats API: moved detector count call outside transport layer and make asynchronous #108

Stats API: moved detector count call outside transport layer and make asynchronous #108

jmazanec15 commented May 5, 2020

wnbts May 5, 2020

jmazanec15 May 5, 2020

kaituo May 6, 2020

jmazanec15 May 6, 2020

wnbts May 5, 2020

jmazanec15 May 5, 2020

wnbts May 5, 2020

wnbts May 5, 2020

jmazanec15 May 5, 2020

wnbts May 5, 2020

jmazanec15 May 5, 2020

kaituo May 6, 2020

jmazanec15 May 6, 2020

wnbts May 5, 2020

jmazanec15 May 5, 2020

wnbts May 5, 2020

kaituo May 5, 2020

jmazanec15 May 5, 2020

kaituo May 5, 2020

jmazanec15 May 5, 2020

kaituo May 5, 2020

jmazanec15 May 5, 2020

kaituo May 5, 2020

jmazanec15 May 6, 2020

kaituo May 6, 2020

jmazanec15 May 6, 2020

kaituo May 6, 2020

jmazanec15 May 6, 2020

kaituo May 6, 2020

jmazanec15 May 6, 2020


		ADStatsResponse otherResponse = (ADStatsResponse) other;

		if (otherResponse.adStatsNodesResponse != null) {

Stats API: moved detector count call outside transport layer and make asynchronous #108

Stats API: moved detector count call outside transport layer and make asynchronous #108

Conversation

jmazanec15 commented May 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment