Fill bulk errors with nested error causes

[ebuildy]

What kind an issue is this?

• Feature Request

Issue description

Add elasticsearch error causes in error reporting in order to help issue resolution. See the todo at https://github.com/elastic/elasticsearch-hadoop/blob/master/mr/src/main/java/org/elasticsearch/hadoop/rest/bulk/BulkProcessor.java#L259.

Or maybe, append all causes with the ErrorExtractor class.

Steps to reproduce

Insert documents from Spark in elasticsearch index that will not respect the mapping, this will give bulk error:

"index": {
        "_index": "consolidation-test",
        "_type": "all",
        "_id": "Qagq-2EBTt5y_FPQwFd7",
        "status": 400,
        "error": {
          "type": "mapper_parsing_exception",
          "reason": "failed to parse",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Could not convert [version.index] to boolean",
            "caused_by": {
              "type": "illegal_argument_exception",
              "reason": "Failed to parse value [not_analyzed] as only [true] or [false] are allowed."
            }
          }
        }
      }

But elasticsearch4hadoop will only report "failed to parse", that dont help develop to understand whats really happened under the hood.

Version Info

OS: CentOS 7
JVM : 1.8
Hadoop/Spark: Spark 1.6.3
ES-Hadoop : 6.2.2
ES : 6.2.2

Feature description

[jbaiera]

I think getting all the causes when extracting the error into an exception object / error message makes more sense. The bulk error pass reason list is more to help users understand why each error handler decided not to handle the error (http code wasn't retryable, couldn't consume the error, etc)

[ebuildy]

Something like:

    private String extractErrorWithCauses(Map errorMap) {
        String error = "";

        if (errorMap.containsKey("reason")) {
            error = errorMap.get("reason").toString();

            if (errorMap.containsKey("caused_by")) {
                error += "; " + extractErrorWithCauses((Map) errorMap.get("caused_by"));
            }
        } else {
            error = errorMap.toString();
        }

        return error;
    }

To get the full "error chain" ?

[jbaiera]

I think that would work for now. The error handler api itself is still experimental for the purposes of api evolution. There hasn't been much feedback yet as to how users prefer these errors are offered to their handler code. This would be a good start to just get the full error text.

[jbaiera]

Updated the title to reflect the core idea detailed here

[jbaiera]

This is implemented now on master and 6.x branches with #1143