Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Transform] Transform fails for unsigned long fields #63871

Closed
hendrikmuhs opened this issue Oct 19, 2020 · 1 comment · Fixed by #63940
Closed

[Transform] Transform fails for unsigned long fields #63871

hendrikmuhs opened this issue Oct 19, 2020 · 1 comment · Fixed by #63940

Comments

@hendrikmuhs
Copy link
Contributor

hendrikmuhs commented Oct 19, 2020

#60050 introduced a new field type unsigned_long. Transform fails in certain combinations if the source index uses a field mapped to unsigned_long.

Repro steps:

PUT test_ul
{
  "mappings": {
    "properties": {
      "metric": {
        "type": "unsigned_long"
      },
      "id": {
        "type": "keyword"
      }
    }
  }
}

POST test_ul/_doc/1
{
  "id": "a",
  "metric": 3333
}

POST test_ul/_doc/2
{
  "id": "b",
  "metric": 3352
}

POST test_ul/_doc/3
{
  "id": "c",
  "metric": 1222
}

POST test_ul/_doc/4
{
  "id": "d",
  "metric": 4452345
}

PUT _transform/ul1
{
  "source": {
    "index": "test_ul"
  },
  "dest": {
    "index": "ul1"
  }, 
  "pivot": {
    "group_by": {
      "id": {
        "terms": {
          "field": "id"
        }
      }
    },
    "aggregations": {
      "m": {
        "max": {
          "field": "metric"
        }
      },
      "a": {
        "avg": {
          "field": "metric"
        }
      }
    }
  }
}

POST _transform/ul1/_start
GET _transform/ul1/_stats

failure:

"reason" : "Failed to index documents into destination index due to permanent error: [org.elasticsearch.xpack.transform.transforms.BulkIndexingException: Bulk index experienced [4] failures and at least 1 irrecoverable [org.elasticsearch.xpack.transform.transforms.TransformException: Destination index mappings are incompatible with the transform configuration.; org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [m] of type [unsigned_long] in document with id 'YWq29udX231QplrSgrzYJUMAAAAAAAAA'. Preview of field's value: '3333.0'; java.lang.IllegalArgumentException: For input string: [3333.0].]. Other failures:; org.elasticsearch.xpack.transform.transforms.TransformException: Destination index mappings are incompatible with the transform configuration.; org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [m] of type [unsigned_long] in document with id 'YWq29udX231QplrSgrzYJUMAAAAAAAAA'. Preview of field's value: '3333.0'; java.lang.IllegalArgumentException: For input string: [3333.0].]",

Root cause: Transform indexes the output as string as it does not know that unsigned_long is a numeric type, which is due to the addition of unsigned_long as plugin.

Non-Issue: Because unsigned_long support has not been released yet, this issue is not a bug by definition. It's still important to fix.

Blocker: In order to not become a bug, I classify it as blocker.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml/Transform)

@hendrikmuhs hendrikmuhs changed the title [Transform] Transform for unsigned long fields [Transform] Transform fails for unsigned long fields Oct 19, 2020
hendrikmuhs pushed a commit to hendrikmuhs/elasticsearch that referenced this issue Oct 20, 2020
writing out integer results properly, because coerce is not
supported for unsigned_long

fixes elastic#63871
hendrikmuhs pushed a commit that referenced this issue Oct 20, 2020
add support for unsigned_long, which required a change in
writing out integer results properly, because coerce is not
supported for unsigned_long

fixes #63871
hendrikmuhs pushed a commit that referenced this issue Oct 20, 2020
add support for unsigned_long, which required a change in
writing out integer results properly, because coerce is not
supported for unsigned_long

fixes #63871
backport #63940
hendrikmuhs pushed a commit that referenced this issue Oct 20, 2020
add support for unsigned_long, which required a change in
writing out integer results properly, because coerce is not
supported for unsigned_long

fixes #63871
backport #63940
pugnascotia pushed a commit to pugnascotia/elasticsearch that referenced this issue Oct 21, 2020
add support for unsigned_long, which required a change in
writing out integer results properly, because coerce is not
supported for unsigned_long

fixes elastic#63871
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants