Scripting: ctx._now is different when using Bulk API compared to Update API

[pangyikhei]

Elasticsearch version (bin/elasticsearch --version): 7.2.0

Plugins installed: ["ingest-attachment", "analysis-icu"]

JVM version (java -version):
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.3+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.3+7, mixed mode)

OS version (uname -a if on a Unix-like system):
Windows Server 2016 Datacenter, Windows Server 2012 Standard, Windows 10

Description of the problem including expected versus actual behavior:
When using the Bulk API, an update operation with a script that sets a field to the value of ctx._now seems to set a numeric value around 1 billion. When using the Update API directly, the value that gets set appears to be the expected milliseconds from Unix epoch time.

It is expected for the update through the bulk API to be the same as a direct update using the Update API.

Steps to reproduce:

Using Kibana to send the requests:

PUT test/_doc/1
{
    "test" : 1565112875490
}

Using Update API:

# This works as expected, setting the test field's value to the milliseconds from epoch time.
POST test/_update/1
{
    "script" : {
        "source": "ctx._source.test = ctx._now",
        "lang": "painless"
    }
}

Response of GET test/_doc/1:

{
  "_index" : "test",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 4,
  "_seq_no" : 3,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "test" : 1565115899006
  }
}

Bulk Request:

# This seems to set an much smaller value
POST _bulk
{"update": {"_id":"1", "_index": "test"}}
{"script": {"source":"ctx._source.test = ctx._now", "lang": "painless"}}

Response of GET test/_doc/1:

{
  "_index" : "test",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 3,
  "_seq_no" : 2,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "test" : 1218165137
  }
}

Possibly related issues:
#23169
#23175

[pangyikhei]

Looks like there was a change in #39793 that uses threadPool::relativeTimeInMillis as the nowInMillisSupplier for the performOnPrimary method, while it used threadPool::absoluteTimeInMillis previously.

elasticsearch/server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAction.java

Line 120 in 48d3119

performOnPrimary(request, primary, updateHelper, threadPool::relativeTimeInMillis,

[elasticmachine]

Pinging @elastic/es-core-infra

[elasticmachine]

Pinging @elastic/es-distributed