Chef automate illegal_argument_exception in logs

[kraangprime]

Describe the bug

Chef automate gives us "illegal_argument_exception" in logs and we cant push any events to automate from our nodes.

To Reproduce

Expected behavior

Screenshots

Versions (please complete the following information):

• Linux 3.10.0-1160.45.1.el7.x86_64 [CI] Remove defunct comment in Expeditor configuration #1 SMP Wed Oct 13 17:20:51 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Browser
• Automate Build Number:
  Version: 2
  CLI Build: 20211125062343
  Server Build: 20211125063136

Additional context

Our initial investigation shows that this started with the latest chef-automate upgrade - https://docs.chef.io/release_notes_automate/

Excerpt from the logs

Nov 30 12:58:03 server-automate1 hab[1015]: authz-service.default(O): time="2021-11-30T12:58:03+01:00" level=info msg="Projects Authorized Query" action="ingest:unifiedEvents:create" projects="[]" resource="ingest:unifiedEvents" result="[*]" subject="[token:arg-data-collector]"
Nov 30 12:58:03 server-automate1 hab[1015]: authz-service.default(O): time="2021-11-30T12:58:03+01:00" level=info msg="Projects Authorized Query" action="ingest:unifiedEvents:create" projects="[]" resource="ingest:unifiedEvents" result="[*]" subject="[token:mfi-data-collector]"
Nov 30 12:58:03 server-automate1 hab[1015]: ingest-service.default(O): time="2021-11-30T12:58:03+01:00" level=info msg="Publishing ChefRun" buffer_size=0 message_id=c06f70d3-9263-47f6-b452-c08469d3a6f5 publisher_id=1
Nov 30 12:58:03 server-automate1 hab[1015]: ingest-service.default(O): time="2021-11-30T12:58:03+01:00" level=info msg="IngestingNode Run Date Info" entity_uuid=59e2b0ee-9ef2-4bbe-9a82-4dfad2543364
Nov 30 12:58:03 server-automate1 hab[1015]: ingest-service.default(O): time="2021-11-30T12:58:03+01:00" level=error msg="Error inserting Node run date object: elastic: Error 400 (Bad Request): failed to execute script [type=illegal_argument_exception]"
Nov 30 12:58:03 server-automate1 hab[1015]: ingest-service.default(O): time="2021-11-30T12:58:03+01:00" level=error msg="Chef run ingestion failure" error="rpc error: code = Internal desc = elastic: Error 400 (Bad Request): failed to execute script [type=illegal_argument_exception]"

Please let me know if you need more details.

[kraangprime]

Issue seems to be fixed in #6295