[SWAT-734][Internal] Fixing conversion to darwin_json_1.0

[Fl4m3Ph03n1x]

Conversions from darwin_2.0 to darwin_1.0 are broken right now.
This PR aims to fix that with a temporary quick fix that reverts the json encoder used.

Ticket: https://linear.app/v7labs/issue/SWAT-734/dataset-conversion-failed-for-ffss
Context: https://vseven.slack.com/archives/C03LKKX7D0U/p1675160311013969

[linear]

SWAT-734 Dataset conversion failed for FFSS

1. FFSS on these datasets number 1 and number 2: Using WebUI on V2 to export Darwin JSON 1.0 format
2. Data Science Team(Toshiba) on this dataset: Using WebUI on V2 to export Darwin JSON 1.0 format
3. Hudl on this dataset(still stuck): Using SDK to generate Darwin JSON 2.0 format
4. Easypicky on this dataset(not stuck anymore): Using SDK to generate Darwin JSON 1.0 format

For FFSS dataset 1:

fields @timestamp, @message, @logStream, @log
| filter job_data.payload.dataset_id = 609331 and job_data.payload.team_id = 1127
| limit 20

Returns:

** (RuntimeError) Dataset conversion failed with exit code 1, output:
(busybee 0.1.0) lib/busy_bee/utils/convert_annotations.ex:33: BusyBee.Utils.ConvertAnnotations.convert/2
(busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:151: BusyBee.Jobs.DatasetsV2.Export.do_process/1
(appsignal 2.2.4) lib/appsignal/instrumentation.ex:10: Appsignal.Instrumentation.instrument/1
(broadway 1.0.3) lib/broadway/message.ex:48: Broadway.Message.update_data/2
(busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:6: BusyBee.Jobs.DatasetsV2.Export.handle_message/3
(broadway 1.0.3) lib/broadway/topology/processor_stage.ex:156: anonymous fn/6 in Broadway.Topology.ProcessorStage.handle_messages/4
(telemetry 1.1.0) /opt/app/deps/telemetry/src/telemetry.erl:320: :telemetry.span/3
(broadway 1.0.3) lib/broadway/topology/processor_stage.ex:143: Broadway.Topology.ProcessorStage.handle_messages/4

Total results:

**CloudWatch Logs Insights**  
region: eu-west-1  
log-group-names: /ecs/darwin-production-busybee  
start-time: -172800s  
end-time: 0s  
query-string:

fields @timestamp, @message, @logstream, @log
| filter job_data.payload.dataset_id = 609331 and job_data.payload.team_id = 1127
| limit 20

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|       @timestamp        |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   @message                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |                  @logStream                  |                    @log                     |
|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------|---------------------------------------------|
| 2023-01-26 17:12:17.378 | {"application":"broadway","file":"lib/broadway/topology/processor_stage.ex","job_data":{"job_id":"5ef357b4-625b-43d6-b591-70b2049ef145","job_timestamp":1674753121750254000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"json\"","export_id":"181476","export_name":"\"v9\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"false","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"c24ae6b6-cab8-42fc-ad74-18db5ae9893b\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"5ef357b4-625b-43d6-b591-70b2049ef145","line":174,"message":"** (RuntimeError) Dataset conversion failed with exit code 1, output: \n    (busybee 0.1.0) lib/busy_bee/utils/convert_annotations.ex:33: BusyBee.Utils.ConvertAnnotations.convert/2\n    (busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:151: BusyBee.Jobs.DatasetsV2.Export.do_process/1\n    (appsignal 2.2.4) lib/appsignal/instrumentation.ex:10: Appsignal.Instrumentation.instrument/1\n    (broadway 1.0.3) lib/broadway/message.ex:48: Broadway.Message.update_data/2\n    (busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:6: BusyBee.Jobs.DatasetsV2.Export.handle_message/3\n    (broadway 1.0.3) lib/broadway/topology/processor_stage.ex:156: anonymous fn/6 in Broadway.Topology.ProcessorStage.handle_messages/4\n    (telemetry 1.1.0) /opt/app/deps/telemetry/src/telemetry.erl:320: :telemetry.span/3\n    (broadway 1.0.3) lib/broadway/topology/processor_stage.ex:143: Broadway.Topology.ProcessorStage.handle_messages/4\n","mfa":["Elixir.Broadway.Topology.ProcessorStage","handle_messages",4],"pid":"#PID<0.5304.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"error","timestamp":"2023-01-26T17:12:17.375"} | ecs/busybee/3632f5ecb1a44e0f8a9ca5cf7687d91d | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 17:12:09.592 | {"application":"busybee","file":"lib/busy_bee/jobs/datasets_v2/export.ex","job_data":{"job_id":"5ef357b4-625b-43d6-b591-70b2049ef145","job_timestamp":1674753121750254000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"json\"","export_id":"181476","export_name":"\"v9\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"false","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"c24ae6b6-cab8-42fc-ad74-18db5ae9893b\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"5ef357b4-625b-43d6-b591-70b2049ef145","line":92,"message":"Started job#5ef357b4-625b-43d6-b591-70b2049ef145 for team id: 1127, dataset: ptx_heidi_qual_class, export id: 181476, format: json","mfa":["Elixir.BusyBee.Jobs.DatasetsV2.Export","do_process",1],"pid":"#PID<0.5304.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"info","timestamp":"2023-01-26T17:12:09.590"}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | ecs/busybee/3632f5ecb1a44e0f8a9ca5cf7687d91d | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 17:12:09.591 | {"application":"broadway","file":"lib/broadway/topology/processor_stage.ex","job_data":{"job_id":"5ef357b4-625b-43d6-b591-70b2049ef145","job_timestamp":1674753121750254000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"json\"","export_id":"181476","export_name":"\"v9\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"false","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"c24ae6b6-cab8-42fc-ad74-18db5ae9893b\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"5ef357b4-625b-43d6-b591-70b2049ef145","line":174,"message":"** (RuntimeError) Dataset conversion failed with exit code 1, output: \n    (busybee 0.1.0) lib/busy_bee/utils/convert_annotations.ex:33: BusyBee.Utils.ConvertAnnotations.convert/2\n    (busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:151: BusyBee.Jobs.DatasetsV2.Export.do_process/1\n    (appsignal 2.2.4) lib/appsignal/instrumentation.ex:10: Appsignal.Instrumentation.instrument/1\n    (broadway 1.0.3) lib/broadway/message.ex:48: Broadway.Message.update_data/2\n    (busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:6: BusyBee.Jobs.DatasetsV2.Export.handle_message/3\n    (broadway 1.0.3) lib/broadway/topology/processor_stage.ex:156: anonymous fn/6 in Broadway.Topology.ProcessorStage.handle_messages/4\n    (telemetry 1.1.0) /opt/app/deps/telemetry/src/telemetry.erl:320: :telemetry.span/3\n    (broadway 1.0.3) lib/broadway/topology/processor_stage.ex:143: Broadway.Topology.ProcessorStage.handle_messages/4\n","mfa":["Elixir.Broadway.Topology.ProcessorStage","handle_messages",4],"pid":"#PID<0.5304.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"error","timestamp":"2023-01-26T17:12:09.589"} | ecs/busybee/3632f5ecb1a44e0f8a9ca5cf7687d91d | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 17:12:01.753 | {"application":"busybee","file":"lib/busy_bee/jobs/datasets_v2/export.ex","job_data":{"job_id":"5ef357b4-625b-43d6-b591-70b2049ef145","job_timestamp":1674753121750254000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"json\"","export_id":"181476","export_name":"\"v9\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"false","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"c24ae6b6-cab8-42fc-ad74-18db5ae9893b\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"5ef357b4-625b-43d6-b591-70b2049ef145","line":92,"message":"Started job#5ef357b4-625b-43d6-b591-70b2049ef145 for team id: 1127, dataset: ptx_heidi_qual_class, export id: 181476, format: json","mfa":["Elixir.BusyBee.Jobs.DatasetsV2.Export","do_process",1],"pid":"#PID<0.5304.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"info","timestamp":"2023-01-26T17:12:01.752"}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | ecs/busybee/3632f5ecb1a44e0f8a9ca5cf7687d91d | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 17:03:15.851 | {"application":"broadway","file":"lib/broadway/topology/processor_stage.ex","job_data":{"job_id":"5c914d4e-58cc-4f72-9182-b7ee329d8403","job_timestamp":1674752580501136000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"json\"","export_id":"181470","export_name":"\"v8\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"true","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"8bdd2298-e85a-4ffe-8d4d-110371edd4a1\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"5c914d4e-58cc-4f72-9182-b7ee329d8403","line":174,"message":"** (RuntimeError) Dataset conversion failed with exit code 1, output: \n    (busybee 0.1.0) lib/busy_bee/utils/convert_annotations.ex:33: BusyBee.Utils.ConvertAnnotations.convert/2\n    (busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:151: BusyBee.Jobs.DatasetsV2.Export.do_process/1\n    (appsignal 2.2.4) lib/appsignal/instrumentation.ex:10: Appsignal.Instrumentation.instrument/1\n    (broadway 1.0.3) lib/broadway/message.ex:48: Broadway.Message.update_data/2\n    (busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:6: BusyBee.Jobs.DatasetsV2.Export.handle_message/3\n    (broadway 1.0.3) lib/broadway/topology/processor_stage.ex:156: anonymous fn/6 in Broadway.Topology.ProcessorStage.handle_messages/4\n    (telemetry 1.1.0) /opt/app/deps/telemetry/src/telemetry.erl:320: :telemetry.span/3\n    (broadway 1.0.3) lib/broadway/topology/processor_stage.ex:143: Broadway.Topology.ProcessorStage.handle_messages/4\n","mfa":["Elixir.Broadway.Topology.ProcessorStage","handle_messages",4],"pid":"#PID<0.5307.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"error","timestamp":"2023-01-26T17:03:15.849"}  | ecs/busybee/9c56ca3db27745b2b8269827bb80d5cc | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 17:03:09.034 | {"application":"busybee","file":"lib/busy_bee/jobs/datasets_v2/export.ex","job_data":{"job_id":"5c914d4e-58cc-4f72-9182-b7ee329d8403","job_timestamp":1674752580501136000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"json\"","export_id":"181470","export_name":"\"v8\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"true","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"8bdd2298-e85a-4ffe-8d4d-110371edd4a1\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"5c914d4e-58cc-4f72-9182-b7ee329d8403","line":92,"message":"Started job#5c914d4e-58cc-4f72-9182-b7ee329d8403 for team id: 1127, dataset: ptx_heidi_qual_class, export id: 181470, format: json","mfa":["Elixir.BusyBee.Jobs.DatasetsV2.Export","do_process",1],"pid":"#PID<0.5307.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"info","timestamp":"2023-01-26T17:03:09.034"}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | ecs/busybee/9c56ca3db27745b2b8269827bb80d5cc | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 17:03:09.033 | {"application":"broadway","file":"lib/broadway/topology/processor_stage.ex","job_data":{"job_id":"5c914d4e-58cc-4f72-9182-b7ee329d8403","job_timestamp":1674752580501136000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"json\"","export_id":"181470","export_name":"\"v8\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"true","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"8bdd2298-e85a-4ffe-8d4d-110371edd4a1\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"5c914d4e-58cc-4f72-9182-b7ee329d8403","line":174,"message":"** (RuntimeError) Dataset conversion failed with exit code 1, output: \n    (busybee 0.1.0) lib/busy_bee/utils/convert_annotations.ex:33: BusyBee.Utils.ConvertAnnotations.convert/2\n    (busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:151: BusyBee.Jobs.DatasetsV2.Export.do_process/1\n    (appsignal 2.2.4) lib/appsignal/instrumentation.ex:10: Appsignal.Instrumentation.instrument/1\n    (broadway 1.0.3) lib/broadway/message.ex:48: Broadway.Message.update_data/2\n    (busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:6: BusyBee.Jobs.DatasetsV2.Export.handle_message/3\n    (broadway 1.0.3) lib/broadway/topology/processor_stage.ex:156: anonymous fn/6 in Broadway.Topology.ProcessorStage.handle_messages/4\n    (telemetry 1.1.0) /opt/app/deps/telemetry/src/telemetry.erl:320: :telemetry.span/3\n    (broadway 1.0.3) lib/broadway/topology/processor_stage.ex:143: Broadway.Topology.ProcessorStage.handle_messages/4\n","mfa":["Elixir.Broadway.Topology.ProcessorStage","handle_messages",4],"pid":"#PID<0.5307.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"error","timestamp":"2023-01-26T17:03:09.032"}  | ecs/busybee/9c56ca3db27745b2b8269827bb80d5cc | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 17:03:00.503 | {"application":"busybee","file":"lib/busy_bee/jobs/datasets_v2/export.ex","job_data":{"job_id":"5c914d4e-58cc-4f72-9182-b7ee329d8403","job_timestamp":1674752580501136000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"json\"","export_id":"181470","export_name":"\"v8\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"true","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"8bdd2298-e85a-4ffe-8d4d-110371edd4a1\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"5c914d4e-58cc-4f72-9182-b7ee329d8403","line":92,"message":"Started job#5c914d4e-58cc-4f72-9182-b7ee329d8403 for team id: 1127, dataset: ptx_heidi_qual_class, export id: 181470, format: json","mfa":["Elixir.BusyBee.Jobs.DatasetsV2.Export","do_process",1],"pid":"#PID<0.5307.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"info","timestamp":"2023-01-26T17:03:00.502"}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | ecs/busybee/9c56ca3db27745b2b8269827bb80d5cc | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 16:19:07.819 | {"application":"busybee","file":"lib/busy_bee/jobs/datasets_v2/export.ex","job_data":{"job_id":"676e59ae-620b-48ee-9faf-e1a33a774ade","job_timestamp":1674749940545523000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"darwin_json_2\"","export_id":"181384","export_name":"\"v8\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"true","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"882d02c3-5f89-479c-ab37-d8f787db9ab2\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"676e59ae-620b-48ee-9faf-e1a33a774ade","line":215,"message":"Export, Finished processing exports/1127/ptx_heidi_qual_class/v8.zip","mfa":["Elixir.BusyBee.Jobs.DatasetsV2.Export","do_process",1],"pid":"#PID<0.5306.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"info","timestamp":"2023-01-26T16:19:07.819"}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | ecs/busybee/0ac8579fed7b4003bf23ebf274d79a0a | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 16:19:00.547 | {"application":"busybee","file":"lib/busy_bee/jobs/datasets_v2/export.ex","job_data":{"job_id":"676e59ae-620b-48ee-9faf-e1a33a774ade","job_timestamp":1674749940545523000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"darwin_json_2\"","export_id":"181384","export_name":"\"v8\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"true","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"882d02c3-5f89-479c-ab37-d8f787db9ab2\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"676e59ae-620b-48ee-9faf-e1a33a774ade","line":92,"message":"Started job#676e59ae-620b-48ee-9faf-e1a33a774ade for team id: 1127, dataset: ptx_heidi_qual_class, export id: 181384, format: darwin_json_2","mfa":["Elixir.BusyBee.Jobs.DatasetsV2.Export","do_process",1],"pid":"#PID<0.5306.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"info","timestamp":"2023-01-26T16:19:00.547"}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | ecs/busybee/0ac8579fed7b4003bf23ebf274d79a0a | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 16:15:07.113 | {"application":"busybee","file":"lib/busy_bee/jobs/datasets_v2/export.ex","job_data":{"job_id":"e8710ebc-332e-47c1-b790-91e4407ff442","job_timestamp":1674749695154072000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"darwin_json_2\"","export_id":"181381","export_name":"\"v7\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"true","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"5129ebb1-8b8e-4f47-a442-8c4a34007fe9\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"e8710ebc-332e-47c1-b790-91e4407ff442","line":215,"message":"Export, Finished processing exports/1127/ptx_heidi_qual_class/v7.zip","mfa":["Elixir.BusyBee.Jobs.DatasetsV2.Export","do_process",1],"pid":"#PID<0.5306.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"info","timestamp":"2023-01-26T16:15:07.113"}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | ecs/busybee/0ac8579fed7b4003bf23ebf274d79a0a | 258327614892:/ecs/darwin-production-busybee |
| 2023-01-26 16:14:55.156 | {"application":"busybee","file":"lib/busy_bee/jobs/datasets_v2/export.ex","job_data":{"job_id":"e8710ebc-332e-47c1-b790-91e4407ff442","job_timestamp":1674749695154072000,"payload":{"annotation_filter":"%{}","api_url":"\"https://darwin.v7labs.com/api\"","dataset_id":"609331","dataset_slug":"\"ptx_heidi_qual_class\"","export_format":"\"darwin_json_2\"","export_id":"181381","export_name":"\"v7\"","filter":"%{\"annotation_class_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"annotator\" => %{\"in\" => nil, \"not_in\" => nil}, \"assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"current_assignee\" => %{\"in\" => nil, \"not_in\" => nil}, \"dataset_id\" => %{\"in\" => [609331]}, \"has_comments\" => %{\"is\" => nil}, \"item_id\" => %{\"in\" => nil, \"not_in\" => nil}, \"item_name\" => %{\"contains\" => nil, \"in\" => nil, \"not_contains\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"item_path\" => %{\"in\" => nil, \"not_in\" => nil, \"not_prefix\" => nil, \"prefix\" => nil}, \"reviewer\" => %{\"in\" => nil, \"not_in\" => nil}, \"status\" => %{\"in\" => [\"complete\"], \"not_in\" => nil}, \"type\" => %{\"in\" => nil, \"not_in\" => nil}, \"workflow_stage_id\" => %{\"in\" => nil, \"not_in\" => nil}}","include_authorship":"true","include_export_token":"false","storage":"%{\"bucket\" => \"darwin-production-data\", \"provider\" => \"aws\", \"region\" => \"eu-west-1\", \"role\" => nil, \"tenant_id\" => nil, \"type\" => \"internal\"}","team_id":"1127","team_slug":"\"ffss\"","token":"\"5129ebb1-8b8e-4f47-a442-8c4a34007fe9\""},"request_queue":"prio_datasets_v2_export_request_queue"},"job_id":"e8710ebc-332e-47c1-b790-91e4407ff442","line":92,"message":"Started job#e8710ebc-332e-47c1-b790-91e4407ff442 for team id: 1127, dataset: ptx_heidi_qual_class, export id: 181381, format: darwin_json_2","mfa":["Elixir.BusyBee.Jobs.DatasetsV2.Export","do_process",1],"pid":"#PID<0.5306.0> (Elixir.BusyBee.Jobs.DatasetsV2.Export.Broadway.Processor_default_0)","service":"darwin-busybee","severity":"info","timestamp":"2023-01-26T16:14:55.155"}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | ecs/busybee/0ac8579fed7b4003bf23ebf274d79a0a | 258327614892:/ecs/darwin-production-busybee |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

For dataset 2, the same issue:

** (RuntimeError) Dataset conversion failed with exit code 1, output:
(busybee 0.1.0) lib/busy_bee/utils/convert_annotations.ex:33: BusyBee.Utils.ConvertAnnotations.convert/2
(busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:151: BusyBee.Jobs.DatasetsV2.Export.do_process/1
(appsignal 2.2.4) lib/appsignal/instrumentation.ex:10: Appsignal.Instrumentation.instrument/1
(broadway 1.0.3) lib/broadway/message.ex:48: Broadway.Message.update_data/2
(busybee 0.1.0) lib/busy_bee/jobs/datasets_v2/export.ex:6: BusyBee.Jobs.DatasetsV2.Export.handle_message/3
(broadway 1.0.3) lib/broadway/topology/processor_stage.ex:156: anonymous fn/6 in Broadway.Topology.ProcessorStage.handle_messages/4
(telemetry 1.1.0) /opt/app/deps/telemetry/src/telemetry.erl:320: :telemetry.span/3
(broadway 1.0.3) lib/broadway/topology/processor_stage.ex:143: Broadway.Topology.ProcessorStage.handle_messages/4

This means that busybee failed to convert an annotation. This merits a more profound BB investigation.

[owencjones]

LGTM

[rslota]

@Fl4m3Ph03n1x Do you have example of such annotation that has non string keys? We never use non-string keys, so allowing non string keys feels like a hack, rather then solution. Maybe I'm wrong here, that's why I want to see an example.
I don't want to block this PR if it unblock clients, but I want us to be aware of tech debt that we're creating (cc @owencjones ).

[owencjones]

Sorry @rslota didn't see this: have sent you an example