target-bigquery error

[visch]

2021-12-22T20:11:26.264433Z [info     ] CRITICAL ['Traceback (most recent call last):\n', '  File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/__init__.py", line 129, in main\n    for state in state_iterator:\n', '  File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/process.py", line 54, in process\n    for s in handler.handle_record_message(msg):\n', '  File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/processhandler.py", line 179, in handle_record_message\n    nr = format_record_to_schema(nr, self.bq_schema_dicts[stream])\n', '  File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/schema.py", line 363, in format_record_to_schema\n    record[k] = conversion_dict[bq_schema[k]["type"]](v)\n', "KeyError: 'RECORD'\n"] cmd_type=loader job_id=clickup-to-bigquery name=target-bigquery run_id=432d8fe3-d497-4215-acdd-46a81a5e7d7b stdio=stderr
2021-12-22T20:11:26.338162Z [error    ] Loading failed                 code=2 job_id=clickup-to-bigquery message=CRITICAL ['Traceback (most recent call last):\n', '  File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/__init__.py", line 129, in main\n    for state in state_iterator:\n', '  File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/process.py", line 54, in process\n    for s in handler.handle_record_message(msg):\n', '  File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/processhandler.py", line 179, in handle_record_message\n    nr = format_record_to_schema(nr, self.bq_schema_dicts[stream])\n', '  File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/schema.py", line 363, in format_record_to_schema\n    record[k] = conversion_dict[bq_schema[k]["type"]](v)\n', "KeyError: 'RECORD'\n"] name=meltano run_id=432d8fe3-d497-4215-acdd-46a81a5e7d7b
2021-12-22T20:11:26.339047Z [info     ] ELT could not be completed: Loader failed cmd_type=elt job_id=clickup-to-bigquery name=meltano run_id=432d8fe3-d497-4215-acdd-46a81a5e7d7b stdio=stderr
ELT could not be completed: Loader failed

From @JohannesRudolph

[visch]

Probably is a bug on target-bigquery's end but it's still curious and I'd like to be sure it's not a clickup thing.

[JohannesRudolph]

Thanks to some additional debugging info added to target-bigquery, see adswerve/target-bigquery#27, I can pin-point the problems to the folder schema in tap-clickup not matching the emitted records.

 INFO Cannot format a record for stream folder to its corresponding BigQuery schema. Details: {'record': {'id': '12933951', 'name': 'Milestone and Project Plan', 'orderindex': 17, 'override_statuses': False, 'hidden': False, 'space': {'id': '2577684', 'name': 'meshDelivery'}, 'task_count': '10', 'archived': True, 'statuses': [], 'lists': [{'id': '25670974', 'name': 'POC <customer/department>', 'orderindex': 0, 'status': None, 'priority': None, 'assignee': None, 'task_count': 10, 'due_date': None, 'start_date': None, 'space': {'id': '2577684', 'name': 'meshDelivery', 'access': True}, 'archived': False, 'override_statuses': None, 'statuses': [{'id': 'p2577684_eDZ87cTk', 'status': 'Open', 'orderindex': 0, 'color': '#d3d3d3', 'type': 'open'}, {'id': 'p2577684_Sf8kB74x', 'status': 'planned', 'orderindex': 1, 'color': '#82CB11', 'type': 'custom'}, {'id': 'p2577684_yG5b2doG', 'status': 'in progress', 'orderindex': 2, 'color': '#4194f6', 'type': 'custom'}, {'id': 'p2577684_BZKpph7f', 'status': 'review', 'orderindex': 3, 'color': '#A875FF', 'type': 'custom'}, {'id': 'p2577684_ouoISXPV', 'status': 'Closed', 'orderindex': 4, 'color': '#6bc950', 'type': 'closed'}], 'permission_level': 'create'}], 'permission_level': 'create'}, 'schema': {'properties': {'id': {'type': ['string']}, 'name': {'type': ['null', 'string']}, 'orderindex': {'type': ['null', 'integer']}, 'override_statuses': {'type': ['null', 'boolean']}, 'hidden': {'type': ['null', 'boolean']}, 'space': {'properties': {'id': {'type': ['null', 'string']}, 'name': {'type': ['null', 'string']}}, 'type': 'object'}, 'task_count': {'type': ['null', 'string', 'integer']}, 'statuses': {'items': {}, 'type': ['array', 'null']}, 'lists': {'items': {}, 'type': ['array', 'null']}, 'archived': {'type': ['null', 'boolean']}, 'permission_level': {'type': ['null', 'string']}}, 'type': 'object'}, 'bq_schema': {'id': {'type': 'STRING', 'mode': 'REQUIRED', 'policyTags': {'names': []}}, 'name': {'type': 'STRING', 'mode': 'NULLABLE', 'policyTags': {'names': []}}, 'orderindex': {'type': 'INTEGER', 'mode': 'NULLABLE', 'policyTags': {'names': []}}, 'override_statuses': {'type': 'BOOLEAN', 'mode': 'NULLABLE', 'policyTags': {'names': []}}, 'hidden': {'type': 'BOOLEAN', 'mode': 'NULLABLE', 'policyTags': {'names': []}}, 'space': {'type': 'RECORD', 'mode': 'NULLABLE', 'fields': {'id': {'type': 'STRING', 'mode': 'NULLABLE', 'policyTags': {'names': []}}, 'name': {'type': 'STRING', 'mode': 'NULLABLE', 'policyTags': {'names': []}}}}, 'task_count': {'type': 'STRING', 'mode': 'NULLABLE', 'policyTags': {'names': []}}, 'statuses': {'type': 'RECORD', 'mode': 'REPEATED', 'fields': []}, 'lists': {'type': 'RECORD', 'mode': 'REPEATED', 'fields': []}, 'archived': {'type': 'BOOLEAN', 'mode': 'NULLABLE', 'policyTags': {'names': []}}, 'permission_level': {'type': 'STRING', 'mode': 'NULLABLE', 'policyTags': {'names': []}}, '_time_extracted': {'type': 'timestamp', 'mode': 'NULLABLE', 'policyTags': {'names': []}}, '_time_loaded': {'type': 'timestamp', 'mode': 'NULLABLE', 'policyTags': {'names': []}}}} cmd_type=loader job_id=clickup name=target-bigquery-truncate run_id=2b24a283-f8d0-44b6-be73-779fa86f68cc stdio=stderr

[JohannesRudolph]

Also affected

• tasks.watchers
• tasks.checklists
• tasks.dependencies
• tasks.linked_tasks

all of these fields are declared as "type": ["array", "null"] and with no schema for items.

[woochica]

I believe this issue was fixed/handled on the target's side recently in adswerve/target-bigquery#33.

[visch]

Thank you @woochica , I'll close this