Error on replicating issue_events table in StitchData with error "Errors during transform"

[tomasrasymas]

I am using https://app.stitchdata.com/ to replicate github data from github to bigquery, for now I was replicating only issues table, today I added issue_events table and now replication process finishes with an error Errors during transform. Log information

2020-09-01 11:28:08,154Z    tap - Traceback (most recent call last):
2020-09-01 11:28:08,154Z    tap -   File "tap-env/bin/tap-github", line 33, in <module>
2020-09-01 11:28:08,154Z    tap -     sys.exit(load_entry_point('tap-github==1.9.0', 'console_scripts', 'tap-github')())
2020-09-01 11:28:08,154Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 225, in wrapped
2020-09-01 11:28:08,154Z    tap -     return fnc(*args, **kwargs)
2020-09-01 11:28:08,154Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 983, in main
2020-09-01 11:28:08,154Z    tap -     do_sync(args.config, args.state, catalog)
2020-09-01 11:28:08,155Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 956, in do_sync
2020-09-01 11:28:08,155Z    tap -     state = sync_func(stream_schema, repo, state, mdata)
2020-09-01 11:28:08,155Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 322, in get_all_issue_events
2020-09-01 11:28:08,155Z    tap -     rec = transformer.transform(event, schemas, metadata=metadata.to_map(mdata))
2020-09-01 11:28:08,155Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/transform.py", line 125, in transform
2020-09-01 11:28:08,155Z    tap -     raise SchemaMismatch(self.errors)
2020-09-01 11:28:08,155Z    tap - singer.transform.SchemaMismatch: Errors during transform
2020-09-01 11:28:08,155Z    tap - 	issue.labels.0.description: Issue XXX does not match {'type': ['null']}
2020-09-01 11:28:08,155Z    tap - 	issue.labels.0: {'id': 11, 'color': '6abf', 'name': 'xxx', 'default': False, 'url': 'https://xxx', 'description': 'Issue xxx', 'node_id': 'xxx'} does not match {'properties': {'id': {'type': ['integer', 'null']}, 'color': {'type': ['string', 'null']}, 'name': {'type': ['string', 'null']}, 'default': {'type': ['boolean', 'null']}, 'url': {'type': ['string', 'null']}, 'description': {'type': ['null']}, 'node_id': {'type': ['string', 'null']}}, 'type': ['object', 'null']}

As I can see integration is expecting 'description': {'type': ['null']} but getting 'description': 'Issue xxx'.

What might be the fix for that?

[jcw-]

The problem is that the validation is not currently expecting description to ever be filled out on an issue label:

tap-github/tap_github/schemas/issue_events.json

Lines 457 to 461 in 7ff6a34

	"description": {
	"type": [
	"null"
	]
	},

A possible workaround would be to remove the descriptions from your issue labels - I'll put up a PR with what I believe is the fix momentarily though.

[tinydylan]

@jcw- Any progress on this issue?

[jcw-]

@tinydylan I submitted the fix and requested a review 45 days ago. I've had zero feedback or response to it. Or to my other fix I submitted.

<rant>Sorry Stitch, not going to upgrade to a paid account if this is the level of community support on a major integration I can expect.</rant>

[luandy64]

@tomasrasymas @jcw- I'm working to merge #102 to fix this soon