Faros Performance Fork of Github connector does not work on M1 Mac

[jeckhart]

Describe the bug
Related to the Faros performant fork of GitHub connector that was made default in (#140) and released as part of v0.2.0.

The Faros fork of the Airbyte Github Source does not appear to be built for M1 Mac architectures (ie. linux/arm64). I looked around to see where I might make a PR to update it but I cannot find the source of that fork. Additionally, the same error occurs for the faros fork of the airbyte-faros-destination connector.

To Reproduce
Steps to reproduce the behavior:

1. Follow the QuickStart guide
2. In Step 2, configure the Github source
3. Run a sync and check the logs
4. Relevant log pasted below
5. Operative errors are the logline below which occurs for both farosai/airbyte-faros-destination and farosai/airbyte-github-source connectors
   Error:
   WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Expected behavior
A full sync of the configured Github repos

Screenshots and logs

2022-09-25 16:11:47 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 �[32mINFO�[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.access_token: is missing but it is required, $.option_title: must be a constant value OAuth Credentials
2022-09-25 16:11:47 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 �[32mINFO�[m i.a.a.LoggingTrackingClient(track):43 - track. version: 0.39.37-alpha, userId: 00000000-1111-2222-3333-444444444444, action: Connector Jobs, metadata: {job_type=sync, config.source.credentials.option_title=PAT Credentials, config.destination.source_specific_configs.bitbucket.application_mapping=set, namespace_definition=source, config.destination.edition_configs.segment_user_id=set, config.source.credentials.personal_access_token=set, config.destination.dry_run=false, frequency=manual, connector_source_definition_id=00000000-1111-2222-3333-444444444445, workspace_id=00000000-1111-2222-3333-444444444446, config.destination.source_specific_configs.jira.use_board_ownership=true, attempt_stage=STARTED, config.destination.edition_configs.hasura_url=set, config.destination.source_specific_configs.squadcast.application_mapping=set, attempt_id=1, connector_destination=Faros Destination, catalog.sync_mode.incremental=set, connector_destination_docker_repository=farosai/airbyte-faros-destination, table_prefix=true, config.destination.jsonata_mode=FALLBACK, workspace_name=00000000-1111-2222-3333-444444444446, config.source.start_date=set, connector_source=GitHub (fork by Faros AI), config.destination.source_specific_configs.jira.exclude_fields=set, connector_source_docker_repository=farosai/airbyte-github-source, config.destination.source_specific_configs.pagerduty.application_mapping=set, config.destination.source_specific_configs.victorops.application_mapping=set, config.source.repository=set, config.destination.invalid_record_strategy=SKIP, catalog.sync_mode.full_refresh=set, connection_id=00000000-1111-2222-3333-444444444447, config.destination.edition_configs.edition=community, job_id=1, connector_source_version=0.0.2, config.destination.source_specific_configs.victorops.application_field=set, catalog.destination_sync_mode.append=set, connector_destination_version=0.3.2, operation_count=0, config.source.page_size_for_large_streams=set, config.destination.source_specific_configs.statuspage.application_mapping=set, config.destination.edition_configs.hasura_admin_secret=set, connector_destination_definition_id=00000000-1111-2222-3333-444444444448}
2022-09-25 16:11:47 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-09-25 16:11:47 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.37-alpha
2022-09-25 16:11:47 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-github-source:0.0.2 exists...
2022-09-25 16:11:48 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-github-source:0.0.2 was found locally.
2022-09-25 16:11:48 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:48 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-github-source-check-1-0-dkhgz --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-github-source:0.0.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-github-source:0.0.2 check --config source_config.json
2022-09-25 16:11:48 �[1;31mERROR�[m i.a.c.i.LineGobbler(voidCall):82 - WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:51 �[33mWARN�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):96 - Syncing `RepositoryStats` stream isn't available for repository `github/gitignore`.
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):97 - Check succeeded
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.37-alpha
2022-09-25 16:11:51 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-faros-destination:0.3.2 exists...
2022-09-25 16:11:51 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-faros-destination:0.3.2 was found locally.
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-faros-destination-check-1-0-ppsxq --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-faros-destination:0.3.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-faros-destination:0.3.2 check --config source_config.json
2022-09-25 16:11:51 �[1;31mERROR�[m i.a.c.i.LineGobbler(voidCall):82 - WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.37-alpha
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):115 - start sync worker. job id: 1 attempt id: 0
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):127 - configured sync modes: {null.workflows=incremental - append, null.pull_requests=incremental - append, null.commits=incremental - append, null.issue_labels=full_refresh - append, null.assignees=full_refresh - append, null.organizations=full_refresh - append, null.issues=incremental - append, null.branches=full_refresh - append, null.issue_milestones=incremental - append, null.reviews=incremental - append, null.review_comments=incremental - append, null.pull_request_stats=incremental - append, null.releases=incremental - append, null.workflow_runs=full_refresh - append, null.users=full_refresh - append, null.collaborators=full_refresh - append, null.tags=full_refresh - append, null.repositories=full_refresh - append}
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.i.DefaultAirbyteDestination(start):69 - Running destination...
2022-09-25 16:11:55 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-faros-destination:0.3.2 exists...
2022-09-25 16:11:56 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-faros-destination:0.3.2 was found locally.
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-faros-destination-write-1-0-qzsue --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-faros-destination:0.3.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-faros-destination:0.3.2 write --config destination_config.json --catalog destination_catalog.json
2022-09-25 16:11:56 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-github-source:0.0.2 exists...
2022-09-25 16:11:56 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-github-source:0.0.2 was found locally.
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-github-source-read-1-0-nbdpg --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-github-source:0.0.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-github-source:0.0.2 read --config source_config.json --catalog source_catalog.json
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$7):405 - Destination output thread started.
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):169 - Waiting for source and destination threads to complete.
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):298 - Replication thread started.
2022-09-25 16:11:56 �[43mdestination�[0m > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:56 �[44msource�[0m > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:59 �[44msource�[0m > Starting syncing SourceGithub
2022-09-25 16:11:59 �[44msource�[0m > Syncing `RepositoryStats` stream isn't available for repository `github/gitignore`.
2022-09-25 16:11:59 �[44msource�[0m > Syncing `Branches` stream isn't available for repository `github/gitignore`.
2022-09-25 16:11:59 �[44msource�[0m > 'github/gitignore'
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 129, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 120, in run
    for message in generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 101, in read
    stream_instances = {s.name: s for s in self.streams(config)}
  File "/airbyte/integration_code/source_github/source.py", line 186, in streams
    default_branches, branches_to_pull = self._get_branches_data(config.get("branch", ""), repository_args)
  File "/airbyte/integration_code/source_github/source.py", line 133, in _get_branches_data
    repo_branches = [default_branches[repo]]
KeyError: 'github/gitignore'
2022-09-25 16:11:59 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):335 - Total records read: 1 (0 bytes)

Deployment

• Local Deployment
• M1 MacBookPro 2022
• MacOS 12.6 - Monterey
• Docker Engine - 20.10.17
• Docker Desktop - 4.12.0

[jeckhart]

I did manage to get this to work on my M1 Mac after discovering a bug in my project list (space vs comma separated). It seems like it works even with those containers failing to start?

[ypc-faros]

Hi @jeckhart , yes, as you noticed, it is a warning. The containers do actually work in M1 although they weren't build for M1.

[ypc-faros]

Happy to help with any other issues/questions @jeckhart. Please feel free to also join our community Slack: https://community.faros.ai/docs/slack