-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: RuntimeError: exception in Python subject: KeyError: 'data' #13
Comments
Hi Vikas! Thank you for reporting this issue! I wasn't able to reproduce it with Linux, however, it indeed reproduces when running on MacOS. We're researching right now why it happens on MacOS, and how to make this showcase work on both platforms. We will keep you updated and will be back a bit later when we understand why there's a platform-specific difference. Right now, please consider running it on Linux, if you have such a possibility. Another option is to run it in Docker, but please keep in mind that you may probably need to deal with the DinD issue because the airbyte-serverless connector uses Docker to access Airbyte connectors. |
Thank you. I tried it with GitHub connector but is this issue across all
AirByte connectors? If yes, it will stall my POC efforts - would appreciate
if it can be expedited. Thanks in advance.
…On Fri, 15 Mar 2024 at 9:59 PM, Sergey Kulik ***@***.***> wrote:
Hi Vikas!
Thank you for reporting this issue!
I wasn't able to reproduce it with Linux, however, it indeed reproduces
when running on MacOS. We're researching right now why it happens on MacOS,
and how to make this showcase work on both platforms. We will keep you
updated and will be back a bit later when we understand why there's a
platform-specific difference.
Right now, please consider running it on Linux, if you have such a
possibility. Another option is to run it in Docker, but please keep in mind
that you may probably need to deal with the DinD issue because the
airbyte-serverless connector uses Docker to access Airbyte connectors.
—
Reply to this email directly, view it on GitHub
<#13 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AUNJK34UKQIEVQHRAICEGVDYYMOWTAVCNFSM6AAAAABEYBMS3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBQGAZDGMBQGE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
After further investigation, I see that the main reason is the updated version of the protocol used in the GitHub connector. Could you please try pinning the connector version in the I'll shortly commit a fix with the compatible state processing and we will release the new version of Pathway which correctly works with the modern protocols. This release will happen at the beginning of the next week, most likely on Monday. |
Thanks Sergey. The workaround did work. Wait wait for the fix commit, to
remove the 1.6.0 version dependency.
…On Sat, Mar 16, 2024 at 12:30 AM Sergey Kulik ***@***.***> wrote:
After further investigation, I see that the main reason is the updated
version of the protocol used in the GitHub connector. Could you please try
pinning the connector version in the ./connections/github.yaml config to
airbyte/source-github:1.6.0 and check if it helped? Of course, it is a
half-measure only to make it possible to run it right now.
I'll shortly commit a fix with the compatible state processing and we will
release the new version of Pathway which correctly works with the modern
protocols. This release will happen at the beginning of the next week, most
likely on Monday.
—
Reply to this email directly, view it on GitHub
<#13 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AUNJK36NUMP564I3K3VBMDLYYNALFAVCNFSM6AAAAABEYBMS3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBQGI3TKNBXGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Although the workaround works, the pipeline breaks after some time, with the following error: RuntimeError: exception in Python subject: AirbyteSourceException: {"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "Could not read json file /mnt/temp/catalog.json: Expecting ':' delimiter: line 1 column 8192 (char 8191). Please ensure that it is a valid JSON.", "stack_trace": "Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/connector.py", line 65, in _read_json_file\n return json.loads(contents)\n File "/usr/local/lib/python3.9/json/init.py", line 346, in loads\n return _default_decoder.decode(s)\n File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File "/usr/local/lib/python3.9/json/decoder.py", line 353, in raw_decode\n obj, end = self.scan_once(s, idx)\njson.decoder.JSONDecodeError: Expecting ':' delimiter: line 1 column 8192 (char 8191)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/airbyte/integration_code/main.py", line 8, in \n run()\n File "/airbyte/integration_code/source_github/run.py", line 17, in run\n launch(source, sys.argv[1:])\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 214, in launch\n for message in source_entrypoint.run(parsed_args):\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 118, in run\n config_catalog = self.source.read_catalog(parsed_args.catalog)\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/source.py", line 91, in read_catalog\n return ConfiguredAirbyteCatalog.parse_obj(cls._read_json_file(catalog_path))\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/connector.py", line 67, in _read_json_file\n raise ValueError(f"Could not read json file {file_path}: {error}. Please ensure that it is a valid JSON.")\nValueError: Could not read json file /mnt/temp/catalog.json: Expecting ':' delimiter: line 1 column 8192 (char 8191). Please ensure that it is a valid JSON.\n", "failure_type": "system_error"} |
Hi Vikas! Thank you for another piece of valuable feedback! It looks like those failures are spurious and the reason is connected to one of the libraries we used to implement the connector. I did some research and created an issue in the related repo - you can see it linked. Besides, we can handle it gracefully on our side by implementing retries for these cases. I've done a PR for that internally and they will also be in the release I announced yesterday. |
Thanks Sergey.. I love the super fast and prompt responsiveness from your
side 😊
…On Sat, 16 Mar 2024 at 7:00 PM, Sergey Kulik ***@***.***> wrote:
Hi Vikas!
Thank you for another piece of valuable feedback!
It looks like those failures are spurious and the reason is in one of the
libraries we used to implement the connector. I did some research and
created an issue in the related repo - you can see it linked.
Besides, we can handle it gracefully on our side by implementing retries
for these cases. I've done a PR for that internally and they will also be
in the release I announced yesterday.
—
Reply to this email directly, view it on GitHub
<#13 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AUNJK3ZRSI52NFI45BGBE6LYYRCPTAVCNFSM6AAAAABEYBMS3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBRHE4DONRYGE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Hi Vikas! We've released the version with an update! It contains a compatibility fix and retries for the spurious errors I've mentioned above. Feel free to test it! |
Thanks, I'll check it out.
…On Tue, 19 Mar 2024 at 5:17 AM, Sergey Kulik ***@***.***> wrote:
Hi Vikas!
We've released the version with an update! It contains a compatibility fix
and retries for the spurious errors I've mentioned above.
Feel free to test it!
—
Reply to this email directly, view it on GitHub
<#13 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AUNJK36VHFLYREYXEQZK3Q3YY54KDAVCNFSM6AAAAABEYBMS3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBVGMZDSMJWGQ>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Hi @zxqfd555-pw , I am now getting error on 1.6.0 version and latest: Traceback (most recent call last): |
Hi Vikas! I suspect that you have a not-filled optional field in the config or an unspecified required field. At least, this part of an error makes me think that:
Could you please ensure that your config doesn't contain them? If there are optional fields you don't use, please just delete them. If there are required fields that are not filled, please fill them in. You can also refer to the config example that we give in the tutorial. Also, for the reference, here is a config I used to run it:
It's stored in |
Thanks, I tried after correcting my config file, with the latest airbyte connector version. I still get this: Traceback (most recent call last): It still does process extraction on older airbyte connector version 1.6.0. |
Please make sure you have the latest pathway version. If you use pip, can do it by calling If the version is below 0.8.4, you would need to upgrade it. It can be done, for example, by reinstalling pathway with |
Hi, Thanks, I tried below suggestions, Suggestion 1:
in Suggestion 2:
to update pathway version to However, the run is breaking after 1-2 mins, and below is the error, I'm getting. Please help !!
|
I am not sure it's related to the Pathway<->Airbyte connector. But, as I can see, you have a variable The main point to consider is that we don't control the data that comes from any of the airbyte connectors, so it can be a good idea to refer to the connector docs and the format, for example, here. You can navigate by the link for "Issues" and check the schema for the fields you're interested at. That is, for
So, null is indeed a possible variant. |
Hi @umarbasha007, have you been able to verify/resolve the issue on your side? Do any further problems persist, or should we close the issue? |
@umarbasha007 we will be closing this issue as resolved on May 13 unless we hear from you by then. |
Steps to reproduce
Getting an error when trying to run the Airbyte showcase example from here: https://pathway.com/developers/showcases/etl-python-airbyte
Relevant log output
What did you expect to happen?
Expected to pull the commits into a jsonlines file, as demonstrated.
Version
0.8.3
Docker Versions (if used)
No response
OS
MacOS
On which CPU architecture did you run Pathway?
None
The text was updated successfully, but these errors were encountered: