Salesforce connector. Failing when encountering anything other than standard utf-8 characters.

[grishick]

Based on this comment in this issue there is a bug in how Salesforce connector treats characters other than standard UTF-8 encoding.

Logs from the original post:

Traceback (most recent call last):
   File "/airbyte/integration_code/source_salesforce/source.py", line 111, in read
     yield from self._read_stream(
   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 159, in _read_stream
     for record in record_iterator:
   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 248, in _read_full_refresh
     for record in records:
   File "/airbyte/integration_code/source_salesforce/streams.py", line 365, in read_records
     for count, record in self.read_with_chunks(self.download_data(url=job_full_url)):
   File "/airbyte/integration_code/source_salesforce/streams.py", line 277, in download_data
     data_file.writelines(self.filter_null_bytes(chunk.decode("utf-8")))
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 3673: unexpected end of data
 2022-06-28 13:22:23 ^[[44msource^[[0m > 'utf-8' codec can't decode byte 0xe2 in position 3673: unexpected end of data
 Traceback (most recent call last):
   File "/airbyte/integration_code/main.py", line 13, in <module>
     launch(source, sys.argv[1:])
   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 127, in launch
     for message in source_entrypoint.run(parsed_args):
   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 118, in run
     for message in generator:
   File "/airbyte/integration_code/source_salesforce/source.py", line 128, in read
     raise e
   File "/airbyte/integration_code/source_salesforce/source.py", line 111, in read
     yield from self._read_stream(
   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 159, in _read_stream
     for record in record_iterator:
   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 248, in _read_full_refresh
     for record in records:
   File "/airbyte/integration_code/source_salesforce/streams.py", line 365, in read_records
     for count, record in self.read_with_chunks(self.download_data(url=job_full_url)):
   File "/airbyte/integration_code/source_salesforce/streams.py", line 277, in download_data
     data_file.writelines(self.filter_null_bytes(chunk.decode("utf-8")))
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 3673: unexpected end of data

[marcosmarxm]

Zendesk ticket #1491 has been linked to this issue.

[marcosmarxm]

Comment made from Zendesk by Marcos Marx on 2022-07-26 at 16:55:

The open issue: #14659

[marcosmarxm]

Zendesk ticket #1522 has been linked to this issue.

[marcosmarxm]

Comment made from Zendesk by Marcos Marx on 2022-07-26 at 16:56:

There is an issue open in Github to fix it: #14659 Any updates I'll return to you.

[marcosmarxm]

Comment made from Zendesk by Marcos Marx on 2022-11-21 at 20:18:

Hello the version of Salesforce connector 1.0.25 solves the issue, can you try it?