Bulk2 fails uploading utf-8 file

[hector-ps]

Even though the data is utf-8 encoding in the file, the job fails and returns a non-friendly message:

<h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR=', '</pre>

Notice the upload of the data is not failing, but the job gets corrupted and cannot be recovered to set its state to UploadComplete.

I'd tried inserting json.dumps in the request to upload the data:

This is the original code:

    def upload_job_data(self, job_id, data: str, content_url=None):
        """Upload job data"""
        if not data:
            raise SalesforceBulkV2LoadError("Data is required for ingest jobs")

        # performance reduction here
        data_size = len(data.encode("utf-8"))
        if data_size > MAX_INGEST_JOB_FILE_SIZE:
            raise SalesforceBulkV2LoadError(
                f"Data size {data_size} exceeds the max file size accepted by "
                "Bulk V2 (100 MB)"
                )

        url = (
                content_url
                or self._construct_request_url(job_id, False) + "/batches"
        )
        headers = self._get_headers(
            self.CSV_CONTENT_TYPE, self.JSON_CONTENT_TYPE
            )
        result = call_salesforce(
            url=url,
            method="PUT",
            session=self.session,
            headers=headers,
            data=data,
            )
        if result.status_code != http.CREATED:
            raise SalesforceBulkV2LoadError(
                f"Failed to upload job data. Error Code {result.status_code}. "
                f"Response content: {result.content}"
                )

Adding the json.dumps method:

    def upload_job_data(self, job_id, data: str, content_url=None):
        """Upload job data"""
        if not data:
            raise SalesforceBulkV2LoadError("Data is required for ingest jobs")

        # performance reduction here
        data_size = len(data.encode("utf-8"))
        if data_size > MAX_INGEST_JOB_FILE_SIZE:
            raise SalesforceBulkV2LoadError(
                f"Data size {data_size} exceeds the max file size accepted by "
                "Bulk V2 (100 MB)"
                )

        url = (
                content_url
                or self._construct_request_url(job_id, False) + "/batches"
        )
        headers = self._get_headers(
            self.CSV_CONTENT_TYPE, self.JSON_CONTENT_TYPE
            )
        result = call_salesforce(
            url=url,
            method="PUT",
            session=self.session,
            headers=headers,
            data=json.dumps(data),  # HERE!
            )
        if result.status_code != http.CREATED:
            raise SalesforceBulkV2LoadError(
                f"Failed to upload job data. Error Code {result.status_code}. "
                f"Response content: {result.content}"
                )

This, however, makes the code to not fail, but the job does not upload anything.

[mattmartin26]

Having a similar issue here - trying to use Bulk 2.0 upsert, my csv file is definitely encoded in utf-8 but I get an error messagesaying 'is not valid Latin-1.

My code:

df_affiliations.to_csv('staging_files/affiliations_staging.csv', encoding='utf-8', index=False)

affiliation_upsert = sf.bulk2.mbfc__Placement__c.upsert('staging_files/affiliations_staging.csv', external_id_field='Archdpdx_Migration_Id__c')

Error message:

UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 3307: Body ('’') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.