Character '"' is not valid at position 20 in 'PartitionKey='"A'aaa"_bbbb3',RowKey='"A'aaa"_bbbb3''.

[646826]

• Package Name: azure-data-tables
• Package Version: 12.1.0
• Operating System: Linux and Windows
• Python Version: 3.8

Describe the bug

TableTransactionError: 0:Syntax error: character '"' is not valid at position 20 in 'PartitionKey='"A'aaa"_bbbb3',RowKey='"A'aaa"_bbbb3''.
RequestId:8452bf63-8002-0099-18eb-9265f1000000
Time:2021-08-16T22:08:14.5833819Z
ErrorCode:InvalidInput
---------------------------------------------------------------------------
TableTransactionError                     Traceback (most recent call last)
/Untitled-1.py in 
     15 azure_table_client = TableServiceClient(endpoint=f"https://{account_name}.table.core.windows.net", credential=credential)
     16 azure_table_service = azure_table_client .create_table_if_not_exists(to_table_name)
---> 17 azure_table_service .submit_transaction(batch)

/usr/local/lib/python3.8/site-packages/azure/data/tables/_table_client.py in submit_transaction(self, operations, **kwargs)
    729             except AttributeError:
    730                 raise ValueError("Unrecognized operation: {}".format(operation[0]))
--> 731         return self._batch_send(*batched_requests.requests, **kwargs)  # type: ignore

/usr/local/lib/python3.8/site-packages/azure/data/tables/_base_client.py in _batch_send(self, *reqs, **kwargs)
    301                     error_message="The transaction request was too large",
    302                     error_type=RequestTooLargeError)
--> 303             raise _decode_error(
    304                 response=error_parts[0],
    305                 error_type=TableTransactionError

TableTransactionError: 0:Syntax error: character '"' is not valid at position 20 in 'PartitionKey='"A'aaa"_bbbb3',RowKey='"A'aaa"_bbbb3''.
RequestId:8452bf63-8002-0099-18eb-9265f1000000
Time:2021-08-16T22:08:14.5833819Z
ErrorCode:InvalidInput

To Reproduce
Steps to reproduce the behavior:
pip install azure-data-tables

from azure.core.credentials import AzureNamedKeyCredential
from azure.data.tables import TableServiceClient
from azure.data.tables import TableEntity

account_name=""
account_key=""
to_table_name=""

batch = []
g = TableEntity({"PartitionKey":"\"A'aaa\"_bbbb3", "RowKey":"\"A'aaa\"_bbbb3", "test":"\"A'aaa\"_bbbb3"})
batch.append(("upsert", TableEntity(g)))

credential = AzureNamedKeyCredential(account_name, account_key)
azure_table_client = TableServiceClient(endpoint=f"https://{account_name}.table.core.windows.net", credential=credential)
azure_table_service = azure_table_client.create_table_if_not_exists(to_table_name)
azure_table_service.submit_transaction(batch)

Expected behavior

• I want to create this PartitionKey: "A'aaa"_bbbb3
• I want to use this value "A'aaa"_bbbb3 for rows

Screenshots

Additional context
I was able to use these values via:

Microsoft Azure Storage Explorer version 1.20.1

and Storage Explorer (preview) at https://portal.azure.com/

#Update from me
I tried loading data through query_entities, I got this result:

{'PartitionKey': '"A\'aaa"_bbbb2',
 'RowKey': '"A\'aaa"_bbbb2',
 'test': '"A\'aaa"_bbbb2'}

After that, I tried to use the response to send this data to the server but I got the same error.

[annatisch]

Thanks for the report @646826!
Looking at the service documentation - the partition key and row key values here should be valid. So this seems like a bug in the SDK.

Have you tried the same upsert operation as a single request rather than a transaction (e.g. client.upsert_entity(...))?
Additionally - looking at your code sample - you shouldn't need to wrap your input entity in TableEntity, the following should also be valid:

batch = []
g = {"PartitionKey":"\"A'aaa\"_bbbb3", "RowKey":"\"A'aaa\"_bbbb3", "test":"\"A'aaa\"_bbbb3"}
batch.append(("upsert", g))

I will try to repro this in a test asap.

[646826]

Thanks for the report @646826!
Looking at the service documentation - the partition key and row key values here should be valid. So this seems like a bug in the SDK.

Have you tried the same upsert operation as a single request rather than a transaction (e.g. client.upsert_entity(...))?
Additionally - looking at your code sample - you shouldn't need to wrap your input entity in TableEntity, the following should also be valid:

batch = []
g = {"PartitionKey":"\"A'aaa\"_bbbb3", "RowKey":"\"A'aaa\"_bbbb3", "test":"\"A'aaa\"_bbbb3"}
batch.append(("upsert", g))

I will try to repro this in a test asap.

I spent a lot of time trying to write/update this data to a table. And I was not completely sure where the problem was, so I tried all possible options. I have added some additional information to the bug, please see above.

[annatisch]

Thanks for your patience - I was able to successfully repro this locally. Investigating the cause now :)

[annatisch]

Okay - I have found the cause for this. It's the single quote character ', the service requires that this be escaped via duplication.
For example, the entity you have in your example:

g = {"PartitionKey":"\"A'aaa\"_bbbb3", "RowKey":"\"A'aaa\"_bbbb3", "test":"\"A'aaa\"_bbbb3"}

When sent over the wire, would need to be formatted like this:

g = {"PartitionKey":"\"A''aaa\"_bbbb3", "RowKey":"\"A''aaa\"_bbbb3", "test":"\"A'aaa\"_bbbb3"}

This is working correctly for deserialization, but not serialization.
I will PR a fix this week to ship in the September release.

[annatisch]

Sorry for the delay - finally got the PR out for this fix:
#20386

Tagged @YalinLi0312 for review :)

[YalinLi0312]

Fixed.

[annatisch]

Unfortunately we have moved this release to the beginning of October. We are still waiting on cross-language consensus on whether this should be considered a breaking change, and if not, how an 'opt-out' strategy will be adopted for each SDK.

Current opt-out strategy for Python is implemented in this PR, pending further design discussions:
#20603

I have reopened this issue for now.

[annatisch]

This bug fix has now been released :)

[ghost]

Hi @646826. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ghost]

Hi @646826, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.