Adding utf-8 as a supported string encoding when dumping

[sa3dany]

I am using this library to read and write to shortcuts.vdf which Steam uses to store non-Steam game shortcuts. While testing, I found that encoded non-ASCII strings were using utf-16 which caused Steam to incorrectly parse the file.

vdf/vdf/__init__.py

Line 368 in f2fd3da

def _binary_dump_gen(obj, level=0, alt_format=False):

I propose adding a new optional parameter to binary_dumps() (with utf-16 as a default) to allow encoding using utf-8. I have tested this in my fork of this library and Steam successfully parsed the resulting vdf file.

[rossengeorgiev]

Hello @sa3dany, you are saying that if I encoded a non-ascii named shortcut.vdf steam won't like it? Only wants WIDECHARS ?

I see that the dump func encodes in ascii, while the load decodes in utf-8. Have you tried simply changing the lines below from ascii to utf-8?

vdf/vdf/__init__.py

Line 379 in f2fd3da

key = key.encode('ascii')

vdf/vdf/__init__.py

Line 393 in f2fd3da

value = value.encode('ascii') + BIN_NONE

I expect that will work, although it will make the code for BIN_WIDESTRING unreachable.

[sa3dany]

Hi @rossengeorgiev, thank you for getting back to me so quickly. Decoding works correctly. The issue is when a string contains a non-ASCII character (for example, the ™ symbol) the value.encode('ascii') call will fail and then it will be encoded using utf-16 (which Steam parses incorrectly):

vdf/vdf/__init__.py

Lines 392 to 397 in f2fd3da

	try:
	value = value.encode('ascii') + BIN_NONE
	yield BIN_STRING
	except:
	value = value.encode('utf-16') + BIN_NONE*2
	yield BIN_WIDESTRING

The change I did in my fork is that I added a new Boolean parameter called utf8 that allows encoding using utf-8 when ASCII fails.

Please let me if this makes sense. Thank you.