Resolve Windows path problems

[mhightower83]

Moved up os.makedirs for ./core/ directory.
Strange problem creating files with file paths with spaces.
Strange that "create file" would work when the path did not contain spaces and the last folder of the path hadn't been created.

Added try/except on main to commit print buffer on traceback for context.

Additional issues with diacritics and locale character encoding for shell vs source code.
build.opt is written with the same encoding as the shell; however, the data read from the Sketch.ino.global.h is UTF-8.

should resolve #8856 (comment)

[mcspr]

>>> locale.getdefaultlocale()
<stdin>:1: DeprecationWarning: Use setlocale(), getencoding() and getlocale() instead
('en_US', 'UTF-8')

For sizes .py, I thought POpen univeral_newlines should already default to locale setting by wrapping input with TextIOWrapper.
https://docs.python.org/3/library/io.html#text-encoding

I guess we do need to source some fixed locale / encoding value by external means, but what are values for the rest of stdlib functions? Why it breaks on buildopts we understand, but where does the size .py issue come from? What are default locale & encoding values for your environment from sys / locale?

---

Some external references, since I am pretty sure both manage to work ok with most installations. PIP change was a pretty recent one, though

https://github.com/platformio/platformio-core/blob/ec5bf1b5e78cf6baa180e8efb51db4e9a30e5546/platformio/compat.py, hover 'get_filesystem_encoding' and 'get_locale_encoding' and see 'References'
https://github.com/pypa/pip/search?q=sys.getfilesystemencoding
https://github.com/pypa/pip/search?q=locale.getpreferredencoding

[mcspr]

missed one odd encoding=

Suggested change

	with open(include_fqfn, 'w', encoding=default_encoding):
	with open(include_fqfn, 'w'):

[mhightower83]

What you are suggesting is I didn't need any of this get_encoding() logic - tangent I went on. In the absence of the -X utf8, I just needed to selectively remove all the , encoding="utf-8" options where I needed the system encoding to dominate.

Since merged - do I push an update or need a new PR?

[mcspr]

I just needed to selectively remove all the , encoding="utf-8" options where I needed the system encoding to dominate.

nothing is written to file, so no need for encoding= to possibly confuse future reader(s)
python won't re-encode stuff, this is just the thing for read / write through text io wrapper

in absence of -Xutf-8, text io selects something default aka depends on version
https://github.com/python/cpython/blob/cbd192b6ff137aec25913cc80b2bf7bf1ef3755b/Modules/_io/textio.c#L1102-L1141 (our bundled version)
https://github.com/python/cpython/blob/ddb65c47b1066fec8ce7a674ebb88b167f417d84/Modules/_io/textio.c#L1126-L1132 (3.11 on any recent installation)

utf-8 output is ok, reading cli / writing stuff for external tools should have system encoding

[mhightower83]

nothing is written to file, so no need for encoding= to possibly confuse future reader(s)

Just for clarification when creating build.opt, we are reading utf-8 from dot-h files and writing with system encoding to build.opt otherwise I can get a file not found for a -include ... if the file path contains diacritics.

[mcspr]

also nit:

diacritics

any time we have characters from codepage different from ansii e.g. C:\Users\Максім in cp1251
just my guess that GCC probably does not use win32 _wfopen and widechar, so we have to use exact codepage bytes for filepath representation so system exec (that also does not encode / decode anything in args) for GCC gets proper paths

path and text have different code sections dealing with them, just to denote the difference

(I am gonna go read Python PEP 529, https://vstinner.github.io/python36-utf8-windows.html series and cpython unicodeobject.c)