Implementation of platform-independ environment manipulations

[Karlson2k]

While debugging of XBMC, I've discovered that win32 environment manipulation isn't working at all.
The problem is that win32 functions SetEnvironmentVariable/GetEnvironmentVariable and runtime functions getenv/putenv operate with separate copies of Environment. So after set (for example) of 'XBMC_HOME' with SetEnvironmentVariable, next call to getenv for 'XBMC_HOME' returns error (variable not set).
Other problem on win32 platform, that XBMC use UTF-8 for CStdString, but when it converted to 'char *' win32 use it as ANSI or OEM encoding. That can cause problems if variable value contains some national characters, as paths variables.
One more problem for win32 - different runtime libraries for third-party modules, which should be updated separately. This is done partially by win32env.c, but that implementation is buggy (new loaded libraries isn't checked, can potentially crash after library unload).
This implementation is uniform for all platforms, so we can drop a lot of "ifdef"s in code.
All UTF-8<->wide conversion is carefully implemented internally where is't needed.
Almost all strings forms used in XBMC is supported as arguments.

This PR is adds new functions support and only a few function uses (as a test), and only for win32 platform.
If it's OK, I'll create more PRs with total replace for setenv/getenv functions.

[wsoltys]

I like the idea in general but I dunno why

1. we need so much code and so many new functions
2. charsetconverter isn't used

For 2) needs to be taken care of as we use it everywhere and if there's something wrong for windows it should be fixed there. Also please replace _WIN32 by TARGET_WINDOWS.
At last I dunno if Linux/OSX boys are willing to add extra code for existing functions on their platform even though we might get rid of some ifdefs ;)

[Karlson2k]

@wsoltys Answers: :)

1. Basically I was going to implement only 'char ', 'wchar_t *', 'CStdStringA' and 'CStdStringW' versions. All those type of variables are used in XBMC and some function need internally CStdStringX versions, some - only Xchar version, so to avoid unnecessary char<->CStdString conversions I implement 4 versions of functions.
   But then I got an error when I tried to compile a simple command like "xbmc_setenv("XBMC_HOME", cstdstringvar). Compiler didn't know which functions should be used "xbmc_setenv(char *, char *)" or "xbmc_setenv(CStdStringA&, CStdStringA&)" as both conversion is possible. So I decided to implement all possible versions, trying each time to use less possible conversions.
   Anyway, this give a great flexibility on using this functions and it's not too hard to maintain all those code as huge part of functions is implemented by calling other similar functions.

2. charsetconverter is used for non-windows platform. But charset conversion is needed mostly for Windows. So I decided to use direct Windows functions for two reasons:

1. When coding for Windows, native Windows conversion should be faster.
2. This is the only way to check conversion success, as XBMC conversion functions in case of fail just return original string converted via CStdString conversion (so converted as ANSI -> wide or vice versa). In case that string contain illegal ANSI characters, returned string can be just empty so if 'value' parameter conversion failed, then xbmc_envset can delete variable instead of setting new value for it.

_WIN32 is replaced.
And of course, it make sense to update only multiplatfrom code. Pure Linux/OSX code should use their native functions.

[jmarshallnz]

Before we even get into the code, where is the bug?

[Karlson2k]

At least one bug here, in Application.cpp:

CUtil::GetHomePath(xbmcPath);
SetEnvironmentVariable("XBMC_HOME", xbmcPath.c_str());

We are trying to set "XBMC_HOME", so next call to CUtil::GetHomePath should use this environment variable instead of full detection. But "GetHomePath" uses runtime function "_wgetenv" (which one operates with separate environment copy) and never get this variable, so every time "GetHomePath" uses full detection again.
I'm sure that in XBMC a lot of places where different environment is used. For example first setting "OS" variable will be ignored by any external libs, like python as it compiled by VS2008.

As I see, XBMC code uses widely "SetEnvironmentVariable" (as it simpler than "putenv") for setting variables, but uses "getenv" for reading it (as it simpler than "GetEnvironmentVariable"). But it's wrong, as "getenv" will not see changes made by "SetEnvironmentVariable" and vice versa.

To properly change variable on win32 platform you need to update it in:

• runtime environment (by "putenv")
• process environment (by "SetEnvironmentVariable", this environment is inherited by child processes )
• other runtime versions environment used by third-party libs (each runtime uses independ copy of environment)

Currently XBMC uses in some places "win32env.c" to update environment for python, but "win32env.c" is buggy and could crash after library unloading.

If this code go to XBMC, we will need to get rid of "win32env.c".

[jmarshallnz]

Given that win32env.cpp comes directly out of PostgreSQL, I'd suggest if there are any bugs in that code, you should direct them to PostgreSQL first and foremost. Perhaps any bugs are really only due to it's use in XBMC?

If you want this accepted, remove all MultiByteToWideChar stuff, get rid of everything except char* (utf8) versions of put/get/set/unset designed so they work exactly like posix where doable.

Once the trees are cleared we might be able to see the forest.

[Karlson2k]

@jmarshallnz May be PostgreSQL don't unload libs at runtime - I don't know.
I just want to implement convenient functions for XBMC with XBMC's specific.

I removed all win32 charset conversions and split code to several commits.
4c5bd47 contains main logic, but main logic includes some CStdStringA and CStdStringW functions. The main reasons for it to explicitly return a copy of variable value (to not confuse with posix behavor as posix specify return value of getenv as modifiable, but this not possible on Windows platform with wide<->UTF-8 conversions). And CStdString is used for putenv (again - posix specify that string should be putted as later modifiable by it pointer and again it's not possible on windows).
CStdStringW is a part of main logic as some parts of XBMC use wide chars, so Environment can be modified without unnecessary wide->UTF8->wide conversions.

convertFromUtf8 and convertToUtf8 is just a wrappers for charsetconverter with additional checks and shortcuts useful mainly to Environment functions.

[Karlson2k]

@jmarshallnz @wsoltys
PR is still waiting.
Just check e777e57 at first.
This could solve number of problems and potential problems.
I'm still not sure about charsetconverter as SetEnv can be used before it initialization. May be implement win32 charset conversion at least as fallback?

[jmarshallnz]

Please drop the wide char versions - they're not required at all except for
internal to win32.

Further, you can probably drop the char * ones from the API as well and
just use const std::string &.

On Mon, Aug 27, 2012 at 10:08 AM, Karlson2k notifications@github.comwrote:

@jmarshallnz https://github.com/jmarshallnz @wsoltyshttps://github.com/wsoltys
PR is still waiting.
Just check e777e57 e777e57 at first.
This could solve number of problems and potential problems.
I'm still not sure about charsetconverter as SetEnv can be used before it
initialization. May be implement win32 charset conversion at least as
fallback?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1272#issuecomment-8038346.

[Karlson2k]

@jmarshallnz OK, wide char is really rare.
About "char *" - I just want to avoid extra char -> CStdString -> char conversion.
And the most common usage for setenv is "setenv("envvar", varvalue)", so if we accept "char *" we'll just get a pointer without any initialization/conversion/c_str.
I'll start with dropping of wide char.

[jmarshallnz]

I know, thus the suggest to accept a const std::string & which will quite happily take both.

[Karlson2k]

@jmarshallnz Rewritten.
Now uses mostly std::string and std::wstring.

[Karlson2k]

@jmarshallnz What can I do to improve this PR even more? :)

[Karlson2k]

@jmarshallnz @wsoltys
Is it time?

[jmarshallnz]

XBMC does everything in utf8. Drop all wstring functions as they're not needed.

Under which circumstances is the charset convertor not available?

[wsoltys]

Apparently a wchar version was needed: c5f50fb

[jmarshallnz]

Isn't that one just being used for utf8?

On Mon, Apr 8, 2013 at 9:15 AM, wsoltys notifications@github.com wrote:

Apparently a wchar version was needed: c5f50fbhttps://github.com/xbmc/xbmc/commit/c5f50fb41e4a8353a1817ffcd127a1706b838ccf

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1272#issuecomment-16024384
.

[wsoltys]

@chadoe can you tell us why the UTF8->wchar conversion was needed?

[chadoe]

I didn't read everything here but for the change @wsoltys is referring to:
Because there's the ansi and the wide putenv api function... and special character paths stored in utf8 converted to ansi could result in loss of characters with certain characters that users use in their windows username. Which python would then reading back from the environment (invalid paths).
But maybe some of the many later fixes makes using the wide api obsolete, I remember making use of system codepages for converting the paths later on...

[Karlson2k]

@wsoltys @jmarshallnz wchar version is mostly for win32-only code sections. It's easy to use wchar in win32 as windows doesn't support UTF-8 as input/output (except charset conversion functions).

@jmarshallnz XBMC's charset converter is not available before initialization / if shutdown in progress. Same as for CLog: https://github.com/xbmc/xbmc/blob/master/xbmc/utils/log.cpp#L226

[jmarshallnz]

Is the environment stuff even used before initialization or after shutdown? If it is, then I'd tend to use the win32 API entirely for the utf8->wide conversion. If it's not, use the charsetconverter everywhere. Mixing and matching doesn't make much sense.

XBMC doesn't use a wchar version of set/putenv at all so requires only a utf8 version (obviously wrapped to the wide API). There's no point having wrapping code that isn't used.

[Karlson2k]

Wchar is used only in https://github.com/xbmc/xbmc/blob/master/xbmc/win32/WIN32Util.cpp#L503 and https://github.com/xbmc/xbmc/blob/master/xbmc/win32/WIN32Util.cpp#L513, so wrappers can be really removed.
And I'll check initialization/deinitialization order, but it can be changed in future (or will require environment manipulation in different moment). As utf8<->wchar conversion needed only for win32 platform, it will be more future proof to use win32 conversion (and no conversion for other platforms).

Should I move all function to some class (XBMCEnv?) or to another namespace?

[jmarshallnz]

Yeah, agreed - just use the win32 API for conversion to keep it simple.

It might be useful to wrap in a (static?) class I guess. If so, keep the name of the functions simple, corresponding to the name of the posix function (SetEnv, PutEnv etc.)

[Karlson2k]

@jmarshallnz Rebased, all public wchar_t/wstring functions removed, WStdString usage removed, everything moved into static class, some functions was refactored, some rewritten.
Makefile updated. Need help on updating Mac projects.

[jmarshallnz]

Just a few cosmetic comments more than anything. One last one: I'd name it Environment.h/cpp (CEnvironment) - we don't need to know it's in Xbmc :)

[Karlson2k]

@jmarshallnz Done + few more cosmetics.

[jmarshallnz]

Looks good. Build needs testing on Linux, else drop the Makefile change. Will merge in May window.

[Karlson2k]

Small correction for Linux (replace "pop_back" with "erase" for std::string). "pop_back" is part of C++11, but for it's available on MSVC C++98.
Tested build on Ubuntu v12.10 x32.

[Karlson2k]

@jmarshallnz Is it OK for May window?

[Karlson2k]

@Memphiz Needs sync xcode projects.

[Karlson2k]

Fixed typo in comment

[Karlson2k]

@jmarshallnz Postpone until next window?

[jmarshallnz]

If you drop the Makefile change it shouldn't affect other platforms at all, right?

Or, if you've build tested on linux, the Makefile change can stay.

Either or, I'm happy to pull this in as soon as that is done (this window).

[Karlson2k]

It's build tested on Linux (Ubuntu).
I think, it's fully ready to merge. :)

[jmarshallnz]

Not according to github - needs rebase.

[Karlson2k]

@jmarshallnz, it's ready again. :)

[jmarshallnz]

Thanks :)

[Karlson2k]

@Memphiz Branch was merged, need xcode projects sync.

[Memphiz]

@Karlson2k - done - i guess it needs codechanges somewhere to make use of it?

[Karlson2k]

Those functions are mostly wrappers around system native function. Special tricks used only on win32.
So for non-win32 you can continue use system native functions in system specific code or use this class if it's more convenient.
No need to convert old system specific code.
Those functions must be used for win32 code and common code parts (for win32 and non-win32).
It's done in #2739.
@Memphiz, could your help on checking PR #2739 for ios/osx?