User Home Folder and Documents Folder are incorrectly determined which leads to problems #bug

[blujedi]

PlatformIO incorrectly determines the locations of the user's Home Folder and Documents Folder.
This leads to serious problems when the folders are not in the default situation, which is often the case when developers put these on a separate (e.g. D:) drive (separate user data from OS).
PlatformIO appears to assume that Windows has a 'Home Directory' just like Linux but Windows has not. Windows has the concept of a 'User Folder' but it is not identical to and does not behave identical to a user Home Directory on Linux (this is explained below).

WHEN DO THE ISSUES HAPPEN

• The user's Home Folder is not C:\Users\<username> (which is the default),
  but e.g. D:\Users\<username>.
• The user's Documents Folder is not C:\Users\<username>\Documents (which is the default), but has been relocated (by changing Location property) to e.g. D:\Users\<username>\Documents.

WHAT HAPPENS

• PlatformIO uses an incorrect path (drive is hard-coded) for determining the user's Home folder and then it places PlatformIO\Projects on the wrong drive/location! And in file dialogs Places/<username> will then open the wrong location. Worse: PlatformIO expects the Documents folder in the wrong drive/location.
• If PlatformIO does not find the user's document folder on the C: drive then it just creates one itself wich is incorrect and BAD!
  The Documents Folder is a special directory in Windows. Its location should be queried via a special Windows API. But PlatformIO incorrectly assumes that it is allways C:%HOMEPATH%\Documents which is WRONG!
• PlatformIO places the the PlatformIO and PlatformIO\Projects directories in the Documents folder.
  Because PlatformIO uses an incorrect location for the Documents Folder it will put all projects in the wrong drive/location and the user is UNABLE to change this incorrect PlatformIO behavior.
  PlatformIO project will be on the wrong drive!

REPRODUCTION STEPS

Requirement: You will need an additional drive to the C: drive (D: will be most logical).
Steps to reproduce:

1. You will need to install the PlatformIO IDE in Visual Studio Code from scratch so uninstall it first.
2. If C:\Users\<your useraccount name>\Documents\PlotformIO exists then rename it.
   (This can later be reversed to restore the original situation.)
3. Create D:\Users\<your useraccount name>
4. Relocate ('move') your Documents Folder C:\Users\<your useraccount name>\Documents to D:\Users\<your useraccount name>\Documents IN THE FOLLOWING WAY: Select your Documents Folder, then select properties in the context menu, then selection Location and change its value from C:\Users\<your useraccount name>\Documents to D:\Users\<your useraccount name>\Documents. Then click OK. You are then answered if all content should be moved to the new location. Select: YES. The Documents folder will then be relocated.
   (This proces can be reversed later by repeating similarly and relocate back to C:).
5. Verify that directory C:\Users\<your useraccount name>\Documents does not exist anymore.
6. Change your user Home Folder: In User Manager change your user account's User Properties / Profile tab / Home folder / Local path from an empty string to D:\Users\<your useraccount name>
   (This can be reversed later by removing the string again or change the D: to C:).
7. Now reboot or logout and login.
8. Verify that environment variable %HOMEDRIVE% now has the value D: (instead of C:).
   When you open a command prompt the current directory will now be the new Home Folder on the D: drive.
9. Install the PlatformIO IDE in Visual Studio Code.
10. Create a new project in PlatformIO in the default ('Projects') location (this also triggers the creation of the PlatformIO\Projects directories).

HOW TO VERIFY THE PROBLEMS

After performing the reproduction step:

• Now check where the project is stored. It will not be stored on the D: drive but on the C: drive!
• Now check C:\Users\<your useraccount name>.
  PlatformIO IDE (installer) has created a 'Documents' directory and it has placed the PlatformIO and Projects in there, on the C: drive!
  Notice that the real Documents Folder (which was previously relocated) is still located at the D: drive, NOT the C: drive.
• Now in the PlatformIO IDE in a file dialog (e.g. open existing project) under 'Places' try to open both locations ('useraccount name' and Projects). Both locations will point to incorrect locations on the C: drive instead of pointing to locations on the D: drive.

HOW TO FIX:

• Use a combination of both the HOMEDRIVE and HOMEPATH environment variables for determining the correct full path to the Home Folder.
  Notes: The Home Folder can point to the same location as User Folder but it can also be different and even point to a different drive.
• Determine the actual location of the Documents special folder via appropriate Windows API's.
  DO NOT make assumptions based on (non-)existence of C:\Users\<username>\Documents (nor on %HOMEDRIVE%%HOMEPATH%\Documents) because it cannot be relied upon.
  Notes: The location of Documents can be different from User Folder and different from Home Folder (meaning it does not have to be a subdirectory of them) and can even be located on a different drive.
  The (relocated) location of the Documents folder is technically unrelated to the location(s) of User Folder and Home Folder.

WHAT PLATFORMIO CURRENTLY DOES WRONG

PlatformIO appears to use Home Folder as the 'user's root folder' which is displayed in file dialogs under 'Places' as the user's username.

• PlatformIO uses only HOMEPATH for determining the Home Folder location but not HOMEDRIVE and uses a hard-coded "C:" for HOMEDRIVE instead.

• PlatformIO assumes that the Documents folder is a subdirectory of the Home Folder, which is true only in default situations but not in many common non-default situations.
  PlatformIO should use Windows API's to query for the actual location of the Documents special folder instead.

• When PlatformIO does not find the Documents folder where it expects it then it creates a Documents folder there itself (bad!). PlatformIO should never touch the Documents Folder itself and leave that to the operating system.

DETAILED DESCRIPTION OF THE UNDERLYING ISSUES

Windows does not have a well defined concept of 'User Home Directory' like Linux. Windows has a concept called the 'User Folder" which is similar but not identical to a user's Home directory on Linux (a folder is a directory).

In recent Windows versions Microsoft is hiding the user from the concept of the 'User Folder' (probably because it is inconsistent and not well defined). Windows is presenting the user special folders like Documents, Pictures, Music, Desktop, Downloads etc. instead.

PlatformIO on Windows uses the User Folder like it uses the User Home Directory on Linux.
But it determines the 'Home Folder' incorrectly. It does not take into account the drive and uses a hard-coded value for the drive instead. This does not become visible in default situations but it leads to problems in (common) non-default situations.

In recent Windows versions (Windows 8 and onward at least) there is a "Users" directory in the root of the C: drive. This is where Windows stores User Folders.
I will use a sample male user with username piouser in the rest of the text. Piouser's User Folder is: C:\Users\piouser.

The User Folder contains at least the following:

• Standard 'Special Folders' like Documents, Pictures, Videos, Music, Downloads, Desktop etc.
• A hidden AppData directory with subdirectories Local, LocalLow and Roaming where the Windows OS and applications store user-specific data like application settings, temporary files etc.
• Hidden operating system files / copy of user profile, registry settings etc.
• Unfortunately, many cross-platform applications - PlatformIO included - are not conforming to Windows guidelines for location of user related (configuration) data. Instead of storing it somewhere in the AppData tree they create a dot-prefixed directory name directly in the User Folder and store their data there instead (like on Linux). PlatformIO creates its own .PlatformIO directory (e.g. C:\Users\piouser\.PlatformIO).
  Applications that do not conform to Windows guidelines for location of user-related (configuration) data will often make their own - sometimes false - assumptions. PlatformIO included.

In practice it is often desired to separate storage of the operating system from storage of user data (documents, source code etc.') and put these on different disks. Reasons can be related to making of backups, easier repair of corrupt OS images, performance and can be related to the type and size of available disks.
Windows unfortunately does not allow a User Folder to be stored on a different disk than where the operating system is stored (C:). It cannot successfully cope with that.

Feature:
It is possible to place special folders like Documents, Pictures, Video, Videos, Downloads, Desktop etc. in a different location on a different disk! This is supported by the Windows operating system.
This is done by changing the Location property of a special folder and then letting Windows move its contents.
IMPORTANT: When taking the Documents special folder as an example, when Documents is relocated/moved (by changing its Location property) to different location e.g. on a different drive, then it will be REMOVED from its old location in the User Folder and nothing will be put in its place as a place holder, not even a symbolic link. So it is not possible to determine the new location of Documents by inspecting the User Folder, nor do any environment variables exist to assist that. The current location of the Documents folder (and other special folders) can only be determined reliably by quering the operating system via Windows API's.

(I always create a directory D:\Users to separate the storage location (disk) for user data storage from the storage location (disk) of the operating system (many Windows developers do). In that new Users directory I create a 'Home Folder' (e.g. D:\Users\piouser) and then relocate al Special Folders from my User Folder to my new Home Folder (e.g. C:\Users\piouser\Documents to D:\Users\piouser\Documents.)

If directory C:\Users\piouser\Documents is missing this should alarm an installer that the situation is not default and indicates that the Documents folder has been relocated. An application should never make it's own assumptions and always query the operating system for the actual location of standard Special Folders.
But the PlatformIO installer does NOT. If PLatformIO finds C:\Users\piouser\Documents missing then it will just create a C:\Users\piouser\Documents (ouch) and create a PlatformIO subdirectory (with Projects subdirectory) in the self-created Documents directory. This is bad practice and leads to problems for (common) non-default situations.

Feature:
User Folders are located on the C: drive in C:\Users but there is also the concept of 'Home folder'.
'Home folder' is the directory pointed to by a combination of the environment variables HOMEDRIVE and HOMEPATH.
HOMEDRIVE contains only the drive and HOMEPATH contains the path relative to the root of the drive (but not the drive). In Linux ~ refers to the full path of the user's Home directory. Not so in Windows (and there is nothing comparable to ~ on Linux).

Feature:
'Home folder' and therefore the values of HOMEDRIVE and HOMEPATH are determined by the value of: 'User properties/Profile tab/Home folder/Local path'. This value is empty by default which means that Home folder points to the same location as User Folder but they can also be different and Home folder can even point to a different drive.

Feature
Changing the value of Home folder/Local path does effect the values of HOMEDRIVE and HOMEPATH but it does not have any effect on (the use of) the User Folder, nor does it have any effect on standard Special Folders, neither in their default location nor when relocated.

So it turns out that User Folder, Home Folder and Special Folders are inconsistent and they can all point to different locations and different drives which does not make things easier.

[ivankravets]

We don't use a hardcoded path for the home directory. We use https://docs.python.org/2.7/library/os.path.html#os.path.expanduser

Please set correctly HOME and USERPROFILE environment variables.

[blujedi]

@ivankravets

Please reopen the issue. The issues (bugs) still exist.

I do not understand why the issue was closed without verifying the details and applying the provided reproduction steps.

You are referring to a Phython function that PlatformIO is using but only mentioning that does not solve any of the issues that I am addressing. It is even possible that part of the issues is caused by that Python function. Please read my above description fully and VERIFY the issues by applying the reproduction steps.

In response to your post,

First

Please set correctly HOME and USERPROFILE environment variables.

• NO: Windows is NOT LINUX! There is no HOME environment variable in Windows and there should not be. In Windows one should use a combination of HOMEDRIVE and HOMEPATH instead.
• NO: USERPROFILE is a FIXED operating system location. Users should never change the USERPROFILE environment variable.

Second

We don't use a hardcoded path for the home directory. We use https://docs.python.org/2.7/library/os.path.html#os.path.expanduser

says:

os.path.expanduser(path)
On Windows, HOME and USERPROFILE will be used if set, otherwise a combination of HOMEPATH and HOMEDRIVE will be used. An initial ~user is handled by stripping the last directory component from the created user path derived above.

• "HOME":
  There is NO HOME environment variable in Windows and there should not be!
• "and USERPROFILE will be used if set":
  The USERPROFILE environment variable is ALLWAYS set and the user should never change it!
• "otherwise a combination of HOMEDRIVE and HOMEPATH will be used":
  Well, that maybe is what it SHOULD do but it DOES NOT. Even while both HOMEDRIVE and HOMEPATH are properly set, HOMEDRIVE is not used (and probably neither HOMEPATH).
  This is a BUG that lead to creating this issue.
• "An initial ~user is handled by stripping the last directory component from the created user path derived above":
  An 'initial' user should be 'stripped' from USERPROFILE path ONLY, NOT from Home Folder (HOMEDRIVE + HOMEPATH) or anything else!
  (FYI from USERPROFILE and not from USERNAME because changing the username will not change the USERPROFILE location.)

And WORSE: PlatformIO blatantly creates a Documents folder in UserProfile when the Documents Folder is not in its default location.
PlatformIO appears NOT to query Windows for the correct Documents Folder location and make its own incorrect assumptions instead which is really BAD!
This is a BUG that lead to creating this issue.

How to fix the issues

• Use the USERPROFILE environment variable ONLY FOR determining the location of the User Profile.
  USERPROFILE is the location where the .PlatformIO directory should be stored.

• Use the SHGetKnownFolderPath Windows API (or a higher level abstraction) to correctly determine the location of the user's Documents Folder.
  This is the only correct and future proof way to determine the location of the Documents Folder.
  Stick to Windows requirements and guidelines.
  The Documents folder is where the PlatformIO and PlatformIO\Projects directories have to be placed.

• Use a combination of HOMEDRIVE and HOMEPATH to determine the location of the Home Folder.
  The Home Folder is what will be the current directory when you open a command prompt in Windows. The Home Folder is also what should be opened when you select 'Places/username in the PlatformIO IDE 'Open Project' dialog.

Avoid these Pitfalls

Note the facts below and prevent any pitfalls caused by making 'your own' (incorrect) assumptions:

• Only in a default configuration do USERPROFILE and Home Folder point to the same location!
• Only in a default configuration is the Documents Folder a subdirectory of USERPROFILE!
• The Documents Folder location is independent from the USERPROFILE location!
• The Documents Folder location is independent from the Home Folder location!
  (While it is most practical when the Documents Folder is a subdirectory of the Home Folder but it is not required.)
• The only reliable way to determine the Documents Folder location is by using the SHGetKnownFolderPath Windows API (or a higher level API that uses it).
  While it is possible to get the same information from the registry, directly accessing the registry (= implementation details) is not preferred and is not future proof.
• The Documents Folder location CANNOT be determined from environment variables, unfortunately.
• There is no equivalent for Linux's user HOME directory on Windows, While USERPROFILE looks similar, it is not identical and the implementation is different. USERPROFILE, Documents Folder and Home Folder all play a role in Windows (sometimes unclear, even messy due to historical changes).

Attached is a simple C# example (GetUserFolderLocations.cpp.txt) with instructions that is very easy to run (without requiring an IDE). It demonstrates how UserProfile, Documents Folder and Home Folder locations can be correctly determined and displays their values.
(Remove the .txt extension after downloading, I was not able to upload it with .cpp extension)

GetUserFolderLocations.cpp.txt

[ivankravets]

1. Please provide an output from python -c "import os; print os.path.expanduser('~')"
2. When creating new project please uncheck "Default location" and you will be able to use custom project folder.

[blujedi]

1. Please provide an output from python -c "import os; print os.path.expanduser('~')" D:\Users\ PioUser>python -c "import os; print os.path.expanduser('~')" C:\Users\ PioUser and D:\Users\PioUser>GetUserFolderLocations UserProfile path is: C:\Users\PioUser Home folder path is: D:\Users\PioUser Documents folder path is: D:\Users PioUser\Documents 2. When creating new project please uncheck "Default location" and you will be able to use custom project folder. Really? That would be a user experience hell. It is PlatformIO that needs to be fixed, not its users. :-)

[blujedi]

Thanks for reopening the issue.

[ivankravets]

1. As you can see, the issue with Python. PlatformIO is written in Python and we use its API

2. No need to fix something. See screenshot below

[ivankravets]

Could I close this issue?

[blujedi]

No of course you cannot close this issue.
Your suggestion is a pure user experience hell.

You should get it fixed in PlatformIO !
If this is a Python issue then contact the Python team and have them fix it
and otherwise try to search for a a different way to fix it.
Your suggestion is not a solution, is not a fix for the problems.

I like PlatformIO and I value the efforts from the PlatformIO team,
but these issues severely impact the usability of PlatformIO.

[ivankravets]

1. We use user's HOME directory, not documents folder.
2. Python handles correctly home directory, see "On Windows, HOME and USERPROFILE will be used if set, otherwise a combination of HOMEPATH and HOMEDRIVE will be used"
3. If you would like to store projects in a custom location, use GUI for that.

In any case, we have separate issue which will allow to override default location for projects platformio/platformio-core#1161

[blujedi]

Documents folder is THE STANDARD for user documents (which includes sourcecode etc.) in Windows.
Visual Studio handles this correctly, even the Arduino IDE handles this correctly.

Can't believe that you have just closed the issue again.

If you want to deliver good software you must properly follow the rules and guidelines for the platforms you are supporting, which in this case is Windows.

[ivankravets]

Please provide an output of python -c "import os; print os.environ"

[blujedi]

I am not going to dump that fully here (privacy).
What entries are you specifically interested in?

[ivankravets]

Do you see env var which is related to Documents folder path is: D:\Users PioUser\Documents?

[blujedi]

No, but that is what I already explained in my descriptions above:

Avoid these Pitfalls

Note the facts below and prevent any pitfalls caused by making 'your own' (incorrect) assumptions:
...

• The Documents Folder location CANNOT be determined from environment variables, unfortunately.

(Thanks Microsoft...)

Another option would be to read them from the registry, but this may also not be possible to do with Python (I don't know).

[riks5]

Hi,
I found a easy solution for eclipse ide.
See attached picture.

[prismv]

When using Linux I can confirm that the PlatformIO extension in VSCode always tries to use %HOME%/Documents/PlatformIO/Projects. There is apparently no way to redefine this as a user as a general setting, which is quite annoying if one wants to use a custom directory structure in the HOME folder. I try to use a simple link %HOME%/Documents/PlatformIO/Projects to the project folder I would like to use, but the link was overwritten when a new project was created.

[ivankravets]

Have you tried to overwrite it via http://docs.platformio.org/en/latest/userguide/cmd_settings.html#projects-dir ?

[prismv]

Okay, it was not clear that the general pio settings are working here. Yes, the projects folder is also changed in vscode ide.
Thanks.

[Galbi3000]

This is still an issue.

I use a limited sized SSD for my system drive and have an HDD for non-essential programs and data, that includes my Documents folder. I would prefer to have the Projects folder in that Documents folder, not on the system drive. Doing lots of development work on an SSD drastically reduces the write cycle lifespan of the drive!

My home directory is C:\users\galbi but my Documents folder is at D:\Documents.

To reproduce the problem yourself open an Explorer window using WIN+E, this should open the window at the "This PC" location. At the top you have the user folders. Right click Documents and select properties. There should be a "Location" tab, go to that. Now change the location of the Documents folder from there (the "Move..." button will move existing files to the new location which can take time).

The correct way to get that location is reading it from the registry entries:
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders\Personal
or
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders\Personal

You should never assume it's in the default location off of the user home directory.

A quick Google on how to read the registry in Python reveals it can be done using the _winreg library. I'm new to Python so I can not give further details but I am sure you know how to obtain and use the library.

Alternatively you could allow manual setting of the PlatformIO folder, or even just the Projects folder :)

[blujedi]

No, the registry is not the correct way to get these settings!
Not future proof therefore error prone. Exactly what is not wanted.

The only CORRECT way to get these settings is by calling the Windows API (as described in the top post).

I spent quite some efforts on this after opening the issue and I even did write some sample Python code to call the relevant Windows API but I have not bothered since (because there is a workaround).

The workaround was already mentioned by @ivankravets above: Use the projects_dir setting to set a (global) custom projects directory.

While the workaround works, it is ugly that one needs to use it for something that can be fixed in PlatformIO itself.

[ivankravets]

We plan to add GUI for pio settings to PIO Home soon. So, you will be able to tune all settings from PIO Home.

[blujedi]

It is nice to have a GUI for pio settings but that does still does not address the core of the issue (incorrect automatic determination of where a Windows users' PlatformIO project directory should be located, while it is possible to determine this correctly).

[Galbi3000]

No, the registry is not the correct way to get these settings!
Not future proof therefore error prone. Exactly what is not wanted.

The only CORRECT way to get these settings is by calling the Windows API (as described in the top post).

I spent quite some efforts on this after opening the issue and I even did write some sample Python code to call the relevant Windows API but I have not bothered since (because there is a workaround).

The workaround was already mentioned by @ivankravets above: Use the projects_dir setting to set a (global) custom projects directory.

While the workaround works, it is ugly that one needs to use it for something that can be fixed in PlatformIO itself.

The Windows API is just as subject to change as the registry so it's not future proof either, nothing is in the programming world. Just take a look at the API function you quoted, SHGetKnownFolderPath, it has gone through at least 3 name changes to get to how it's named now!

The Windows API also gives software access to the registry which is where software is suposed to store settings, that's what the registry was designed for, to replace the multitude of .ini files. So accessing the registry directly is allowed making using the API not the only correct way.

As Platform IO is written in Python for cross platform compatibility, used in VSC on Linux, MacOS and Windows, this issue is made more complex. I assume it would be easier to just read the registry directly in the place of using environment variables in Windows than to try and make it use the Windows API. As I said, I am new to Python so it's beyond my scope. But I am not new to Windows API and programming for Windows using C++, I've been doing it since the days of Windows 3.1, professionally until 2000 and on & off since then as one of many programming hobbies :)

[blujedi]

Still, using the registry is just bad practice. API is public interface, registry is implementation details.

Api access from Python is possible. I have done it when I opened this issue.
Still have it somewhere. So should not be a problem to do in PlatformIO.

[Galbi3000]

Well, we have given them the info on how to fix it, whether or not they impliment either way is yet to be seen but looking at the time span between your OP and today where it's still an issue it looks like they just do not take it seriously enough. But it is serious for users like me who do not want to do compiling and development work on an SSD which has a finite number of times it can be written to.

For now I will have to use symbolic links to force it to use the D: drive. Not ideal but it works :)

[qvdm]

I just came here after spending hours to try and fix the problems created by this issue and was astonished to read that it has been a known issue since 2017.

In my case, I use OneDrive, which moves the Documents folder (and references to it) to inside the OneDrive folder. Every_single_one of my windows applications know how to deal with this, by just following Windows recommended practices, EXCEPT for PIO.

The fact that this issue keeps getting closed without a fix reflects very poorly on the quality of the Platformio project.

[ivankravets]

@qvdm Have you read all comments in this issue? If no, please re-read. The issue is very specific to your OS. PlatformIO is cross-platform tool and Windows is one of the OS which it supports. We use Python API to get the user home directory. There is a setting which you can override if don't like default value. See http://docs.platformio.org/en/latest/userguide/cmd_settings.html#projects-dir

very poorly on the quality of the Platformio project

Do you have enough software/architect expertise to say that?

P.S: This is free and open source project. I recommend helping improve it instead of claiming that this is "very poor and low-quality project".

[ivankravets]

People in this thread, could you open your system terminal and type command below:

python -c "import ctypes.wintypes; buf = ctypes.create_unicode_buffer(ctypes.wintypes.MAX_PATH); ctypes.windll.shell32.SHGetFolderPathW(None, 5, None, 0, buf); print(buf.value)"

Do you see a valid value for your documents directory?

[qvdm]

The command above reports the correct location of the documents directory for me (i.e. not the location that Platformio uses).

[ivankravets]

Resolved in platformio/platformio-core@ffcf6b8

Please re-test with upcoming PlatformIO Core 4.0 => https://community.platformio.org/t/platformio-core-4-0-beta/8304