RetroArch saves the real path of a symbolic link to playlist files

[x6fan]

Description

When RetroArch opens a symbolic link as content using the file browser, it loads any directory or content specific configuration files, resolves the path to the real file and then opens it. This works as expected.

The problem is RetroArch saves the resolved path to the playlist files. When the same content is opened via history, favorites or any other playlist, the directory and content specific configuration files may not be loaded because the real directory and file names may not match the ones the user loaded and configured via the file browser.

Expected behavior

• Playlist files contain the path to the symbolic link to the content.
• The symbolic link is resolved every time content is loaded from a playlist.
• Directory or content specific options are applied according to the symbolic link name and its containing directory.

Actual behavior

• Playlist files contain the real path to the content.
• The symbolic link is resolved once before the playlist is written to storage.
• Directory or content specific options are applied according to the real file name and its containing directory.

Steps to reproduce the bug

1. prefix=~/retroarch-symlink-test
2. mkdir -p $prefix/symbolic $prefix/real
3. Copy a content file to $prefix/real/content
4. ln -s $prefix/real/content $prefix/symbolic/game
5. Launch RetroArch
6. Open $prefix/symbolic/game in the file browser
7. Open Quick Menu
8. Add content to favorites using the Quick Menu
9. Go to Controls and Save Content Directory Remap File
10. Close RetroArch
11. Verify that the content directory remap file has been saved as symbolic.rmp
12. Verify that $prefix/real/content was saved to the content_favorites.lpl playlist file
13. Open RetroArch
14. Open the content from the favorites playlist
15. Open Quick Menu, go to Controls and verify that symbolic.rmp has not been applied
16. Open $prefix/symbolic/game in the file browser
17. Open Quick Menu, go to Controls and verify that symbolic.rmp has been applied

Since $prefix/real/content was saved to the playlist, RetroArch looks for a content directory remap file named real.rmp instead of symbolic.rmp. When the symbolic link is loaded from the file browser, RetroArch loads symbolic.rmp as expected.

Workarounds

Use hard links instead of symbolic links.

Version/Commit

$ retroarch --version
RetroArch: Frontend for libretro -- v1.7.7 -- 62da78ca3d --
Compiler: GCC (8.3.0) 64-bit Built: May 12 2019

Environment information

• OS: Arch Linux

[orbea]

@jdgleaver Maybe you would like to look at this one?

[jdgleaver]

Ah, okay - the reason for switching over to using resolved symlink paths in playlists was that users were complaining about duplicate entries (e.g. if you load content via the 'real' path, and subsequently load it via the symlinked path, you get a duplicate in content history)

I guess we may have been overzealous - we need to check the resolved path when searching/pushing content, but we can probably save the 'input' path without issue.

There's currently this PR awaiting approval: #9137. This makes some changes to playlist.c (specifically the path resolving code), so I think we need to wait until this is merged before changing anything else. Once that's done, I can have a look at fixing this issue.

[x6fan]

Thank you.

[DocMAX]

Bump. Any updates?

[supertrianguloid]

Just ran into this bug as well. I'm trying to replicate and synchronise my retroarch setup on multiple systems, and symlinks are vital for me. Hard links are horrible so my workaround is currently to just sed the playlist files to restore the correct path to the symlink.

[i30817]

Just ran into this bug as well. I'm trying to replicate and synchronise my retroarch setup on multiple systems, and symlinks are vital for me. Hard links are horrible so my workaround is currently to just sed the playlist files to restore the correct path to the symlink.

This is what portable playlists are for.

1. enable on settings->playlists portable playlists on both systems running RA
2. change settings->directory->File Browser on both systems to where you expect roms to appear (note, this can be a parent of the playlist scan dir, it's not necessary to be the same, which is useful for manual scans, which have to separate platform roms)
3. Change the playlist dir to one in a place both systems can access, external drive or partition accessible to both OSes (change settings->directory->playlists on both systems).
4. scan playlists in one of the systems, like you want, manual, automatic, whatever, as long as the playlists and roms can be accessed from both systems. For that reason, if you use m3u, cues, dosbox.conf as playlist entry points (files with paths inside), it's best they are the kind that uses relative paths in the same directory, so they are os portable by not using the directory separator. Since redump cues are this type, this is a rare problem to trigger, but just in case I mention it, it's very possible to have a dosbox.conf that is not portable on the other hand, unless you do some unintuitive things there.
5. (Optional) if you do this you might as well share bios ('system'), configs, thumbnails and saves and savestates directories on both systems too, which is just changing those directories on each system to shared dirs. I suppose you can use symbolic links here, although I'm not sure it works across OSes. Savestates on some cores might cause problems if they have endian bugs though if the endianess is different on both systems.

And you get playlist transparency for free (as long as you don't use the same config file for different platforms of course, which is already impossible for other reasons unless your system has the exact same paths anyway). If you want a 'completely portable' RetroArch this can be done by having 2 directories in a pen drive, one with (for example), the windows executable and windows retroarch.cfg, the other with the Linux executable and retroarch.cfg. Or if you prefer a single dir, a sh\bat file to use different cfg files for each executable.

I find portable installs of RA unnecessary though, if you can install it on both systems.

Portable playlists work by fixing up paths in playlists if the playlist base_content_directory (added to playlists if you have file browser path set and portable playlists on) doesn't equal the current file browser path setting. If it doesn't, it replaces, all the parts of the entries paths that start with it by the new one, and in windows replaces / by \ and vice versa for from windows to unix - note this has no problems with dumping groups games filenames because they forbid \ and / in filenames.

[i30817]

Here is how it looks, parts with double asterisks are the parts replaced (not counting the slashes)

{
  "version": "1.5",
  "default_core_path": "/data/data/com.retroarch.aarch64/cores/puae_libretro_android.so",
  "default_core_name": "Commodore - Amiga (PUAE)",
  "base_content_directory": "**/storage/emulated/0/RetroArch/downloads**",
  "label_display_mode": 0,
  "right_thumbnail_mode": 0,
  "left_thumbnail_mode": 0,
  "sort_mode": 0,
  "scan_content_dir": "**/storage/emulated/0/RetroArch/downloads**/Commodore - Amiga",
  "scan_file_exts": "lha",
  "scan_dat_file_path": "**/storage/emulated/0/RetroArch/downloads**/Commodore - Amiga/games.dat",
  "scan_search_recursively": true,
  "scan_search_archives": false,
  "scan_filter_dat_content": true,
  "scan_overwrite_playlist": true,
  "items": [
    {
      "path": "**/storage/emulated/0/RetroArch/downloads**/Commodore - Amiga/Amiga WHDLoad/0/1000ccTurbo_v1.0.lha",
      "label": "1000ccTurbo (v1.0)",
      "core_path": "DETECT",
      "core_name": "DETECT",
      "crc32": "00000000|crc",
      "db_name": "Commodore - Amiga.lpl"
    },

The core path also has a fallback, although I didn't really understand the code to explain it.

[DocMAX]

Symlinks are still resolved to their real paths in the playlists. How can we disable this and let playlist point to the symlink path???

[i30817]

RetroArch solution to this is portable playlists, because symlinks, frankly, will give you a false sense of security without a lot more scripting than you probably expect.

This is because, in general, the config file (retroarch.cfg), that holds the paths to most files, cant be shared across systems (there are hidden random paths for things like downloaded asset zip files, os and system dependent settings) and in the case of paths only inside the the playlists, symlinks switches will only work in the same type of OS, with the same path seperators and drive names.

The playlist can also only hold absolute paths (because different platforms have different relative locations for the current directory), but thats different than canonical paths (symlinks eliminated) so it's irrelevant to that decision.

In short, it doesn't work, probably by accident, but if it did work and you ran a script before running RetroArch to switch locations... It would still be doing it wrong. Probably as soon as you wanted to run the same games drive in windows and linux\mac\openbsd. Or your path to the games directory required the current username on it and you needed a sudo on your symlink switch script in a system with another username you're using...

The portable playlists transformation also has a limitation it can't avoid. Directories names can't have either \ or / in the playlist paths. But that's easy, since redump\nointro\windows already avoids that afaik.