Add game list for Heroic Games Launcher

[DavidoTek]

Add a game list for the Heroic Games Launcher.
The game list could be implemented in a similar way as the Steam game list.

Details

(As of December 2022)

Configuration for the Flatpak version of HGL can be found in ~/.var/app/com.heroicgameslauncher.hgl/config/heroic
Additional configuration for Legendary is stored in ~/.var/app/com.heroicgameslauncher.hgl/config/legendary

Games: Side-loaded apps

The file ~/.var/app/com.heroicgameslauncher.hgl/config/heroic/store/sideload_apps/library.json contains a list of manually added games. Example:

{
	"games": [
		{
			"runner": "sideload",
			"app_name": "sPZQ5kmzYj5KnZKdxE2bR1",
			"title": "Test Title",
			"install": {
				"executable": "test",
				"platform": "Windows"
			},
			"folder_name": ".",
			"art_cover": "file:///app/bin/heroic/resources/app.asar/build/assets/heroic_card.dae295a6.jpg",
			"is_installed": true,
			"art_square": "file:///app/bin/heroic/resources/app.asar/build/assets/heroic_card.dae295a6.jpg",
			"canRunOffline": true
		}
	]
}

Games: GOG games

The file ~/.var/app/com.heroicgameslauncher.hgl/config/heroic/store/gog_store/library.json contains a list of GOG games. Example:

{
	"games": [
		{
			"runner": "gog",
			"store_url": "https://gog.com/en/game/akalabeth_world_of_doom",
			"developer": "Richard Garriott",
			"app_name": "1207666073",
			"art_cover": "https://images-1.gog-statics.com/56262850671fd7232b8ff46deba530007ff7a437fab0ddd67147f8d41aa6bca0.jpg",
			"art_square": "https://images.gog.com/511cf474562bdaf2c8b9eec1a0c94e1eb5cf4e3900a618516adff2ebd89118e0.jpg?namespace=gamesdb",
			"cloud_save_enabled": false,
			"extra": {
				"about": {
					"description": "In the game, the player character visits Lord British and is instructed by him to kill certain monsters. The final mission is to kill a balrog, and after this has been accomplished, Lord British proclaims that \"thou hast proven thyself worthy of knighthood\".",
					"longDescription": ""
				},
				"reqs": []
			},
			"folder_name": "",
			"install": {
				"is_dlc": false
			},
			"is_installed": false,  /* comment: install status */
			"namespace": "akalabeth_world_of_doom",
			"save_folder": "",
			"title": "Akalabeth: World of Doom",
			"canRunOffline": true,
			"is_mac_native": true,
			"is_linux_native": true
		}
	],
	"totalGames": 1,
	"totalMovies": 0,
	"cloud_saves_enabled": true
}

Game details

The files in ~/.var/app/com.heroicgameslauncher.hgl/config/heroic/GamesConfig contain the configuration for a specific game.
Example (side-loaded app sPZQ5kmzYj5KnZKdxE2bR1.json):

{
  "sPZQ5kmzYj5KnZKdxE2bR1": {
    "autoInstallDxvk": true,
    "autoInstallVkd3d": true,
    "preferSystemLibs": false,
    "nvidiaPrime": false,
    "enviromentOptions": [],
    "wrapperOptions": [],
    "showFps": false,
    "useGameMode": false,
    "language": "",
    "wineVersion": {
      "bin": "/home/user/.local/share/Steam/compatibilitytools.d/GE-Proton7-43/proton",
      "name": "Proton - GE-Proton7-43",
      "type": "proton"
    },
    "winePrefix": "/home/user/Games/Heroic/Prefixes/TitleTest"
  },
  "version": "v0",
  "explicit": true
}

Legendary: Installed games

The file ~/.var/app/com.heroicgameslauncher.hgl/config/legendary/installed.json contains all installed Legendary/Epic Games games. Example:

{
  "5b60142e120c4f2d88027595c21d4a04": {
    "app_name": "5b60142e120c4f2d88027595c21d4a04",
    "base_urls": [
      "https://fastly-download.epicgames.com/Builds/Org/o-bthbhn6wd7fzj73v5p4436ucn3k37u/c5a7ca9faed74eb99c1a7291eec2a2ff/default",
      "https://epicgames-download1.akamaized.net/Builds/Org/o-bthbhn6wd7fzj73v5p4436ucn3k37u/c5a7ca9faed74eb99c1a7291eec2a2ff/default",
      "https://download.epicgames.com/Builds/Org/o-bthbhn6wd7fzj73v5p4436ucn3k37u/c5a7ca9faed74eb99c1a7291eec2a2ff/default",
      "https://download2.epicgames.com/Builds/Org/o-bthbhn6wd7fzj73v5p4436ucn3k37u/c5a7ca9faed74eb99c1a7291eec2a2ff/default",
      "https://download3.epicgames.com/Builds/Org/o-bthbhn6wd7fzj73v5p4436ucn3k37u/c5a7ca9faed74eb99c1a7291eec2a2ff/default",
      "https://download4.epicgames.com/Builds/Org/o-bthbhn6wd7fzj73v5p4436ucn3k37u/c5a7ca9faed74eb99c1a7291eec2a2ff/default"
    ],
    "can_run_offline": true,
    "egl_guid": "",
    "executable": "DOOM64_x64.exe",
    "install_path": "/home/user/Games/Heroic/DOOM64",
    "install_size": 120291379,
    "install_tags": [],
    "is_dlc": false,
    "launch_parameters": "",
    "manifest_path": null,
    "needs_verification": false,
    "platform": "Windows",
    "prereq_info": null,
    "requires_ot": false,
    "save_path": null,
    "title": "DOOM 64",
    "version": "2124"
  }
}

[sonic2kk]

I've got some very basic and very WIP work done for this for GOG at least, sideloaded should work as well but this is untested at the time of writing. Right now it can get Heroic games and it also works for the ctinfodialog, but no work yet for a Games List yet. I'm planning to look into this after the #192 to avoid annoying conflicts 🙂

Just wondering: What does the Epic Games Store library.json (or equivalent) look like and what is its path? Using the Legendary installed.json may be an issue since we can't get the app name from this alone as far as I know. I don't have an Epic Games account so I cannot check this myself 😅

If the library.json is similarly structured to gog and sideloaded games then very little extra work may be required for the initial implementation I have right now, I would just need to add its path.

[sonic2kk]

Based on discussion in #193, Legendary handles games with a separate structure using an installed.json as the example showed, it doesn't have a library.json it seems.

[sonic2kk]

With #193 merged (woo!!), I'm gonna take a look at implementing at least an initial Heroic Games list similar to #192.

The plan currently is an initial draft implementation that just displays game information. I am not sure at least initially if there will be functionality like the Steam games list for updating the compat tool, but for Heroic it should be possible (famous last words 😅).

---

I did a bit of investigating and want to try and explain what I found with regards to implementing this compat tool updating, hopefully I can explain it in a way that makes sense:

How Heroic Stores Wine Information

Heroic Wine information is stored for each HeroicGame under wine_info. I have only checked GOG and sideloaded games so this will need to be confirmed for Legendary, but under GamesConfig/<app_name>.json, the wineVersion key looks like this:

    "wineVersion": {
      "bin": "/home/user/.var/app/com.heroicgameslauncher.hgl/config/heroic/tools/wine/lutris-GE-Proton7-37-x86_64/bin/wine",
      "name": "Wine - lutris-GE-Proton7-37-x86_64",
      "type": "wine",
      "lib": "/home/user/.var/app/com.heroicgameslauncher.hgl/config/heroic/tools/wine/lutris-GE-Proton7-37-x86_64/lib64",
      "lib32": "/home/user/.var/app/com.heroicgameslauncher.hgl/config/heroic/tools/wine/lutris-GE-Proton7-37-x86_64/lib",
      "wineserver": "/home/user/.var/app/com.heroicgameslauncher.hgl/config/heroic/tools/wine/lutris-GE-Proton7-37-x86_64/bin/wineserver",
      "wineboot": "/home/user/.var/app/com.heroicgameslauncher.hgl/config/heroic/tools/wine/lutris-GE-Proton7-37-x86_64/bin/wineboot"
    },

Heroic stores the wineserver, wine boot command, Wine name and Wine binary all separately. If you change these paths to a valid Wine path, Heroic will launch using that Wine command. And if any of the paths are invalid, Heroic will complain and suggest another tool to launch with.

It only appears to store the further wineboot etc keys if a Wine runner is used. For Proton it doesn't use these. I guess for Proton it just runs /path/to/proton waitforexitandrun /path/to/game but for Wine it might build a full Wine command, and use the wineboot command to create a fresh prefix/setup a prefix with a new Wine version.

Something interesting here is name is not the name used in the Heroic wine version list for a game, it is only used on the main game info screen. This is the screen after you click into a game and it shows the first played, last played etc information, and the "Play Now" button. The game I was testing (HoloCure) was using lutris-GE-Proton7-37, which was installed to the Heroic Wine tool path. I then changed the path to point to a build of lutris-GE-Proton7-28. The paths were valid and HoloCure booted up, however in the settings for that game it still said I was using lutris-GE-Proton7-37! I cannot find out why it does this or where this is set, maybe there is some internal cache for Heroic that defines this. At the moment though, I cannot find it. It doesn't get this from the path or from the VERSION file for the given Proton version, otherwise it would update correctly in both of those instances.

So this means if we give the ability to update the Wine version in the Games List for Heroic Games, the text may not update in the "Wine Version" dropdown in the game settings. This doesn't appear to impact functionality as the Proton version on the game info screen updates, and the game appears to update to use the Proton version pointed to in the paths, so it's just this display issue. It could lead to confusion, though.

How Heroic Finds Wine/Proton Tools

For some background: Heroic lets you choose Wine and Proton versions from at least the following places:

• Steam's compatibilitytools.d directory
• Valid Proton installations across Steam Library Folders e.g. Proton 7.0
  • Proton versions are installed to the same library folder as the first game that requires it. So if you have DOOM Eternal at /run/media/gaben/Games/steamapps/DOOM Eternal, and if Proton 7.0 is the default Proton for that game (either selected by Valve or as the default Steam Play version, which usually just defaults to latest if no compat tool is forced under Properties), Proton 7.0 will be installed to /run/media/gaben/Games/steamapps
• Lutris Wine/Proton installations at e.g. ~/.local/share/lutris/runners/wine
• Heroic Wine at /path/to/heroic/config/heroic/tools/wine
• Heroic Proton at /path/to/heroic/config/heroic/tools/proton

For a game's settings, Heroic has a little "Help" button that helpfully outlines the locations as well:

• ~/.config/heroic/tools/wine
• ~/.config/heroic/tools/proton
• ~/.steam/root/compatibilitytools.d
• ~/.steam/steamapps/common
• ~/.local/share/lutris/runners/wine
• ~/.var/app/com.valvesoftware.Steam (Steam Flatpak)
• /usr/share/steam

Interestingly this doesn't list the separate Steam library folders, but I have verified that it shows Proton versions for me that are not in ~/.steam/root/steamapps. Also note that actual path it shows, ~/.steam/steamapps/common, is incorrect, at least on my system this doesn't exist! It should be in ~/.steam/root/steamapps. It also doesn't correctly list the Heroic Flatpak paths, only the folders for things like package manager/AppImage/etc paths.

This help text also mentions that it looks for "CrossOver" tools as well. As far as I know this is a Mac-only tool by CodeWeavers based on Wine, but it doesn't mention where, and I don't think this tool is available for Linux, so we shouldn't need to worry about it.

Differences in Paths for Compatibility Tools

Heroic will only select valid Wine/Proton tools from these directories (so e.g. SteamTinkerLaunch or Luxtorpeda will be excluded). However, Valve Proton, GE-Proton/Proton-tkg and lutris-GE-Proton (aka Wine-GE)/Wine-tkg each have different paths to the Wine binaries.

• Valve Proton is stored at e.g. Proton 7.0/dist/bin (and dist/lib64, etc)
• GE-Proton/Proton-tkg (Valve Exp Bleeding Edge and Wine Master builds) is stored at GE-Proton7-49/files/bin (and files/lib64, etc)
• lutris-GE-Proton/Wine-tkg (seems to apply to all Wine-tkg versions I could see) is stored at lutris-GE-Proton7-37-x86_64/bin (it doesn't have a dist or files subdir, the files are right as you open the folder)
  • This also applies to "regular" Lutris Wine builds, that is the builds of Wine you can download from within Lutris - Likely this is because lutris-GE-Proton uses the Lutris Wine buildsystem so the files should always line up with however Lutris stores them

For Proton tools (Proton-tkg, GE-Proton and Valve Proton) it appears that it only stores bin (which is the path to the proton script, which is always in the root dir e.g. GE-Proton-whatever/proton, Proton 7.0/proton, etc). However I am not sure if this always applies, this will need further investigation. For completeness, I still listed the path to the wine binary for Proton builds, though we may only need to be concerned with the wine binary paths for Wine builds.

Implementing Compat Tool Selection Logic

This means that if/when we implement the ability to change Wine/Proton versions for a Heroic game on the Heroic Games List, we may have three different paths we have to account for.

We can probably check this based on their paths, we could have a list of heroic_valve_proton_paths for Valve Proton, heroic_proton_paths for GE-Proton/Proton-tkg etc (will include compatibilitytools.d and config/heroic/tools/proton), and heroic_wine_paths (config/heroic/tools/wine as well as Lutris Wine runner paths). From the paths we can infer if a tool is an official Valve Proton build, a community Proton build, or a Wine build, and from there we can infer how to set the bin, lib, lib32, wineserver and wineboot values for Wine tools, and how to set the path to the proton script for Proton builds (if they only take the /path/to/proton as its bin key).

Probably we would have a function that can check for this based on a selected Heroic tool (it could take the HeroicGame and the tool_path or something), and then we can write out the relevant keys to that tool's config JSON. We could rebuild the wineVersion dictionary in Python and write that out to the JSON, that way we can write out only the relevant keys based on the type of Wine, e.g. don't write out the wineserver etc keys for Proton.

We should also make sure we follow the pattern for how Heroic names Wine and Proton tools, which is Proton - Proton 7.0, Wine - lutris-GE-Proton7-37-x86_64 - Basically it prefixes the type key (with an uppercase letter) follow by a -, and then the tool name. This mirrors the compat tool's folder basename, e.g.,

• Valve Proton shows up as Proton - Proton 7.0 and not something like proton 7.0-6e.
• Proton-tkg shows up as Proton - proton_tkg_8.1.r3.g87f33695
• Wine-tkg shows up as wine_tkg6d9ef1cbd06ca35053bd1edb0b1fb51f3fb47410

So we need to ensure we set the type and name fields correctly for consistency if nothing else.

I'll also mention that I am using the config files for sideloaded and GOG games as reference only, I don't have any Legendary config files to check with. This is working off of the assumption that Heroic, sideloaded and Legendary games have the same config structure in this respect. I would assume that they would but I wanted to make that assumption clear in case I am wrong :-)

---

So aside from the Wine/Proton name display issue mentioned in the game's config settings, where the name does not update in this single instance to reflect the currently selected tool. This could be a Heroic bug but surely it has to be stored somewhere if its stored, right? I'm just not sure where. Perhaps it's some weird Electron cache folder. I wonder if it's worth reporting some of these (apparent) inconsistencies upstream as we seem to have discovered a small handful at this point :-)

I don't think the compat tool selection will be completed for the initial draft of the Heroic Games List but I hope that the research here made sense, and that it can help in future when it comes to potentially implementing this functionality.

[DavidoTek]

Heroic stores the wineserver, wine boot command, Wine name and Wine binary all separately. If you change these paths to a valid Wine path, Heroic will launch using that Wine command

Is there any case (except Proton) where they don't share a common prefix like <winepath>/bin/wine, <winepath>/lib, <winepath>/bin/wineserver, etc.? If they do that, it should be a simple task to read/modify them.

So this means if we give the ability to update the Wine version in the Games List for Heroic Games, the text may not update in the "Wine Version" dropdown in the game settings.

I did a quick search but wasn't able to find a file (at least not plaintext/json) that contains the Proton version except the files in GameConfig.
After looking at the Heroic "game settings" code, it seams like the list of available compatibility tools is fetched by getAlternativeWine. There it uses the foldername instead of the name attribute for the version name: config.ts#L323-L327
I haven't looked at the code for the "game info" in depth though.

Also note that actual path it shows, ~/.steam/steamapps/common, is incorrect

I think it was changed at some point.

"CrossOver" tools as well. As far as I know this is a Mac-only tool by CodeWeavers

It's actually available for Linux (and ChromeOS) as well. I don't think it is well suited for gaming though (doesn't use DXVK by default).

Heroic, sideloaded and Legendary games have the same config structure in this respect.

I wasn't able to spot any difference between them, at least the wineVersion section is exactly the same for all games using the same compatibility tool.
I noticed some games are using ~/.wine for the winePrefix and some are using ~/Games/Heroic/Prefixes/<game name>, though I cannot remember if I have changed that manually (both were GOG games).

I wonder if it's worth reporting some of these (apparent) inconsistencies upstream as we seem to have discovered a small handful at this point :-)

I'm not sure whether that behavior is intended. Ideally (currently) the name attribute is always equal to the folder name, this way it cannot be broken. I guess the name attribute only exists so the name doesn't has to be read from the binary path.

I don't think the compat tool selection will be completed for the initial draft of the Heroic Games List but I hope that the research here made sense, and that it can help in future when it comes to potentially implementing this functionality.

I think that is a good start. Thanks for doing the research on that!

[sonic2kk]

Is there any case (except Proton) where they don't share a common prefix

As far as I know, Wine builds should share a common prefix - That is, Wine-GE and Wine-tkg.

It should only be the Proton tools that have different prefixes - One prefix for Valve Wine and another for GE-Proton/Proton-tkg (didn't check things like NorthstarProton though).

It's actually available for Linux (and ChromeOS) as well. I don't think it is well suited for gaming though (doesn't use DXVK by default).

Ah yes, I think it has actually been around since before DXVK (an era that feels so long ago at this point :nerd:). I don't think it will impact us too much on the ProtonUp-Qt side, though as always bug reports from users will be the deciding factor there :-)

I wasn't able to spot any difference between them, at least the wineVersion section is exactly the same for all games using the same compatibility tool.

That's really good news, I think that's all we need to be concerned about with regards to allowing the option to change the Wine version. Since the format is the same we'll just need to rebuild the wineVersion section and write it out again (at least in theory that's how I would foresee it working at a high level 😅)

---

At least an initial version of the Heroic games list should be ready soon. One thing that completely slipped my mind somehow during the Heroic work I've been doing is testing with native games and seeing how data (i.e. wineVersion) is stored for them. I somehow completely forgot that native Linux games can be sideloaded and played from GOG.

So I'll do some investigation around native games, and a couple of other potential minor touchups, and then get a PR out for review on the games list :-)

[sonic2kk]

A basic version of this has been implemented now with #199. This could be kept open though, for discussion and further improvement i.e. switching Proton versions and so on. Just updating here for clarity that this feature is in main now for users reading :-)

[sonic2kk]

I think Heroic is getting a UI update soon I think (GamingOnLinux says "Heroic Games Launcher has a big UI upgrade coming in the 3.0 update which isn't quite ready yet [...]"), we should keep tabs on this and monitor in case there are any breaking changes we need to deal with here.

If 3.0 comes and there are no breaking changes, we could possibly close this issue :-) The games list has been in place for a while now and seems to work well.

[sonic2kk]

Heroic Games List was not loading for me temporarily, but adding permission to ~/.var/app/com.heroicgameslauncher.hgl/config/, opening ProtonUp-Qt, closing it, and then removing that permission resolved the problem. Uninstalling ProtonUp-Qt Flatpak and removing its configuration data, and re-installing also did not bring the problem back. So I guess this was some weird Flatpak issue I encountered.

Original response

Actually, the Heroic Games List is blank for me on 2.8.2 Flatpak, but not on 2.8.2 AppImage or main... I don't think my Flatpak permissions are messed up either and I don't remember it being blank before. There must be some kind of permissions issue...? The Steam and Lutris game lists still display correctly.

Reset all my ProtonUp-Qt Flatpak settings with Flatseal and still not displaying. No errors in the terminal when running the Flatpak either (I guess we catch them and just continue somewhere?).

From what I'm remembering while looking at the code in heroicutil, we parse the Heroic JSON files that contain info about the installed games, and build HeroicGame objects from those, and put them all in a list. That's what pupgui2gameslistdialog's self.games is for the Heroic launcher. I guess for some reason this isn't populating correctly in Flatpak, and since it only happens in Flatpak, it might not be a stretch to say the Flatpak can't read the JSON files for some reason.

ProtonUp-Qt should have all the correct permissions though. Unless we need to specify reading specific files? But we don't need to specify that for Steam and we can read config.vdf, libraryfolders.vdf, and shortcuts.vdf, and write to config.vdf and shortcuts.vdf...

• store_cache gives us Amazon Games' nile_library.json
• sideload_apps gives us sideloaded games' library.json
• gog_store gives us GOG games' library.json
• ../legendary gives us the Epic Games Store integration's install.json (I don't own any games from the Epic Games Store, but dummy data in this file that populates for 2.8.2 AppImage / main branch does not populate for Flatpak)

Compatibility tools still populate correctly for Heroi (I have a DXVK, vkd3d-proton, and a Proton version installed), so it can access ~/.var/app/com.heroicgameslauncher.hgl/config/heroic/tools, it's just the games list that won't populate...

[DavidoTek]

Heroic Games List was not loading for me temporarily, but adding permission to ~/.var/app/com.heroicgameslauncher.hgl/config/, opening ProtonUp-Qt, closing it, and then removing that permission resolved the problem

That's interesting. Had no issues on my side. Also checked Heroic Flatpak Manifest just in case, but there seem to be no relevant changes.