Reflections daily driving EnderChest

[OpenBagTwo]

I've been daily driving EnderChest for about six months now.

What I like

• I love how changing a config file (say, options.txt) in one instance changes it everywhere that file is linked, not just on one computer, but all of my computers. For things like resourcepack selection, keybinds, mod options (like disabling replaymod recording by default), this has been ace
• The fact that I can control how I link each file (which instances and whether these settings should only be used locally) is amazing (if a little hard to keep up with)
• syncing files across machines, and doing it in a way as to push updates to any machines that are online, is great, especially when paired with content version management running on at least one machine
• updating a file (without changing its name) is pretty easy to do using either tab completion in the terminal or by copying a filename in a file browser
• To my surprise, I've never had issues with "merge conflicts" (changes on one system overwriting newer changes on another). Part of that is the centralized version control, and part of that is just following good hygeine and running enderchest open && enderchest place at the start of each session (and enderchest close at the end of it), which are steps that can absolutely be scripted.
• having files explicitly linked means it's really clear when a file should be replaced (upgraded across all instances) vs. when some instances might still need the old version, and the new version should be kept alongside

What I dislike

• Onboarding a new instance is a nightmare and a half--the best strategy is to identify the most similar instance and manually go through each file symlinked to that instance across the entire EnderChest and add the tag to the new instance. This is very prone to typos.
• File names rapidly become way too long when you're juggling even a half dozen instances. See also File tagging can easily hit the max filename limit #23
• The correct way to experiment with a new mod, resource pack or world is to add it to EnderChest with symlinks and run a place. The easy way to do it is to drop it into the instance folder directly, which can cause all sorts of issues (see also: EnderChest interferes with auto-updating systems #24)
• It's so darn hard to make sure you're not adding two different versions of the same file to an instance when it could live in client, global or local-only (and that's leaving aside Tagged folders and tagged assets in untagged folders creates conflicts #22)
• I wish there were a way to manage options.txt and iris.properties specifically in a way to share individual settings while not sharing others...

What I've learned

The common theme running through is: symlinks good; syncing good; intent captured in filenames bad

What's next

It's time to rethink the design paradigm and move from a system that directly links individual files to individual instances to one that maps groups of files to groups of instances. I'm branding this approach Shulker Boxes. This will restructure the EnderChest file system from its current state:

EnderChest/README.md

Lines 32 to 42 in 0f17b4f

	```
	<minecraft root>: a single folder where all your minecraft data will live
	├── EnderChest: the root directory for all resources managed by this package
	│ ├── global: put anything here that should go in every minecraft installation, client or server
	│ ├── client-only: put anything here that should only go in a client-side installation and not a server
	│ ├── local-only: anything that's only for THIS PARTICULAR INSTALLATION and not for sharing across different computers
	│ └── server-only: anything that's only needed on a server (properties file, banlist, etc.)
	│
	├── instances: this is where EnderChest will assume your curseforge / Multi-MC-fork instances live
	├── servers: this is where EnderChest will assume all your server installations live
	```

to one where the EnderChest top-level will be populated by uniquely named Shulker Boxes (folders) that each contain a config file that specifies the intent of where the files inside should be deployed.

At its most basic, that config could just be a list of instances crossed with global/server/client/local, but it could also specify, say,

[minecraft-version]
1.19.1
1.19.2
1.19.3

[loader]
fabric
quilt

or even include tags such as

[tags]
vanilla-plus
# modded
multiplayer

with the instances that those tags link to being controlled either centrally (inside an instance.cfg file stored in the EnderChest root, or an enderchest.cfg file stored in the .minecraft folder of each instance)

The core idea is that Shulker Boxes should map to instances many-to-many--files across many shulkers can feed into a single instance, and many instances can share files stored in a shulker

The advantages of this approach are:

• Swapping out files becomes a matter of drag & drop without needing to rename
• Onboarding new instances requires editing a small number of config files rather than renaming a metric ton of them
• This is still compatible with the "Chest Monster" paradigm (where all files are actually stored elsewhere, untagged and unsorted), as shulkers can store symlinks without issue.

The disadvantages is:

• Onboarding new instances is still far from painless
• This doesn't solve the conflict problems, either Tagged folders and tagged assets in untagged folders creates conflicts #22 or EnderChest interferes with auto-updating systems #24
• This will 💯 break backwards compatibility with old versions of EnderChest. Which ultimately is fine.

I have to think through how to solve #24 specifically. I'm wondering if I just need to employ a cleaning lady

[OpenBagTwo]

Here's what I'm thinking in terms of revised file/folder layout. In this example, we have one client instance and one server instance that are potentially sharing assets across two shulker boxes.

<minecraft root> : a single folder where all your minecraft data will live 
├── EnderChest/ : the root directory for all resources managed by this package 
│   ├──  instances.cfg : a file that gives the path to and metadata for each locally installed instance
│   ├──  open.sh : script for pulling remote changes to local
│   ├──  close.sh : script for pushing local changes to remotes
│   ├── 01 red/ : an example shulker box, with name chosen by the user with the expectation of lexical sorting 
│   │   ├── shulker.cfg : specification of which instances this shulker should map to (DOES NOT SYNC)
│   │   ├── config/ : folder structure pre-populates to mirror .minecraft folder
│   │   │   ├── iris.properties : an example of a file that would be synced
│   │   │   ├── mymod.json
│   │   ├── mods/
│   │   │   ├── mymod_v1.0.jar
│   │   ├── resourcepacks/
│   │   │   ├── teavsrp.zip
│   │   │   ├── custom_foxnap_playlist  ↩ : an example of a symlink used instead of a file
│   │   ├── saves/
│   │   │   ├── bigblue  ↩
│   │   ├── shaderpacks/
│   │   │   ├── minecraft_rtx.txt
│   │   ├── backups/ ↩ : the box is set up to overwrite instance backups folders with this symlink
│   │   ├── logs/ : the box is set up to overwrite the instance log folders with a link to this folder
│   │   │   ├── 1970-01-01-1.log.gz
│   │   │   ├── 1970-01-02-1.log.gz
│   │   │   ├── 1970-01-03-1.log.gz
│   │   ├── screenshots/ ↩
│   │   ├── options.txt : some synced files will be top-level
│   ├── 02 orange/ : another shulker box with lower priority than the one above 
│   │   ├── shulker.cfg
│   │   ├── config/ : empty folder with nothing to sync
│   │   ├── mods/ : files in this folder will be synced alongside files from the above box
│   │   │   ├── iris-mc2.0-3.1.4.jar
│   │   ├── resourcepacks/
│   │   │   ├── barebones.zip
│   │   ├── saves/
│   │   ├── shaderpacks/
│   │   │   ├── minecraft_rtx.zip
│   │   ├── backups/ : the box is configured not to treat this as an empty folder with nothing to sync
│   │   ├── <deleting an empty and unused folder will have no ill effects>
│   │   ├── options.txt : because this box is lower priority, this will be ignored if both map to the same instance
├── instances/ : this is where EnderChest will automatically look for curseforge / Multi-MC-fork instances
│   ├── instgroups.json : MultiMC variants encode tags in this file
│   ├── minecraft_20 / 
│   │   ├── mmc-pack.json : information about the instance can be parsed from this file for MultiMC variants
│   │   ├── .minecraft/
├── servers/ : this is where EnderChest will automatically look for server instances
/opt/
├── mc_servers/
│   ├──  myserver/ : an example instance that would not be automatically detected but could still be managed

with configuration files that read:

; instances.cfg
[minecraft_20]
minecraft = 2.0
loader = fabric
role = client
tags = vanillaplus

[/opt/mc_servers/myserver]
minecraft = 2.0
loader = fabric
role = server

; 01 red/shulker.cfg
[instances]
trendy

[groups.vanillaplus]
minecraft = 2.0
loader = fabric
role = client
tags = vanillaplus

[link_folders]
backups
logs
screenshots

; 02 orange/shulker.cfg
[groups.clients]
minecraft = 2.0
loader = fabric
role = client

[groups.servers]
minecraft = 2.0
loader = fabric
role = server

Some key features:

• instances and servers now need not be centrally managed under a root "minecraft" folder. The extension of this is that the EnderChest folder could actually go anywhere
• that being said, instances and servers folders who share a parent directory with the EnderChest folder can be automatically discovered and their configurations parsed
• the authoritative list of managed instances lives in EnderChest/instances.cfg. This information can be automatically populated when an instance is discovered or manually onboarded.
• shulker boxes are assumed to be sorted lexically to deterministically resolve conflicts (in this proposal, earlier boxes take precedence)
• each shulker box contains its own shulker.cfg "intent" file. This may contain:
  • an explicit list of instances for linking
  • sets of match conditions for linking (with the logic that an instance gets linked if it's explicitly listed or if it matches all conditions of at least one match group)
  • a list of any folders that should be linked entirely

Still missing from this proposal is a way pulling in assets from each instance. That's needed to resolve #24 and could also be generalized as an easy method of onboarding existing instances that weren't previously managed by EnderChest.

[OpenBagTwo]

I think I've captured everything I needed from this into other issues.