1.18.x Custom datapack registries, round three-ish

[Commoble]

This is a fork of #8522, Silverminer's custom datapack registries PR, with some improvements to documentation and error presentation and streamlining of userdev APIs.

Datapack Registries, what are they and why might modders want them

Datapack registries (sometimes called dynamic registries or worldgen registries) are a system added by Mojang several major versions ago, neither static registries nor reloadable data, but a third system separate from the other two. Datapack registries are created by registering a loading codec and (optionally) a network codec (if the network codec exists, we consider the registry to be a Synced Datapack Registry). Examples of vanilla datapack registries include biomes, dimensiontypes, and placed features.

Features of datapack registries (relative to reloadable data)

• Datapack registries have a simplified standard implementation for loading and optionally syncing (which cannot be changed, as opposed to reloadable data which provides lower-level access to data)
• Datapack registries have half-builtin support for datagen, using their loading codecs and RegistryOps
• Datapack registries cannot be reloaded with /reload, they can only be reloaded by rebooting the server
• Datapack registries automatically support tag jsons
• Unsynced datapack registries' elements can directly reference any other datapack registries' elements

Lifecycle of datapack registries:

• The datapack registry and its codec(s) are registered (custom datapacks can be registered via the NewRegistryEvent or DeferredRegisters)
• "Builtin" objects can optionally be registered in java (via DeferredRegister/RegistryObjects). This is required to datagen objects using RegistryOps.
• When the server starts, using the registries' loading codecs, builtin objects are deep-copied and pasted into the server's personal RegistryAccess, and then json data from datapacks is loaded and pasted into the server's RegistryAccess (possibly overriding builtin objects).
• When a client logs in, any synced datapack registries and their contents are sent to the client using the network codecs, and stored in the client's personal RegistryAccess (which is stored in the client's connection). Clients do not have to have any of the server's registrables pre-registered (and any builtin registrables are ignored on the client), but the synced registries themselves are required to have been registered on the client.

Userdev APIs

When using a RegistryBuilder to register a registry (either via DeferredRegister or NewRegistryEvent), the userdev may invoke RegistryBuilder#dataPackRegistry(Codec<T> codec, @Nullable Codec<T> networkCodec), which marks the registry as a datapack registry and registers codecs for it. An overload that only takes the loading codec is also provided (using null for the network codec). If the network codec is non-null, the registry is considered to be a synced registry. Unsynced datapack registries registered on the server do not need to be registered on clients, but synced registries do. RegistryBuilder#disableSync is ignored as datapack registries use the presence of the network codec (or lack thereof) to determine whether to sync the registry.

Registries that are named modid:folder cause data to be loaded from data/<datapack_namespace>/modid/folder/. The folder path can include / for subdirectorying, e.g. worldgen/biome.

Network API Changes

• Bump netversion from 2 to 3.
• The S2CModList packet now may or may not contain a list of synced non-vanilla datapack registry IDs at the end of the packet bytestream.
• Backwards-compatibility is kept for clients on netversion 2, see below. This backwards-compatibility may be dropped in 1.19 (sending an empty list instead of nothing).

End-User Concerns

If no changes are made to login netcode, then if a client were to attempt to connect to a server when the server has syncable datapack registries that don't exist on the client, the login packet fails to decode and the client is presented with an unreadable error message (due to the decoder dumping the entire contents of all synced datapack registries into the login error screen).

To avoid this, we validate the syncability of custom datapack registries in these ways:

• The server disallows forge clients on netversion 2 and vanilla clients from joining the server if the server has any non-vanilla syncable datapack registries
• Forge clients on netversion 3 check incoming registries in the mod list packet and terminate their connection if any of the incoming datapack registries don't exist on the client

Forge clients on netversion 2 will experience an "incompatible server" warning in the server list menu due to the netcode version mismatch, however they can still join the server if the server has no syncable custom datapack registries..

Summary of Patches

• RegistryAccess has a hook patched into its codec registry so forge can add custom datapack registries' codecs later
• RegistryResourceAccess has a patch to include the registry namespace in the directory (like tags) if namespace isn't "minecraft" and the registry was created via the RegistryBuilder API

[sciwhiz12]

I tried to test this PR using the provided test mod, but I got an error while trying to load a singleplayer world in DataPackRegistriesTest.ClientEvents#onClientTagsUpdated because the player is null (and the attempt to read the connection field fails). I've fixed it in my local copy by getting the RegistryAccess from TagsUpdatedEvent#getTagRegistry instead.

---

• Datapack registries cannot be reloaded with /reload, they can only be reloaded by rebooting the server

[...]

Lifecycle of datapack registries:

[...]

• When the server starts, using the registries' loading codecs, builtin objects are deep-copied and pasted into the server's personal RegistryAccess, and then json data from datapacks is loaded and pasted into the server's RegistryAccess (possibly overriding builtin objects).

To clarify for me, this means that only the JSON data from datapacks which affect a datapack registry that are present during server startup will affect the contents of the datapack registry, even if said JSON data is removed later on and a /reload is issued?

[sciwhiz12]

What are the cases that a data pack registry registers a network codec that is different from the main codec?

[SizableShrimp]

See vanilla usages. Stuff that is only ever needed/used/processed by the server is just wasted networking data if sent through the main codec over networking.

[Commoble]

There also exist codecs that don't work at all when used for networking because they're hardcoded to be used by RegistryOps and the netcode doesn't use RegistryOps.

[sciwhiz12]

Formatting: space after catch keyword.

Suggested change

	catch(IOException e) // we're inside this ifpresent so the throws can't deal with this
	catch (IOException e) // we're inside this ifpresent so the throws can't deal with this

Also, perhaps you mean as ("as the throws can't deal with this") and not so? Unless I'm interpreting the comment's intention incorrectly, the reason the catch exists is because we cannot propagate the IOException upwards normally because of the use of a lambda, so as would be more semantically correct here.

[Commoble]

You're interpreting it correctly, but I don't think that would be a correct use of "as" either. I'll use "because" for minimum ambiguity -- "The throws can't deal with this exception, because we're inside this ifPresent."

[Commoble]

To clarify for me, this means that only the JSON data from datapacks which affect a datapack registry that are present during server startup will affect the contents of the datapack registry, even if said JSON data is removed later on and a /reload is issued?

Yes, after the first load of datapack registries during server startup, the datapack registry jsons are never read again until the server reboots.

I tried to test this PR using the provided test mod, but I got an error while trying to load a singleplayer world in DataPackRegistriesTest.ClientEvents#onClientTagsUpdated because the player is null (and the attempt to read the connection field fails). I've fixed it in my local copy by getting the RegistryAccess from TagsUpdatedEvent#getTagRegistry instead.

I really want to make sure the player's connection's registryaccess gets populated, as that's what clientside mods should be querying during server runtime -- I'll try moving the test to a different event again.

[sciwhiz12]

The test works now without changes.

---

As a side-note, during testing of this PR, I noticed that the /forge tags command doesn't suggest dynamic/datapack registries, but works fine when manually entered.

With help from @Commoble, the cause and solution was found. The cause was because the registry suggestor used Registry.REGISTRY, which only contains the static registries and excludes the dynamic registries. As a fix, one can replace TagsCommand#suggestRegistries with the following snippet:

private static CompletableFuture<Suggestions> suggestRegistries(final CommandContext<CommandSourceStack> ctx,
        final SuggestionsBuilder builder)
{
    ctx.getSource().registryAccess().registries()
            .map(RegistryAccess.RegistryEntry::key)
            .map(ResourceKey::location)
            .map(ResourceLocation::toString)
            .forEach(builder::suggest);
    return builder.buildFuture();
}

This will be fixed soon, either through a PR from me to 1.18 or in the 1.19 branch.