Move vehicle part variants to json

[irwiss]

Summary

None

Purpose of change

Move hardcoded variants and the stuff that works around the fact that variants are hardcoded into normal json

Fixes #64827

Describe the solution

JSON changes are mostly done by script, can be found in dev branch. The script is a bit of a mess but looks to be doing the correct changes (it prints most changes it makes).

Makes variants field for vparts, collects all the symbol/broken_symbol/symbols fields into it, each object in it defines a variant with a unique id, label for translation, symbol and symbol broken. At least 1 variant is required per part, the first variant is defaulted to if no variant is specified.

symbols define either string of 1 character for non-rotating or 8 characters for rotating; where first character = north, second = NE, third = E etc

The symbols are normal ascii, a small amount of unicode box drawing symbols are accepted for the lines and corners, which get translated in code to curses ACS, json modders can human-read them and copy/paste instead of remembering which vi-like key is which corner/direction

Variants in vehicle prototypes were split by _ which makes splitting to id and variant a bit of guesswork; depends on specific variant order and relies on not defining specific strings as part of part id, they are now split by # character (it is now also an error to use it in vpart id), so there's no guesswork involved

By stuff that works around hardcoded variants I mean the windowshield_full, saddle_scooter etc - parts that are exactly the same but have different id to work around the hardcoded variants map. They are migrated to their "normal" vpart with variant name prepended with _full, _scooter etc. To do this the variants_base field is used - it defines a list of prefixes, which are then cartesian multiplied by them, e.g. wheel_small defines "variants_base": ["scooter"], and front/rear in variants; this will generate variants for [front, rear, scooter_front, scooter_rear].

Tilesets shouldn't require an update for the most part except for minor string replaces as some ids didn't fit with the rest of the ids:
seat_windshield_leather->seat_leather_windshield
reclining_seat_windshield_leather->reclining_seat_leather_windshield

Variants also perform a small looks_like search - each resolution failure will chop off right side of the string until a _, so when reclining_seat_leather_windshield is not present in a tileset looks_like will search for reclining_seat_leather and then fallback to reclining_seat (no variant tile)

Describe alternatives you've considered

Reorder the fields while i'm at it, vpart definitions are super annoying to read when you have to search top to bottom for the type/abstract/id/copy-from fields which appear all over the place, the price is 4 more files and around +500 -500 lines in the diff, but gave up as PR is already super chonk

Testing

There's part dumps in the dev branch master...irwiss:Cataclysm-DDA:vpart-variants-dev but most changes are visual so the game needs to be run

There might be minor hiccups but main tilesets tested - MSXotto, Ultica work correctly
smashbutton iso also worked correctly, i didn't notice changes in level of brokenness in ultica_iso

For non-tiles there seems to be some small issues related to symbols for parts that have no variants, e.g. small battery appliance defines b for symbol, which made it draw a corner rather than b letter, now draws a b, freezer was H which was drawn as =, now it is draws an actual H character, but they should be solvable with later PRs, if they are in fact incorrect and not mistakes

Loading older saves also seems to be working, migrations aren't 100% lossless - the _full variants convert to default "full" variant, the difference should be trivial to fix vs a non trivial amount of migration entries to generate otherwise

Additional context

Apparently not the first time this is looked at https://discord.com/channels/598523535169945603/598529174302490644/690345214736728269

Player-facing stuff should mostly remain the same, there's no "base"s to choose, instead every cosmetic variant is similar to any other

Backup of the python script as I'm deleting the dev branch:

#!/usr/bin/env python3

import argparse
import json
import os
import shutil
from subprocess import PIPE, run

args_parser = argparse.ArgumentParser()
args_parser.add_argument("dir", action="store", help="specify json directory")
args_parser.add_argument("migrations_dir", action="store", help="specify json migrations directory")
args_parser.add_argument("--formatter", help="path to formatter executable", default="tools/format/json_formatter.exe")
args = vars(args_parser.parse_args())

if shutil.which(args["formatter"]) is None:
    print(f"formatter '{args['formatter']}' seems not existing or not runnable")
    exit(1)


def format_json(json_in):
    p = run([args["formatter"]], input=json_in, stdout=PIPE, stderr=PIPE, text=True, encoding='utf-8')
    return p.stdout


def object_to_json(obj):
    json_string = json.dumps(obj, ensure_ascii=False)
    return format_json(json_string)


def write_object_as_json_file(filename, json_in):
    with open(filename, "w", encoding="utf-8", newline="\n") as fp:
        json_formatted = object_to_json(json_in)
        fp.write(json_formatted)

labels = {
"cover_left": "Cover Left",
"cover_right": "Cover Right",
"hatch_wheel_left": "Hatchback Wheel Left",
"hatch_wheel_right": "Hatchback Wheel Right",
"wheel_left": "Wheel Left",
"wheel_right": "Wheel Right",
"cross_unconnected": "Unconnected Cross",
"cross": "Cross",
"horizontal_front_edge": "Front Edge Horizontal",
"horizontal_front": "Front Horizontal",
"horizontal_rear_edge": "Rear Edge Horizontal",
"horizontal_rear": "Rear Horizontal",
"horizontal_2_front": "Front Thick Horizontal",
"horizontal_2_rear": "Rear Thick Horizontal",
"ne_edge": "Front Right Corner",
"nw_edge": "Front Left Corner",
"se_edge": "Rear Right Corner",
"sw_edge": "Rear Left Corner",
"vertical_right": "Right Vertical",
"vertical_left": "Left Vertical",
"vertical_2_right": "Right Thick Vertical",
"vertical_2_left": "Left Thick Vertical",
"vertical_T_right": "Right T Joint",
"vertical_T_left": "Left T Joint",
"front_right": "Front Right",
"front_left": "Front Left",
"rear_right": "Rear Right",
"rear_left": "Rear Left",
"cover": "Cover",
"vertical": "Vertical",
"horizontal": "Horizontal",
"vertical_2": "Thick Vertical",
"horizontal_2": "Thick Horizontal",
"ne": "Front Right",
"nw": "Front Left",
"se": "Rear Right",
"sw": "Rear Left",
"front": "Front",
"rear": "Rear",
"left": "Left",
"right": "Right",
}

suffixes = ["_" + x for x in labels.keys()]

variants_bases = {
    "box": { "box_wheelbarrow": "wheelbarrow" },
    "door": { "door_full": "full" },
    "hddoor": { "hddoor_full": "full" },
    "hddoor_opaque": { "hddoor_opaque_full": "full" },
    "seat_leather": { "seat_windshield_leather": "windshield" },
    "door_opaque": { "door_opaque_full": "full" },
    "reclining_seat_leather": { "reclining_seat_windshield_leather": "windshield" },
    "reclining_seat": { "reclining_seat_windshield": "windshield" },
    "saddle": { "saddle_pedal": "pedal", "saddle_motor": "motor", "saddle_scooter": "scooter" },
    "seat": { "seat_windshield": "windshield", "seat_swivel_chair": "swivel_chair" },
    "basketlg": { "basketlg_cart": "cart" },
    "wheel_small": { "wheel_small_scooter": "scooter" },
    "basketsm": { "basketsm_bike_rear": "bike_rear", "basketsm_wheelchair": "wheelchair" },
    "windshield": { "windshield_full": "full" },
    "reinforced_windshield": { "reinforced_windshield_full": "full" },
}
variant_bases_labels = {
    "wheelbarrow": "Wheelbarrow",
    "full": "Full",
    "windshield": "Windshield",
    "pedal": "Pedals",
    "motor": "Motorcycle",
    "scooter": "Scooter",
    "swivel_chair": "Swivel Chair",
    "cart": "Cart",
    "bike_rear": "Bike Rear",
    "wheelchair": "Wheelchair",
}
variant_separator = "_"
converted_types = {}
converted_count = {}
types = {}
type_variants = {}
missing_variants = {}
part_id_migrations = {}
bases_remaps = {}

def load_vpart_types(path):
    # Having troubles with this script halting?
    # Uncomment below to find the file it's halting on
    #print(path)
    with open(path, "r", encoding="utf-8") as json_file:
        try:
            json_data = json.load(json_file)
        except json.JSONDecodeError:
            print(f"Json Decode Error at {path}")
            print("Ensure that the file is a JSON file consisting of"
                  " an array of objects!")
            return

        for jo in json_data:
            if type(jo) is not dict:
                break

            if "type" not in jo or jo["type"] != "vehicle_part":
                continue
            id = None
            if "id" in jo:
                id = jo["id"]
            elif "abstract" in jo:
                id = jo["abstract"]
            types[id] = jo


def fetch_copied_field(jo, field, default):
    if field in jo:
        return jo[field]
    if "copy-from" in jo:
        if "id" in jo and jo["copy-from"] != jo["id"]:
            return fetch_copied_field(types[jo["copy-from"]], field, default)
        if "abstract" in jo and jo["copy-from"] != jo["abstract"]:
            return fetch_copied_field(types[jo["copy-from"]], field, default)
    return default


# splits id_variant string to [id, variant] according to the possible suffixes
def split_id_and_variant(id_concatted):
    for suffix in suffixes:
        if id_concatted.endswith(suffix):
            len_suffix = len(suffix)
            len_from = len(id_concatted)
            id = id_concatted[:len_from - len_suffix]
            variant = id_concatted[len(id)+1:]
            if id in types:
                return [id, variant]
    return [id_concatted, ""]


# logs the converted id_var pairs
def converted_log(from_var: str, to_var: str):
    if from_var == to_var:
        return
    converted_types[from_var] = to_var
    if from_var in converted_count:
        converted_count[from_var] += 1
    else:
        converted_count[from_var] = 1


# concats [id, var] with separator
def concat_id_to_variant(id, variant, separator = '#'):
    if variant == "": return id
    return id + separator + variant


# converts id, variant pair into the new id/variants
# accounts for migrated _full parts
def convert_id_and_variant(id, variant):
    old_variant = variant
    old_combined_id = concat_id_to_variant(id, variant, "_")
    if old_combined_id in bases_remaps:
        remap = bases_remaps[old_combined_id]
        new_combined_id = concat_id_to_variant(remap[0], remap[1])
        converted_log(old_combined_id, new_combined_id)
        return new_combined_id
    if old_combined_id in part_id_migrations:
        mig = part_id_migrations[id]
        id = mig["to"]
        if variant == "":
            x:str
            for x in type_variants[mig["to"]]:
                if x.startswith(mig["variant"]):
                    variant = x
                    break
            if variant == '':
                print("failed finding default variant")
                exit(1)
        else:
            # doesn't look like this case is triggered
            exit(1)
    elif id in part_id_migrations:
        mig = part_id_migrations[id]
        id = mig["to"]
        variant = mig["variant"]

    if variant == old_variant and variant == "":
        return id
    if id not in type_variants or variant not in type_variants[id]:
        if id not in missing_variants:
            missing_variants[id] = {}
        if variant not in missing_variants[id]:
            missing_variants[id][variant] = 0
        missing_variants[id][variant] += 1
        variant = ""
    new_combined_id = concat_id_to_variant(id, variant)
    converted_log(old_combined_id, new_combined_id)
    return new_combined_id


# converts part definition in vehicle prototype;
# changes id/variant to new id#variant format
# pulls any extras like ammo/fuel into the object
def convert_variant(part: str | dict, extras: dict ):
    if type(part) is str:
        [id, variant] = split_id_and_variant(part)
        new_id = convert_id_and_variant(id, variant)
        if len(extras) != 0:
            extras = extras.copy()
            extras["part"] = new_id
            return extras
        else:
            return new_id
    elif type(part) is dict:
        [id, variant] = split_id_and_variant(part["part"])
        new_id = convert_id_and_variant(id, variant)
        part["part"] = new_id
        if len(extras) != 0:
            exit(1)
        return part
    if len(extras) != 0:
        extras = extras.copy()
        extras["part"] = part
        return extras
    return part


# moves a key of a dict to the last place
def move_to_last(x, member):
    if type(x) is not dict:
        exit(1)
    if member in x:
        temp = x[member]
        del x[member]
        x[member] = temp


# converts parts array to new object format
def convert_parts(parts, extras):
    new_parts = [convert_variant(x, extras) for x in parts]
    field_order = ["part", "ammo", "ammo_types", "ammo_qty", "fuel", "tools"]
    for p in new_parts:  # make part definitions more human readable with same order
        if type(p) is not dict:
            continue
        for f in field_order:
            move_to_last(p, f)
    return new_parts


def ascii_to_block_draw_utf(x):
    if x == "j": return "│"
    if x == "h": return "─"
    if x == "c": return "┼"
    if x == "y": return "┌"
    if x == "u": return "┐"
    if x == "n": return "┘"
    if x == "b": return "└"
    if x == "H": return "┃"
    if x == "=": return "━"
    return x


def dir_symbols(x):
    # offset is east, we'll convert to north later
    if x == "j": return "h\\j/h\\j/" # vertical line
    if x == "h": return "jhjh" # horizontal line
    if x == "y": return "unby" # corner, top left
    if x == "u": return "nbyu" # corner, top right
    if x == "n": return "byun" # corner, bottom right
    if x == "b": return "yunb" # corner, bottom left
    if x == "^": return ">v<^"
    if x == ">": return "v<^>"
    if x == "v": return "<^>v"
    if x == "<": return "^>v<"
    if x == "c": return "cXcXcXcX"
    if x == "X": return "XcXcXcXc"
    if x == "[": return "-\\[/-\\[/"
    if x == "]": return "-\\]/-\\]/"
    if x == "|": return "-\\|/-\\|/"
    if x == "-": return "|/-\\|/-\\"
    if x == "=": return "┃━┃━"
    if x == "H": return "━┃━┃"
    if x == "\\": return "/-\\|/-\\|"
    if x == "/": return "\\|/-\\|/-"
    return x


def convert_symbols(x):
    ds = list(dir_symbols(x))
    if len(ds) == 8:
        ret = [ascii_to_block_draw_utf(x) for x in ds]
        return "".join(ret[-2:] + ret[:-2])
    elif len(ds) == 4:
        ds = [ds[0], ds[0], ds[1], ds[1], ds[2], ds[2], ds[3], ds[3]]
        ret = [ascii_to_block_draw_utf(x) for x in ds]
        return "".join(ret[-2:] + ret[:-2])
    else:
        return x


def refactor_vpart_type(jo: dict):
    if "id" in jo and jo["id"] in variants_bases:
        bases = [{ "id": x, "label": variant_bases_labels[x] } for x in variants_bases[jo["id"]].values()]
        jo["variants_bases"] = bases
    
    variants = jo["variants"] if "variants" in jo else []
    if "symbol" not in jo and "broken_symbol" not in jo and "symbols" not in jo:
        return
    jo_symbol = fetch_copied_field(jo, "symbol", "h")
    jo_broken_symbol = fetch_copied_field(jo, "broken_symbol", "#")

    if "symbols" in jo:
        for v in jo["symbols"]:
            s = jo["symbols"][v]
            vv = {
                "id": v,
                "label": labels[v],
                "symbols": convert_symbols(s),
                "symbols_broken": jo_broken_symbol
            }
            variants.append(vv)
        del jo["symbols"]
        if "symbol" in jo:
            # make the symbol variant the first one so it can be defaulted into
            jo_sym = ascii_to_block_draw_utf(jo["symbol"])
            default_variant = next((x for x in variants if x["symbols"][0] == jo_sym), None)
            if default_variant is None:
                variants.insert(0, {
                    "symbols": convert_symbols(jo_sym),
                    "symbols_broken": jo_broken_symbol
                })

            del jo["symbol"]
    elif "symbol" in jo:
        variants.append({
            "symbols": jo_symbol,
            "symbols_broken": jo_broken_symbol
        })
        if "symbol" in jo: del jo["symbol"]

    if "broken_symbol" in jo: del jo["broken_symbol"]

    if len(variants) > 0:
        jo["variants"] = variants
    elif "variants" in jo:
        del jo["variants"]


def extract_extra_field(extras, field, pp):
    if field in pp:
        extras[field] = pp[field]
        del pp[field]


def refactor_vehicle_prototype(jo):
    parts = jo["parts"]
    pp: dict
    for pp in parts:
        if "part" in pp:
            extras = {}
            extract_extra_field(extras, "fuel", pp)
            extract_extra_field(extras, "tools", pp)
            extract_extra_field(extras, "ammo", pp)
            extract_extra_field(extras, "ammo_qty", pp)
            extract_extra_field(extras, "ammo_types", pp)
            pp["parts"] = convert_parts([pp["part"]], extras)
            del pp["part"]
        elif "parts" in pp:
            pp["parts"] = convert_parts(pp["parts"], {})
        else:
            exit(1)
    
    jo["parts"] = parts


def refactor_folding_item(jo):
    vars = jo["variables"]
    parts_json = vars["folded_parts"]
    parts = json.loads(parts_json)
    for p in parts:
        old_id = p["id"]
        old_variant = p["variant"] if "variant" in p else ""
        old_combined_id = concat_id_to_variant(old_id, old_variant, '_')
        new_part: str = convert_id_and_variant(old_id, old_variant)
        if old_combined_id != new_part:
            print(f"folding veh item {jo['id']} part {old_combined_id} converted to {new_part}")
            part_split = new_part.split(sep='#')
            p["id"] = part_split[0]
            if len(part_split) > 1:
                p["variant"] = part_split[1]
            else:
                del p["variant"]
   
    new_parts_json = json.dumps(parts, ensure_ascii=False, separators=(',', ':'))
    jo["variables"]["folded_parts"] = new_parts_json


def gen_new(path):
    changed = False
    # Having troubles with this script halting?
    # Uncomment below to find the file it's halting on
    #print(path)
    with open(path, "r", encoding="utf-8") as json_file:
        try:
            json_data = json.load(json_file)
        except json.JSONDecodeError:
            #print(f"Json Decode Error at {path}")
            #print("Ensure that the file is a JSON file consisting of"
            #      " an array of objects!")
            return None

        for jo in json_data:
            if type(jo) is not dict or "type" not in jo:
                continue
            if jo["type"] == "vehicle":
                refactor_vehicle_prototype(jo)
                changed = True
            elif jo["type"] == "TOOL" and "variables" in jo and "vehicle_name" in jo["variables"]:
                refactor_folding_item(jo)
                changed = True
            elif jo["type"] == "vehicle_part":
                refactor_vpart_type(jo)
                changed = True
        
        def should_delete_variant_base(jo):
            return type(jo) is dict and "type" in jo and jo["type"] == "vehicle_part" and "id" in jo and jo["id"] in ids_for_deletion
        
        json_data = [x for x in json_data if not should_delete_variant_base(x)]

    return json_data if changed else None


for root, directories, filenames in os.walk(args["dir"]):
    for filename in filenames:
        if not filename.endswith(".json"):
            continue

        path = os.path.join(root, filename)
        load_vpart_types(path)

# fetch type variants from "symbols" field
for k, t in types.items():
    if "id" not in t: continue
    vpid = t["id"]
    type_variants[vpid] = fetch_copied_field(t, "symbols", [])
    if len(type_variants[vpid]) > 0:
        type_variants[vpid] = list(type_variants[vpid].keys())
    else:
        del type_variants[vpid]
    if vpid in variants_bases:
        extra_variants = []
        if vpid in type_variants:
            extra_variants = [x for x in type_variants[vpid]]
        else:
            extra_variants = [""] + list(variants_bases[vpid].values())
        if vpid not in type_variants:
            type_variants[vpid] = []
        for v in extra_variants:
            if v not in type_variants[vpid]:
                type_variants[vpid].append(v)

# write migrations from "bases" to variants
for k, v in variants_bases.items():
    for k2, v2 in v.items():
        if k not in type_variants:
            print(f"cant find {k} in type variants to fetch default variant")
            exit(1)
        to_variant = v2
        if type_variants[k][0] != "":
            to_variant = v2 + "_" + type_variants[k][0]

        migration = {
            "type": "vehicle_part_migration",
            "from": k2,
            "to": k,
            "variant": to_variant,
        }
        for variant in type_variants[k]:
            if variant == "":
                bases_remaps[k2] = [migration["to"], v2]
            else:
                bases_remaps[k2 + "_" + variant] = [migration["to"], v2 + "_" + variant]
        part_id_migrations[migration["from"]] = migration
        type_variants[migration["to"]] += [v2 + "_" + x for x in type_variants[k]]

    for k in v:
        if k in type_variants:
            del type_variants[k]

# migrated-from part ids will later get cleared from json
ids_for_deletion = list(part_id_migrations.keys())

# convert "bases" to variants
for k in list(type_variants.keys()):
    if k not in part_id_migrations:
        continue
    mig = part_id_migrations[k]
    for v in type_variants[k]:
        type_variants[mig["to"]] += [mig["variant"] + "_" + v]
    del type_variants[k]
for k, mig in part_id_migrations.items(): # parts that had no variants (basketsm etc)
    if k in type_variants:
        continue
    if mig["to"] not in type_variants:
        type_variants[mig["to"]] = []
    type_variants[mig["to"]] += [mig["variant"]]

for root, directories, filenames in os.walk(args["dir"]):
    for filename in filenames:
        if not filename.endswith(".json"):
            continue

        path = os.path.join(root, filename)
        new = gen_new(path)
        if new is not None:
            write_object_as_json_file(path, new)

k:str
for k in sorted(converted_types):
    print(str(converted_count[k]).rjust(4) + " x " + k.ljust(50) + " → " + converted_types[k])

if len(missing_variants) > 0:
    print("================")
    print("Missing variants")
    for id in sorted(missing_variants):
        print(id + ":")
        for variant in sorted(missing_variants[id]):
            print(f"    {str(missing_variants[id][variant]).rjust(5)} {variant}")

migrations_path = os.path.join(args["migrations_dir"], "migration_vehicle_parts.json")
write_object_as_json_file(migrations_path, list(part_id_migrations.values()))

# write tileset changes that need to be applied manually
for k, v in variants_bases.items():
    for k2, v2 in v.items():
        if f"{k}_{v2}" != k2:
            print(f"tileset change needed: replace '{k2}' with '{k}_{v2}'")

[kevingranade]

This pull request has been mentioned on Cataclysm: Dark Days Ahead. There might be relevant details there:

https://discourse.cataclysmdda.org/t/changes-to-vehicle-welding/28799/1