Add options for cel shading DL generation

[sauraen]

This is the cel shading implementation created by Sauraen and glank in October 2021, used for the Breath of the Wild characters in Triforce% OoT ACE Showcase.

A patch to F3DZEX is also needed, to optionally move the lighting information into the shade alpha channel based on a new geometry mode bit: https://github.com/sauraen/n64celshading

With cel shading turned on for a material, fast64 generates special display lists which draw the triangles multiple times, once per cel level, changing RDP options between each draw pass. This is automated and does not require any CPU intervention or any different treatment from drawing normal display lists or skinned meshes. It is compatible with OoT in-game lighting including multiple moving directional lights, but it does not support colored lights (the green light color is used and the other two channels are ignored).

Cel shading is intended to be used in 1 cycle mode with a single texture or a solid color. Alpha compare is used to draw or not draw each pixel based on the lighting level (shade alpha). After each cel level, the combiner and blender options are changed, usually to invert the threshold for the other cel level. In Triforce%, BotW Link uses three cel levels for his hair, but all other materials on BotW Link and Zelda use the standard two cel levels.

Here is an example of an outputted display list for a solid colored object:

Gfx testDL_Cube_mesh_layer_Opaque_tri_0[] = {
	gsSPVertex(testDL_Cube_mesh_layer_Opaque_vtx_0 + 0, 24, 0),
	gsDPPipeSync(),
	gsDPSetCombineLERP(ENVIRONMENT, 0, PRIMITIVE_ALPHA, 0, 1, SHADE, ENVIRONMENT, 0, ENVIRONMENT, 0, PRIMITIVE_ALPHA, 0, 1, SHADE, ENVIRONMENT, 0),
	gsDPSetPrimColor(0, 0, 255, 255, 255, 200),
	gsDPSetBlendColor(255, 255, 255, 127),
	gsSPDisplayList(testDL_Cube_mesh_layer_Opaque_tri_0_cel0),
	gsDPPipeSync(),
	gsDPSetCombineLERP(1, ENVIRONMENT, PRIMITIVE_ALPHA, ENVIRONMENT, SHADE, 0, ENVIRONMENT, 0, 1, ENVIRONMENT, PRIMITIVE_ALPHA, ENVIRONMENT, SHADE, 0, ENVIRONMENT, 0),
	gsDPSetPrimColor(0, 0, 255, 255, 255, 0),
	gsDPSetBlendColor(255, 255, 255, 128),
	gsSPDisplayList(testDL_Cube_mesh_layer_Opaque_tri_0_cel0),
	gsSPEndDisplayList(),
};

Gfx mat_testDL_f3d_material_layerOpaque[] = {
	gsDPPipeSync(),
	gsSPLoadGeometryMode(G_ZBUFFER | G_SHADE | G_CULL_BACK | G_LIGHTING | G_SHADING_SMOOTH | G_CELSHADING),
	gsSPSetOtherMode(G_SETOTHERMODE_H, 4, 20, G_AD_DISABLE | G_CD_MAGICSQ | G_CK_NONE | G_TC_FILT | G_TF_BILERP | G_TL_TILE | G_TD_CLAMP | G_TP_PERSP | G_CYC_1CYCLE | G_PM_NPRIMITIVE),
	gsSPSetOtherMode(G_SETOTHERMODE_L, 0, 32, G_AC_THRESHOLD | G_ZS_PIXEL | AA_EN | Z_CMP | Z_UPD | CVG_DST_CLAMP | ZMODE_OPA | GBL_c1(G_BL_CLR_IN, G_BL_0, G_BL_CLR_IN, G_BL_1) | GBL_c2(G_BL_CLR_IN, G_BL_0, G_BL_CLR_IN, G_BL_1)),
	gsSPTexture(65535, 65535, 0, 0, 1),
	gsDPSetEnvColor(254, 187, 0, 255),
	gsSPEndDisplayList(),
};

This works on real N64 hardware and on any emulator which does RSP LLE and supports alpha compare in the RDP.

Two other, completely different, implementations of cel shading on the N64 were created in January 2022, one by James Lambert and one by Wiseguy (no public video). Our implementation was made in October 2021 but was hidden in a branch of a fork so not really made public, and is completely different from both of these.

[thecozies]

is there any reason why vertex UV scrolling wouldn't work?

[sauraen]

To be honest I don't understand what this is, so I didn't try to implement compatibility with it. The way scrolling textures is done in OoT involves updating the tile definition every frame; any UV-based scheme would not be able to independently scroll both textures in a multitexture setup. Is this UV scrolling an SM64 thing or is it something made up by the community?

[sauraen]

Cel shading presets have been separated and won't be visible unless you check the box saying you have the microcode patch installed (off by default). If you've tested this branch before, you'll need to delete the presets in ~/.config/blender/X/scripts/presets/f3d/* (except for the user directory). Otherwise the cel shading presets will still show up even if the box isn't checked.

[sauraen]

Closed in favor of V5 version.