Question about RSX: Precedence between vertex array and vertex register ?

[vlj]

I found why character in After Burner Climax were buggy, it's because of an ambiguity over vertex attribute declaration.

The relevant command dump looks like :

Vertex array 1: (disabled)
Vertex array 1: Offset = 0
Vertex array 7: (disabled)
Vertex array 7: Offset = 0
(invalidate vertex cache file)
(invalidate vertex file)
(invalidate vertex file)
(invalidate vertex file)
(invalidate vertex cache file)
(invalidate vertex file)
(invalidate vertex file)
(invalidate vertex file)
Vertex array 8: (disabled)
Vertex array 8: Offset = 0
(invalidate vertex cache file)
(invalidate vertex file)
(invalidate vertex file)
(invalidate vertex file)
(invalidate vertex cache file)
(invalidate vertex file)
(invalidate vertex file)
(invalidate vertex file)
Vertex array 0: Type = Float size = 3 stride = 64 frequency = 0
Vertex array 0: Offset = 172088752
Vertex array 2: Type = Float size = 3 stride = 64 frequency = 0
Vertex array 2: Offset = 172088764
Vertex array 8: Type = Float size = 2 stride = 64 frequency = 0
Vertex array 8: Offset = 172088776
Vertex array 1: Type = Float size = 4 stride = 64 frequency = 0
Vertex array 1: Offset = 172088784
Vertex array 7: Type = Float size = 4 stride = 64 frequency = 0
Vertex array 7: Offset = 172088800
CELL_GCM_NV4097_SET_TRANSFORM_CONSTANT_LOAD : 0x00000064
Transform constant 0: 1065353216/1.000000
Transform constant 1: 0/0.000000
Transform constant 2: 0/0.000000
Transform constant 3: 0/0.000000
Transform constant 4: 0/0.000000
Transform constant 5: 1065353216/1.000000
Transform constant 6: 0/0.000000
Transform constant 7: 0/0.000000
unknown/illegal method [0x000018c3] : 0x01bb0cc0
unknown/illegal method [0x000018c0] : 0x0000000b
unknown/illegal method [0x000018c1] : 0x10001000
unknown/illegal method [0x000028c1] : 0x00000000
unknown/illegal method [0x000028c2] : 0x00010004
unknown/illegal method [0x000028c3] : 0x00010004
unknown/illegal method [0x00002900] : 0x00003f80
unknown/illegal method [0x00002901] : 0x00000000
unknown/illegal method [0x00002902] : 0x00000000
unknown/illegal method [0x00002903] : 0x00000000
CELL_GCM_NV4097_SET_SHADER_PROGRAM : 0x01bb0c41
CELL_GCM_NV4097_SET_VERTEX_DATA4F_M + 112 : 0x00000000
CELL_GCM_NV4097_SET_VERTEX_DATA4F_M + 116 : 0x00000000
CELL_GCM_NV4097_SET_VERTEX_DATA4F_M + 120 : 0x00000000
CELL_GCM_NV4097_SET_VERTEX_DATA4F_M + 124 : 0x00000000
(invalidate vertex file)
Index array: Address = 0x0a550630
Index array: DMA mode = Local memory type = unsigned short

• Begin: Triangle_strip -
  Draw 37 index starting from 0
• End -

There's no pretty printing for CELL_GCM_NV4097_SET_VERTEX_DATA4F_M yet, but 112 is 7 * 16 so basically there are 2 vertex array 7 declarations, one is a float4 array at offset 172088800, the other is a single one (0., 0., 0., 0.).
Currently RSXThread is using the last declaration as the one to consider but After Burner Climax shows it's wrong. In the CELL_GCM_NV4097_SET_VERTEX_DATA4F_M hook, if I check that previous vertex array size is == 0 (ie the vertex array is disabled) before overwriting vertex array content I have correct rendering in ABC. In addition the transform program input list is the set of all "active" vertex attributes and all vertex register values that were set during current frame and weren't overwritten by a vertex array declaration.

My current theory is that for every 16 vertex attributes :

1. RSX will use vertex array format/offset if the size is > 0.
2. Then it will fallback to the register declared immediate value.
3. To know which vertex input is used by current transform program CELL_GCM_NV4097_SET_VERTEX_ATTRIB_INPUT_MASK is used.

I don't know if it's the complete story or not so I'd like to have some more opinion on the subject.

[tambry]

@AlexAltea knows a lot about RSX, so he might be able to help.

[vlj]

hmmm with some studies on DBZ and SH3 I'm starting to have a more complete figure of how transform program works...

[tambry]

@vlj Can this be closed?

[vlj]

yes