Critical Bug Report: Latest ZhiZi b40c78nbt Model Exhibits Severe Evaluation Inconsistencies in KataGo Analysis (Fan Xiping vs. Huang Yougong, 3-Stone Handicap)

[GuangHui68]

Description
While re-analyzing the historic game "Fan Xiping vs. Huang Yougong (3-Stone Handicap)" using the latest ZhiZi b40c78nbt model, I have encountered severe and persistent bugs.
I want to be clear: my goal is not to "defeat" KataGo by exploiting its bugs. I rely on it to provide clear and trustworthy analysis. When the analysis data exhibits obvious, visually apparent bugs, it fundamentally damages KataGo's credibility.
Before detailing the specific issues in this game, I would like to outline what I consider to be clear indicators of a bug:

Massive Point Swing Between Consecutive Moves: If the evaluation shows a swing of +10 points or more (even +tens of points) between two consecutive moves, this cannot be attributed to both players playing "divine moves." It is almost certainly a blind spot in KataGo's calculation.
Universal Catastrophic Evaluation: If, on a given turn, all candidate moves are evaluated at -10 points or worse (even -30 points or worse), this cannot mean the previous move was a "divine move." It can only indicate a significant oversight or miscalculation.
Extreme Loss Followed by Immediate Massive Gain: If a move incurs a loss of around -100 points, and then a few moves later (or even immediately after) there is a gain of +50 points, this is highly likely a calculation error.
Inconsistent Evaluation of the Same Move: The evaluation (and color indicator) for a single move changes drastically depending on whether it is viewed before it's played, after it's played, or after the next move is played. Furthermore, undoing the move does not revert the evaluation. What is the cause of this?

I propose that KataGo should implement a system to detect these anomalous scenarios and establish a rollback mechanism to re-calculate the evaluations for the preceding few moves to correct these obvious errors. Regardless of how complex the board position is—whether it's a chaotic multi-group fight—if the evaluation becomes this unstable, can we truly trust a program that is supposed to give a 3-stone handicap to top professional players?
Specific Bugs Found in Moves 131-150
Within the short span of just 20 moves (131-150), the following clear bugs are present:

A. Move 140: This is a textbook example of Bug Type #2 (see Figure 2). The preceding move, 139, is shown in Figure 1.
B. Move 142: Bug Type #2 appears again (see Figure 4). The preceding move, 141, is shown in Figure 3.
C. Move 139: Before being played, its evaluation was -8.0 points (Figure 1). However, after playing it, the evaluation changed to dark purple, indicating a loss of more than -12 points (Figure 2). Critically, undoing the move does not revert the evaluation back to -8.0.
D. Move 140: The same issue recurs. In Figure 2, move 140 is evaluated at -14.2 points. But in Figure 3 (after the next move is played), it changes to yellow, meaning a loss of less than -3 points. Why does the evaluation for the same move fluctuate so wildly over just one ply?

The fact that this bug occurs on two consecutive moves indicates a severe systemic error in the entire calculation process, which should be completely restarted. This is the result even at 200,000 visits, representing a massive waste of computational time (the cost of electricity is the least of our concerns).
On a positive note, compared to the previous b28c512 networks, the analysis of White's move 145 in this run was correct and showed no errors.

Attached Files:
Zip package ：
Original Game Record: O1.sgf
Analyzed Game Record: A3.sgf
Reference Figures: 01.png, 02.png, 03.png, 04.png

Environment

UI: Latest KaTrain version
Engine: Latest KataGo TRT version 1.16.4

Models:
kata1-zhizi-b40c768nbt-fdx6c.bin.gz (current strongest)

Analysis settings:
Single move analysis: 200,000 visits
Quick play: 8,000 visits per move
Wide Root Noise: 0.045

numAnalysisThreads = 5
numSearchThreadsPerAnalysisThread = 24
nnMaxBatchSize = 128

Backend: TensorRT
OS: Windows 11
GPU：RTX 3080

在用最新的智子b40c78nbt model重新分析范西屏vs三子 黄友功时，发现仍旧存在巨大bug
我先明确一下，我不是用katago的bug要战胜它，而是需要它提供明确、可信赖的分析结果。如果分析数据出现肉眼可见的bug，其实损害的是katago的信誉
在此之前，我先列出个人认为明显bug的几种情形：
1、前后两手，目差都出现10目+的增加，甚至+数十目。这绝非双方都下出了神之一手，必然是katago计算出现盲点
2、某一手棋，所有的选点，目差都是-10目以上，甚至-30目以上，不可能是上一手是神之一手，只能是出现了漏算
3、某一手出现-100目左右的极大亏损，然后数手或者紧接着出现+50目，大概率是计算失误
4、对同一手的目差计算，下之前、之后、更下一手，出现明显区别（色标巨变），而且回退后不会改变。希望明确这是什么原因？
我建议katago应该检测这种异常情形，并且建立回溯机制，重算前面数手目差，及时纠正这种显然的失误。不管局势有多复杂，是几块棋混战，如果混战就目差混乱，那还是能让职业高段3子的katago么？
接下去，在本局的131-150这短短20手之间，存在如下的明确bug：
A、140手，就是我提出的类型2，见图2；此前的139手，见图1。
B、142手，再次出现类型2，见图4；此前的141手，见图3。
C、139手在下之前，目差-8.0（图1），但下完后变色为-12目以上的大失误（图2），但是倒退回来不会改变。
D、同样的情况在140上再次出现。可以看到，图2时，140是-14.2目，到了图3，就变成了黄色，意味着＜-3目，隔了一手就如此变动目差，why？
连续2手出现这种bug，说明整个计算都有严重错误，应该全部重来。这还是20万visits下的结果，明显是浪费大量的时间（算力的电费倒是小头）
不过相比此前的两个b28c512 ，此次在白145的分析上没有出现失误
原始棋谱，O1.sgf；分析棋谱，A3.sgf。图：01-04

[GuangHui68]

Attachments: ZIP File

06.zip

[GuangHui68]

Bug Supplement Report – Issue #1: Inconsistent & Excessive Score Difference (Moku) Drift Across Moves
As shown in the screenshot for move 135, the observed behavior confirms Issue #1: the score difference (in moku) between Black and White monotonically increases across consecutive moves — and the magnitude is substantial.

In the legacy model b28c512 (Fig. 10–14):
Score differences: +20，+30, +30, +43, +43
→ 4 consecutive moves with large, non-converging swings.
In the updated model b40c768 (Fig. 21–23):
Score differences: +17, +16, +21
→ Still exhibits drift (though reduced in count to 3 moves — “at least it’s shorter” 🙃).

🔍 Conclusion: The core issue persists — the engine’s evaluation is unstable across adjacent playouts/moves, producing inconsistent yet systematically biased score estimates. This undermines reliability for endgame analysis and AI-assisted review.
📎 Supplementary materials (including screenshots & ZIP archive) are attached in a follow-up message.

kata1-zhizi-b28c512nbt-muonfd2.bin.gz
kata1-zhizi-b40c768nbt-fdx6c.bin.gz

bug补充报告：
如图是135的一个手数变化，符合我说的问题1，双方手数的目差，依次上升，而且很巨大
在原先的b28c512 model，如图11-14所示，目差依次为+20、+30、+30、+43、+43；换成新的b40c768，换汤不换药，依次为：+17、+16、+21（图21-图23，好在只有3个了……）

07.zip

[nicewk]

Hi GuangHui68,
建议如下：

1. 可以试一下 LizzieYzy，我在这个界面上没有遇到你描述的情况，也许能帮助更好地对照问题。

2. 建议将最大访问数（maxVisits）设置为 0，让引擎能够充分分析。因为我认为 b40c768nbt 这个模型需要更强的算力和更长的思考时间，限制访问数可能会导致分析不充分。

3. 关于你提到的第 4 个问题，可以尝试在 config 中关闭缓冲相关的参数，看是否与此有关。

以上建议仅参考。

My suggestions are as follows:

1. You may try using LizzieYzy, since I didn’t encounter the issue you described when using it, and it might help provide a clearer comparison.

2. Consider setting maxVisits to 0, so the engine can analyze without limits. I believe the b40c768nbt model requires stronger computation and longer thinking time, so limiting visits might cause incomplete analysis.

3. Regarding the fourth issue you mentioned, you can try disabling the buffering-related parameters in the config file to check whether it’s connected.

These suggestions are for your reference only.

[GuangHui68]

Hi GuangHui68, 建议如下：

1. 可以试一下 LizzieYzy，我在这个界面上没有遇到你描述的情况，也许能帮助更好地对照问题。
2. 建议将最大访问数（maxVisits）设置为 0，让引擎能够充分分析。因为我认为 b40c768nbt 这个模型需要更强的算力和更长的思考时间，限制访问数可能会导致分析不充分。
3. 关于你提到的第 4 个问题，可以尝试在 config 中关闭缓冲相关的参数，看是否与此有关。

以上建议仅参考。

My suggestions are as follows:

1. You may try using LizzieYzy, since I didn’t encounter the issue you described when using it, and it might help provide a clearer comparison.
2. Consider setting maxVisits to 0, so the engine can analyze without limits. I believe the b40c768nbt model requires stronger computation and longer thinking time, so limiting visits might cause incomplete analysis.
3. Regarding the fourth issue you mentioned, you can try disabling the buffering-related parameters in the config file to check whether it’s connected.

These suggestions are for your reference only.

能否告我怎么设置 config 参数？具体哪个？毕竟20万visits，已经很大了

[GuangHui68]

这里是继续发现的一个bug
是136的一个演变：黑13、15冲，17断明明不成立，却非要走下去，到白26；黑貌似发现了27断的“妙手”，所以突然给白-30目
但白28打后，黑又发现算错了，结果这里的每一手，都＞-40目；白30切断后，katago总算看清了，黑+12目

[GuangHui68]

Hi GuangHui68, 建议如下：

1. 可以试一下 LizzieYzy，我在这个界面上没有遇到你描述的情况，也许能帮助更好地对照问题。
2. 建议将最大访问数（maxVisits）设置为 0，让引擎能够充分分析。因为我认为 b40c768nbt 这个模型需要更强的算力和更长的思考时间，限制访问数可能会导致分析不充分。
3. 关于你提到的第 4 个问题，可以尝试在 config 中关闭缓冲相关的参数，看是否与此有关。

以上建议仅参考。

My suggestions are as follows:

1. You may try using LizzieYzy, since I didn’t encounter the issue you described when using it, and it might help provide a clearer comparison.
2. Consider setting maxVisits to 0, so the engine can analyze without limits. I believe the b40c768nbt model requires stronger computation and longer thinking time, so limiting visits might cause incomplete analysis.
3. Regarding the fourth issue you mentioned, you can try disabling the buffering-related parameters in the config file to check whether it’s connected.

These suggestions are for your reference only.

LizzieYzy，搞不定啊…… json的不能安装成功，我也不知道原因。

[GuangHui68]

我继续添加这个超级难局的分析图，这是白143后，黑144在200K下的分析图，此图说明啥？
说明katago出现了超级漏算，突然发现了后面的145尖后，对杀出现了完全逆转，所以疯狂的全局各种乱走，妄图找出一丝希望来。
这种情况，我在它少有的几次大漏算中看见过，其本质表现，与当年 α Go遭到李世石的神之一挖后的表现，完全一样。本质上，程序的内核bug，还是这么奇葩啊……

I'm continuing to add analysis diagrams for this extremely difficult position. This is the analysis diagram after White 143, with Black 144 analyzed at 200K playouts. What does this diagram show?
It shows that Katago experienced a massive oversight — it suddenly discovered that after the subsequent move 145
(a hane/attachment), the capturing race underwent a complete reversal. As a result, the AI went into panic mode, making wild, chaotic moves all over the board, desperately trying to find any glimmer of hope.
I've witnessed this behavior during the rare occasions when Katago makes such major blunders. Its fundamental manifestation is identical to AlphaGo's reaction after Lee Sedol's legendary "divine move" (the wedge/sabaki move). At its core, the program's internal bug remains just as bizarre as ever...

[GuangHui68]

为方便大家理解这个bug，我贴一下，测试的几个model，在黑144、白145的表现
第一个，自然就是最新的 zhizi-b40c768nbt
黑144已经在上一楼了，白145在最新model上表现正常

[GuangHui68]

接下去，是 adam-b28c512nbt，这个以前是很稳定的
且不论 144是否分析对，145的所有选点，全部-40目以上，肯定不对

[GuangHui68]

这个是刚刚公布不久的 kata1-zhizi-b28c512nbt-muonfd2.bin.gz
黑144倒是只有-34目，但是，白145，+71目！太夸张了吧？

[nicewk]

Hi GuangHui68, 建议如下：

1. 可以试一下 LizzieYzy，我在这个界面上没有遇到你描述的情况，也许能帮助更好地对照问题。
2. 建议将最大访问数（maxVisits）设置为 0，让引擎能够充分分析。因为我认为 b40c768nbt 这个模型需要更强的算力和更长的思考时间，限制访问数可能会导致分析不充分。
3. 关于你提到的第 4 个问题，可以尝试在 config 中关闭缓冲相关的参数，看是否与此有关。

以上建议仅参考。
My suggestions are as follows:

1. You may try using LizzieYzy, since I didn’t encounter the issue you described when using it, and it might help provide a clearer comparison.
2. Consider setting maxVisits to 0, so the engine can analyze without limits. I believe the b40c768nbt model requires stronger computation and longer thinking time, so limiting visits might cause incomplete analysis.
3. Regarding the fourth issue you mentioned, you can try disabling the buffering-related parameters in the config file to check whether it’s connected.

These suggestions are for your reference only.

LizzieYzy，搞不定啊…… json的不能安装成功，我也不知道原因。

找config中这一节：

Limits for search on the current turn.

If commented out or unspecified, the default is to have no limit.

maxVisits = 500

maxPlayouts = 300

maxTime = 10.0

[nicewk]

@GuangHui68
你试试lizzieyzy-next

[GuangHui68]

Hi GuangHui68, 建议如下：

1. 可以试一下 LizzieYzy，我在这个界面上没有遇到你描述的情况，也许能帮助更好地对照问题。
2. 建议将最大访问数（maxVisits）设置为 0，让引擎能够充分分析。因为我认为 b40c768nbt 这个模型需要更强的算力和更长的思考时间，限制访问数可能会导致分析不充分。
3. 关于你提到的第 4 个问题，可以尝试在 config 中关闭缓冲相关的参数，看是否与此有关。

以上建议仅参考。
My suggestions are as follows:

1. You may try using LizzieYzy, since I didn’t encounter the issue you described when using it, and it might help provide a clearer comparison.
2. Consider setting maxVisits to 0, so the engine can analyze without limits. I believe the b40c768nbt model requires stronger computation and longer thinking time, so limiting visits might cause incomplete analysis.
3. Regarding the fourth issue you mentioned, you can try disabling the buffering-related parameters in the config file to check whether it’s connected.

These suggestions are for your reference only.

LizzieYzy，搞不定啊…… json的不能安装成功，我也不知道原因。

找config中这一节：

Limits for search on the current turn.

If commented out or unspecified, the default is to have no limit.

maxVisits = 500

maxPlayouts = 300

maxTime = 10.0

能把这个变化中的相关判断，你的katago给出的目差，截图贴一下么，发个zip、rar都可以，谢谢！
至于maxvisits=，其实受控制的是katrain中的选项，这个我选的是4万或5万。新的b40c768太需要算力了，显卡危机啊，最近严重缺钱……
第2项是模拟次数吧，这个是默认的啊，你取多少？我好像这个参数从未设置过

[nicewk]

@GuangHui68
附件是我这里的截屏。

1. 140的评估为白棋62.6
2. 141黑棋-64.6（二选），一选为 -60.6，这和62.6 基本一致
3. 白143并非一选，所以对144的一选达到72没有问题，谱中的144是败招，完全排除在10个预选点之外。
4. 145白棋反杀黑棋，目差达到30左右，评估正常。
5. 146的评估验证145评估没有问题。
   我用的是质子b26c512的模型，在katago的config文件注释掉上面的三个参数。在lizzieyzy也不对最大访问量进行设置。
   用最新的b40c738模型，结果差不多只是要等更长的时间（几乎一倍的时间）。显卡为笔电5090。

1.zip

[nicewk]

@GuangHui68

上面的截屏是对你136bug的回复，你的136评估在这里是二选。目差>42