Help, training is OK but evaluation had problem

[AI-Passionner]

@FateScript, need your help. Here is the summary.
windows 10, python3.8, pycharm

1. The YOLOX codebase works in the other computer
2. It works for the VOC dataset
3. It seems training is OK. The error happens during the evaluation step.
4. Googled, couldn't figure out.
   5.ninja installed. but run

ninja --version
1.10.2.git.kitware.jobserver-1

100%|##################################################################################################################################################################################################| 80/80 [00:26<00:00, 3.00it/s]
2022-05-11 20:15:28 | INFO | yolox.evaluators.coco_evaluator:235 - Evaluate in main process...
2022-05-11 20:15:28 | INFO | yolox.evaluators.coco_evaluator:268 - Loading and preparing results...
2022-05-11 20:15:29 | INFO | yolox.evaluators.coco_evaluator:268 - DONE (t=0.17s)
2022-05-11 20:15:29 | INFO | pycocotools.coco:366 - creating index...
2022-05-11 20:15:29 | INFO | pycocotools.coco:366 - index created!
Using C:\Users\AppData\Local\torch_extensions\torch_extensions\Cache\py38_cu113 as PyTorch extensions root...
C:\Users\Anaconda3\envs\YOLOX\lib\site-packages\torch\utils\cpp_extension.py:322: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Emitting ninja build file C:\Users\u\AppData\Local\torch_extensions\torch_extensions\Cache\py38_cu113\fast_cocoeval\build.ninja...
INFO: Could not find files for the given pattern(s).
2022-05-11 20:15:29 | INFO | yolox.core.trainer:195 - Training of experiment is done and the best AP is 0.00
2022-05-11 20:15:29 | ERROR | yolox.core.launch:98 - An error has been caught in function 'launch', process 'MainProcess' (7892), thread 'MainThread' (7812):
Traceback (most recent call last):

File "c:\users\u\pycharmprojects\yolox\yolox\layers\jit_ops.py", line 83, in load
return importlib.import_module(self.absolute_name())
│ │ │ └ <function FastCOCOEvalOp.absolute_name at 0x000002AA52589CA0>
│ │ └ <yolox.layers.jit_ops.FastCOCOEvalOp object at 0x000002AA06913AF0>
│ └ <function import_module at 0x000002AA04778DC0>
└ <module 'importlib' from 'C:\Users\u\Anaconda3\envs\YOLOX\lib\importlib\init.py'>

File "C:\Users\u\Anaconda3\envs\YOLOX\lib\importlib_init_.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
│ │ │ │ │ └ 0
│ │ │ │ └ None
│ │ │ └ 0
│ │ └ 'yolox.layers.fast_cocoeval'
│ └ <function _gcd_import at 0x000002AA041DE430>
└ <module 'importlib._bootstrap' (frozen)>

File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 973, in _find_and_load_unlocked

ModuleNotFoundError: No module named 'yolox.layers.fast_cocoeval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "tools\train.py", line 136, in
launch(
└ <function launch at 0x000002AA0C638EE0>

File "c:\users\u\pycharmprojects\yolox\yolox\core\launch.py", line 98, in launch
main_func(*args)
│ └ (╒═══════════════════╤═══════════════════════════════════════════════════════════════════════════════════════════════════════...
└ <function main at 0x000002AA0E49EAF0>

File "tools\train.py", line 120, in main
trainer.train()
│ └ <function Trainer.train at 0x000002AA0EE05F70>
└ <yolox.core.trainer.Trainer object at 0x000002AA0EE0AEE0>

File "c:\users\u\pycharmprojects\yolox\yolox\core\trainer.py", line 76, in train
self.train_in_epoch()
│ └ <function Trainer.train_in_epoch at 0x000002AA0EE0B5E0>
└ <yolox.core.trainer.Trainer object at 0x000002AA0EE0AEE0>

File "c:\users\u\pycharmprojects\yolox\yolox\core\trainer.py", line 86, in train_in_epoch
self.after_epoch()
│ └ <function Trainer.after_epoch at 0x000002AA0EE0B940>
└ <yolox.core.trainer.Trainer object at 0x000002AA0EE0AEE0>

File "c:\users\u\pycharmprojects\yolox\yolox\core\trainer.py", line 222, in after_epoch
self.evaluate_and_save_model()
│ └ <function Trainer.evaluate_and_save_model at 0x000002AA0EE0BC10>
└ <yolox.core.trainer.Trainer object at 0x000002AA0EE0AEE0>

File "c:\users\u\pycharmprojects\yolox\yolox\core\trainer.py", line 325, in evaluate_and_save_model
ap50_95, ap50, summary = self.exp.eval(
│ │ └ <function Exp.eval at 0x000002AA0EE0B550>
│ └ ╒═══════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════...
└ <yolox.core.trainer.Trainer object at 0x000002AA0EE0AEE0>

File "c:\users\u\pycharmprojects\yolox\yolox\exp\yolox_base.py", line 321, in eval
return evaluator.evaluate(model, is_distributed, half)
│ │ │ │ └ False
│ │ │ └ False
│ │ └ YOLOX(
│ │ (backbone): YOLOPAFPN(
│ │ (backbone): CSPDarknet(
│ │ (stem): Focus(
│ │ (conv): BaseConv(
│ │ (conv): ...
│ └ <function COCOEvaluator.evaluate at 0x000002AA0EDFE3A0>
└ <yolox.evaluators.coco_evaluator.COCOEvaluator object at 0x000002AA6483E970>

File "c:\users\u\pycharmprojects\yolox\yolox\evaluators\coco_evaluator.py", line 195, in evaluate
eval_results = self.evaluate_prediction(data_list, statistics)
│ │ │ └ tensor([ 9.5465, 0.5645, 79.0000], device='cuda:0')
│ │ └ [{'image_id': 1, 'category_id': 4, 'bbox': [1537.7379150390625, 299.33587646484375, 825.9945068359375, 699.987548828125], 'sc...
│ └ <function COCOEvaluator.evaluate_prediction at 0x000002AA0EDFE4C0>
└ <yolox.evaluators.coco_evaluator.COCOEvaluator object at 0x000002AA6483E970>

File "c:\users\u\pycharmprojects\yolox\yolox\evaluators\coco_evaluator.py", line 276, in evaluate_prediction
cocoEval = COCOeval(cocoGt, cocoDt, annType[1])
│ │ │ └ ['segm', 'bbox', 'keypoints']
│ │ └ <pycocotools.coco.COCO object at 0x000002AA52555A90>
│ └ <pycocotools.coco.COCO object at 0x000002AA648FDAC0>
└ <class 'yolox.layers.fast_coco_eval_api.COCOeval_opt'>

File "c:\users\u\pycharmprojects\yolox\yolox\layers\fast_coco_eval_api.py", line 24, in init
self.module = FastCOCOEvalOp().load()
│ └ <class 'yolox.layers.jit_ops.FastCOCOEvalOp'>
└ <yolox.layers.fast_coco_eval_api.COCOeval_opt object at 0x000002AA1898B700>

File "c:\users\u\pycharmprojects\yolox\yolox\layers\jit_ops.py", line 87, in load
return self.jit_load(verbose)
│ │ └ True
│ └ <function JitOp.jit_load at 0x000002AA525898B0>
└ <yolox.layers.jit_ops.FastCOCOEvalOp object at 0x000002AA06913AF0>

File "c:\users\u\pycharmprojects\yolox\yolox\layers\jit_ops.py", line 102, in jit_load
op_module = load(
└ <function load at 0x000002AA7CB40670>

File "C:\Users\u\Anaconda3\envs\YOLOX\lib\site-packages\torch\utils\cpp_extension.py", line 1144, in load
return _jit_compile(
└ <function _jit_compile at 0x000002AA7CB40790>

File "C:\Users\u\Anaconda3\envs\YOLOX\lib\site-packages\torch\utils\cpp_extension.py", line 1357, in _jit_compile
_write_ninja_file_and_build_library(
└ <function _write_ninja_file_and_build_library at 0x000002AA7CB408B0>

File "C:\Users\u\Anaconda3\envs\YOLOX\lib\site-packages\torch\utils\cpp_extension.py", line 1456, in _write_ninja_file_and_build_library
_write_ninja_file_to_build_library(
└ <function _write_ninja_file_to_build_library at 0x000002AA7CB40EE0>

File "C:\Users\u\Anaconda3\envs\YOLOX\lib\site-packages\torch\utils\cpp_extension.py", line 1898, in _write_ninja_file_to_build_library
_write_ninja_file(
└ <function _write_ninja_file at 0x000002AA7CB40F70>

File "C:\Users\u\Anaconda3\envs\YOLOX\lib\site-packages\torch\utils\cpp_extension.py", line 2023, in _write_ninja_file
cl_paths = subprocess.check_output(['where',
│ └ <function check_output at 0x000002AA04EC8940>
└ <module 'subprocess' from 'C:\Users\u\Anaconda3\envs\YOLOX\lib\subprocess.py'>

File "C:\Users\u\Anaconda3\envs\YOLOX\lib\subprocess.py", line 415, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
│ │ │ └ None
│ │ └ -1
│ └ (['where', 'cl'],)
└ <function run at 0x000002AA04EC89D0>

File "C:\Users\u\Anaconda3\envs\YOLOX\lib\subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
│ │ │ └ ['where', 'cl']
│ │ └ <subprocess.Popen object at 0x000002AA18D384F0>
│ └ 1
└ <class 'subprocess.CalledProcessError'>

subprocess.CalledProcessError: Command '['where', 'cl']' returned non-zero exit status 1.

[AI-Passionner]

Haha, it was solved, although it is not easy, taking me almost half day.
This error is directly related to the last line `subprocess.CalledProcessError: Command '['where', 'cl']' returned non-zero exit status 1.'

1. By googling, I found a solution for ['where', 'cl'] in subprocess.CalledProcessError: Command '['where', 'cl']' returned non-zero exit status 1. HRNet/HRNet-Semantic-Segmentation#39
2. Secondly in Pycharm, file/settings/tool/terminal, the shell path: cmd.exe (or powershell.exe)
3. Have to close the command line and restart.

""""
for few of us who cannot read Chinese :

Add two paths to the Path item in the environment variable (remember to add a semicolon):
E:\VS2017\VC\Tools\MSVC\14.10.25017\bin\HostX64\x64;
E:\VS2017\Common7\IDE
The first line is the directory of the compiler, choose it according to the computer, but at least HostX86\x86 is available for all platforms, and the 64-bit system is filled in as in the example

Create a new LIB variable in the environment variable and add three paths (remember to add a semicolon):
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.10.25017\lib\x64;
C:\Program Files (x86)\Windows Kits\10\Lib\10.0.14393.0\ucrt\x86;
C:\Program Files (x86)\Windows Kits\8.1\Lib\winv6.3\um\x86;
The first path is the lib path, if HostX64\x64 is filled in the path variable, then x64 lib is used
I don’t know if the second path is filled in, but if I fill in the latest version, no problem is found so far.

Create a new INCLUDE variable in the environment variable and add two paths (remember to add a semicolon):
E:\VS2017\VC\Tools\MSVC\14.10.25017\include;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.14393.0\ucrt;
The second path corresponds to the second path above

Click the batch program vcvarall.bat in the path E:\VS2017\VC\Auxiliary\Build. Then open the command line, enter cl, if no error is reported, the configuration is successful...
""""""