QICE metric is orders of magnitude smaller

[Arya-Honarpisheh]

I ran the code with the specified conditions in the scripts file, and the QICE metric is usually very small. Is there anything specific about that?

For example, the code:

export PYTHONPATH=./ CUDA_DEVICE_ORDER=PCI_BUS_ID \ python3 ./src/experiments/NsDiff.py \ --dataset_type="ExchangeRate" \ --device="cuda:0" \ --batch_size=32 \ --horizon=1 \ --pred_len=24 \ --windows=168 \ --rolling_length=24 \ --epochs=50 \ --patience=10 \ runs --seeds='[1, 2, 3]'

outputs

test_results: {'crps': 0.15433384478092194, 'crps_sum': 0.6592183709144592, 'mae': 0.17655259370803833, 'mse': 0.050670161843299866, 'picp': 0.0005942450952716172, 'qice': 0.039921876043081284, 'rmse': 0.2251003384590149}
{"crps": 0.17393125593662262, "crps_sum": 0.5157219171524048, "mae": 0.22581951320171356, "mse": 0.0931800976395607, "picp": 0.0005098978872410953, "qice": 0.0442708358168602, "rmse": 0.30525416135787964}
{"crps": 0.14975784718990326, "crps_sum": 0.7755491137504578, "mae": 0.1927216798067093, "mse": 0.06403543800115585, "picp": 0.0004980299272574484, "qice": 0.039270833134651184, "rmse": 0.2530522346496582}
{"crps": 0.15433384478092194, "crps_sum": 0.6592183709144592, "mae": 0.17655259370803833, "mse": 0.050670161843299866, "picp": 0.0005942450952716172, "qice": 0.039921876043081284, "rmse": 0.2251003384590149}

QICE is around 6, but I get 0.0399.

[wwy155]

QICE is calculated as Interval percentage averages, so is in % unit.
For clarity, in the paper, QICE is multiplied by 100 (% unit), as in many previous works e.g. TMDM.

In your posted log, QICE should be 3.99 in the paper.

[wwy155]

I think the issue is clearly stated in my response.
The issue is thus closed.