Evaluation failing because target is not integer Tensor.

[geoffreyangus]

When evaluating with Ray, the user may get the following error:

 File "/home/ray/anaconda3/lib/python3.7/site-packages/torchmetrics/classification/accuracy.py", line 222, in update
    mode = _mode(preds, target, self.threshold, self.top_k, self.num_classes, self.multiclass, self.ignore_index)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/torchmetrics/functional/classification/accuracy.py", line 66, in _mode
    ignore_index=ignore_index,
  File "/home/ray/anaconda3/lib/python3.7/site-packages/torchmetrics/utilities/checks.py", line 265, in _check_classification_inputs
    _basic_input_validation(preds, target, threshold, multiclass, ignore_index)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/torchmetrics/utilities/checks.py", line 44, in _basic_input_validation
    raise ValueError("The `target` has to be an integer tensor.")
ValueError: The `target` has to be an integer tensor.

This error can be reproduced by running the following:

from ludwig.api import LudwigModel
from ludwig.backend import RAY

model = LudwigModel(config, backend="ray")
model.train(
    dataset=data_csv,
    skip_save_training_description=True,
    skip_save_training_statistics=True,
    skip_save_model=True,
    skip_save_progress=True,
    skip_save_log=True,
    skip_save_processed_input=True,
    output_directory=output_dir,
)
model.evaluate(
    dataset=data_csv,
    skip_save_unprocessed_output=True,
    skip_save_predictions=True,
    skip_save_eval_stats=True,
    collect_predictions=False,
    collect_overall_stats=False,
    output_directory=output_dir,
)

Alternatively, simply run a test which calls model.evaluate such as tests/integration_tests/test_api.py::test_api_skip_parameters_evaluate where all skip params are True, and set the LudwigModel created in run_api_commands to have backend="ray".

[geoffreyangus]

#1992 attempts to fix the bug. Two things, however:

1. The regression test included in that PR unfortunately doesn't actually surface the bug. Follow-up testing showed that category output features needs to have vocab_size > 2 in order for the bug to appear. Not currently sure why this is the case.
2. Piero brings up the point that the casting fix in dask.py may cause a loss of precision. Will investigate and update here.

[geoffreyangus]

Seems related: dask/dask#3907

[geoffreyangus]

Fixed in #2029.

[Aslan1995-guo]

Sorry, I had the same problem of evaluating accuracy in Ray. How did you solved it?