Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[torchbench] nvidia_deeprecommender fails to run. #6006

Closed
ysiraichi opened this issue Dec 3, 2023 · 0 comments · Fixed by #6076
Closed

[torchbench] nvidia_deeprecommender fails to run. #6006

ysiraichi opened this issue Dec 3, 2023 · 0 comments · Fixed by #6076
Labels

Comments

@ysiraichi
Copy link
Collaborator

🐛 Bug

Running the upstreamed benchmarking scripts with the following command results in an unexpected error.

python xla/benchmarks/experiment_runner.py \
       --suite-name torchbench \
       --accelerator cuda \
       --xla PJRT --xla None \
       --dynamo openxla --dynamo None \
       --test eval --test train \
       --repeat 30 --iterations-per-run 5 \
       --print-subprocess \
       --no-resume -k nvidia_deeprecommender
Traceback (most recent call last):
  File "xla/benchmarks/experiment_runner.py", line 601, in <module>
    main()
  File "xla/benchmarks/experiment_runner.py", line 597, in main
    runner.run()
  File "xla/benchmarks/experiment_runner.py", line 65, in run
    self.run_single_experiment(experiment_config, model_config)
  File "xla/benchmarks/experiment_runner.py", line 154, in run_single_experiment
    benchmark_model = self.model_loader.load_model(model_config,
  File "xla/benchmarks/benchmark_model.py", line 59, in load_model
    benchmark_model.set_up()
  File "xla/benchmarks/torchbench_model.py", line 196, in set_up
    self.module, self.example_inputs = benchmark.get_module()
  File "torchbenchmark/models/nvidia_deeprecommender/__init__.py", line 40, in get_module
    return self.model.rencoder, (self.model.toyinputs,)
AttributeError: 'DeepRecommenderInferenceBenchmark' object has no attribute 'rencoder'

Environment

Additional context

The issue here is, most likely, that we aren't initializing the model with neither CPU nor CUDA. But, with xm.xla_device(). nvidia_deeprecommender assigns an instance of different classes depending on the device. However, there's no case for XLA devices.

Maybe, a better solution would be to change that part of the benchmark code for handling XLA devices.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant