fix vision models to use the correct device

[Krovatkin]

fix vision models to use the correct device

[wconstab]

Can you also add an assert as @asuhan had described? I'd like to enforce this so it doesn't happen again.

I guess it's not possible to do this outside of the benchmark code as things are structured now. What's the penalty of adding one assert into every measurement? should be a wash, i guess?

[Krovatkin]

I guess it's not possible to do this outside of the benchmark code as things are structured now. What's the penalty of adding one assert into every measurement? should be a wash, i guess?

@wconstab
we should follow up in another PR since

1. for non-vision models, it's a bit tricky to get at models' inputs, let alone models' outputs.
2. I'm thinking we could do a little bit restructuring to make freezing a little bit easier and hopefully this will let us keep asserts outside of eval at least
   3a if we keep asserts within eval and train, new models might have this exact same issue since folks might not just add them
   3b If we decide to do asserts, I'd still like to do measurements to make sure we aren't affecting a score too much.

[wconstab]

ok I will not block this then.