I'm going to have a casual relationship with an ex's housemate.
Why just a casual relationship?
It's just for fun, no feelings are related to it. It's just fun.
As long as your ex is okay with it, it's okay to have a casual relationship with any consenting adult. Have you talked with your ex? I would make sure that it would be okay with them first before committing to the relationship no matter how casual it is.
My ex is on a holiday and there's no way I can't have fun with housemate. I'm going to ask him out on weekend.
Good for you, see how it goes. As long as you aren't doing this to pester your ex.