Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue in generating scenarios that make sense and are as realistic as possible #39

Closed
basteran opened this issue Jun 10, 2024 · 3 comments

Comments

@basteran
Copy link

Hello everyone, thank you for the great work.

I've been using this repository for a while now trying to generate some scenarios that make sense and are as realistic as possible. Currently, I am facing some little problems, such as:

  • no wc in the bathroom
  • microwave or pans on the floor in the kitchen
  • wardrobes or shelves in front of the windows
  • no isle in the kitchen although I manually specified this constraint to ChatGPT in the prompt
  • huge, but quite empty, bedrooms
  • missing specified objects (e.g. a book on the table, or the remote controller in the sofa)

Do you think this is to be expected? Or is it my problem (maybe the prompt is not that stable and my edits lead to unstable results?)?
I even tried to generate some variations, but I see no improvement besides the aspect and the assets.

Could you help me out?
Thanks again.

@YueYANG1996
Copy link
Collaborator

I encountered some of the problems you raised. Here are some suggestions for each of them:

  • No WC in the bathroom: I feel this is a rare case. It may happen, but usually, the toilet is the most important object in the bathroom, and the holodeck should be able to generate it first.

  • microwave or pans on the floor in the kitchen: I also noticed this microwave issue. One thing you can do is to add another check for the object list here You can have another LLM to verify if this plan is reasonable and you can fix some mistakes here, e.g., the microwave should be on the top of table not on the floor.

  • wardrobes or shelves in front of the windows: you can change this variable to True. In this case, we will treat windows as objects, so the object placement will consider the collision with windows.

  • no isle in the kitchen although I manually specified this constraint to ChatGPT in the prompt: This is due to the lack of good isle assets in our subset of objaverse, but I think most of the time, if the kitchen is large enough, there should be an isle or kitchen counter.

  • huge, but quite empty, bedrooms: you can specify the size of the room in the prompt, for example, "5m x 5m". Or if you want more objects in the room, you can increase this value. This value controls the capacity ratio of objects for each room.

  • missing specified objects (e.g., a book on the table or the remote controller on the sofa): adding them in the prompt can help; however, since scene generation is a long and complicated process, it's hard to ensure every detail can be satisfied.

@basteran
Copy link
Author

Thanks for the exhaustive answer!

missing specified objects (e.g., a book on the table or the remote controller on the sofa): adding them in the prompt can help; however, since scene generation is a long and complicated process, it's hard to ensure every detail can be satisfied.

is there a way to force (or at least to control) the generation and placement of objects? I am interested in creating some realistic scenarios with a few constraints, but I feel like most of the time the constraints I am adding are not satisfied. I have a few examples:

  • a book on the dining table
  • a chair near the sofa
  • a bottle far from the kitchen
  • a blanket (wherever in the environment)

I would like to make sure that at least these constraints are met. I can also modify the code if necessary, but I would at least appreciate some indication of where I should start to modify it.

@YueYANG1996
Copy link
Collaborator

Same way as the microwave solution, adding another step to double-check the object selection plan here. You can implement a function taking in the original object plan and your prompt, and let a LLM to modify the original plan to satisfy your requirement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants