Model Inference Output Issue | Base Color not Showing

[usamashami11]

Hi, kudos for the amazing work, and the results shown in paper looks promising!
However, I'm facing issues while generating/reproducing the exact output as highlighted in the results.
I have created a docker image, and uploaded the provided controlnet weights, as suggested by you.

Now, when I run the inference, the generated 3D model lacks color. Seems like the PBR materials are not generating properly, especially the Albedo output (3rd image in gif), which must include the base color. Here is the 'a brown basketball' evaluation.gif.

I am using this command (same as guided), so I don't know where the issue lies. Maybe the trained model is not working properly, or there is some parameter related issue.
python launch.py --config configs/dreammat.yaml --test --gradio --gpu 0 system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj system.geometry.shape_init_params=0.6 data.blender_generate=true

I'm aiming to achieve these results:

Please help me resolve the issue, as I am doing research in the domain of 3D and really impressed by your work. I am looking forward to getting a quick response from your end, so I can incorporate your work in my pipeline accordingly. Thanks!

[zzzyuqing]

It seems that the values of albedo, roughness, and metallic are still at their initial settings. Could you please check if the material representation is indeed being optimized?

[usamashami11]

I appreciate your prompt response!
Basically, after setting up the repo, I just ran the DreamMat inference code from .cmd/run_examples.sh on the provided Objects (3D meshes) located in the ./load/shapes/objs. Ideally, the code shall work because I tried my best to ensure all dependencies/packages are installed. Can you please confirm from the Inference Command I'm using, if the issue can be probed from there? Or let me know any .py file, where I may dig further.

One more thing, does it have something to do with the system.geometry.shape_init_params=0.6 or any other parameters being passed in the inference command (which I pasted earlier)?
Actually, the commands in cmd/run_examples.sh are for training and not testing, and I myself had to figure out what the inference commands are. So, I'm not sure if the command I used, will generate the desired results. Any guidance in this perspective would be beneficial regarding the issue observed.

[zzzyuqing]

Previously, someone encountered a similar problem when using Docker images, and he mentioned that pulling the image again solved the issue. Although I haven't personally experienced this, you might want to try setting up the environment directly by installing packages in requirements.txt as a possible solution.

Indeed, the cmd/run_examples.sh script is used to generate PBR materials. You can run it directly, and this issue should be unrelated to the parameter settings.

[usamashami11]

Previously, someone encountered a similar problem when using Docker images, and he mentioned that pulling the image again solved the issue. Although I haven't personally experienced this, you might want to try setting up the environment directly by installing packages in requirements.txt as a possible solution.

I switched to building a Dockerfile since the Conda environment was not being setup, due to the issues caused by the dependencies which are based on other GitHub repos e.g.

git+https://github.com/KAIR-BAIR/nerfacc.git@v0.5.2
git+https://github.com/NVlabs/tiny-cuda-nn.git#subdirectory=bindings/torch
git+https://github.com/NVlabs/nvdiffrast.git
git+https://github.com/ashawkey/envlight.git

The Dockerfile had some issues too, but they were less compared to Conda environment. I can try 2 things if the issue is related to Environment Setup

1. Try setting up Anaconda Environment once again (and ask for specific issues observed during setup)
2. Try pulling the Dockerfile you hosted on Docker Hub, since I built the Docker Image from the Dockerfile you provided, currently.

Indeed, the cmd/run_examples.sh script is used to generate PBR materials. You can run it directly, and this issue should be unrelated to the parameter settings.

About this, if you check the file, by opening the following URL:
https://github.com/zzzyuqing/DreamMat/blob/bd33e9122d9f7c05734b5f419af2cd0d621b9f05/threestudio_dreammat/cmd/run_examples.sh

The --train param is passed, which will not initiate the Inference (Testing) rather the full fledge Training (proof below). It must be --test to run the Inference (like I showed earlier)

Last, I don't think it is an Environment related issue rather a Command/Code related issue, where the model is getting some default parameters, instead of the actual ones (as you stated). And no matter you start Training or Testing, both commands run the code, which means my environment has been set up properly.

Anyways, please keep this issue open, till I experiment further keeping in view your suggestions. I'll update you accordingly. Thanks!

[zzzyuqing]

The --train parameter is used for score distillation sampling to train the material representation network, while the --test parameter is solely for rendering some visual results from the trained material representation network.

Could you please specify what exact problems you encountered when installing other GitHub repositories in the Conda environment? One potential solution could be to try cloning the repositories and installing them.

[usamashami11]

I apologize for getting back to you a bit late, as I missed your message and was a bit busy.
Mainly while setting up the Conda environment, the GitHub repos I mentioned in my previous comment (which are actually being installed by running pip install -r requirements.txt) were creating issues. I even tried to install them separately, but there were some issues. I'll try to set up the Conda environment again today, and update the exact issue/error here.

Moreover, my main concern is regarding reproducing the exact inference results, as discussed earlier, and which I feel is due to some initialized value being passed (as stated by you earlier) or some other code-related issue (and NOT environment issue).

Can you please comment if the Inference for both the steps, i.e. PBR Material Generation and the output generation (3D textured model) is being done in 1 step via the command below, OR it requires running multiple commands?

python launch.py --config configs/dreammat.yaml --test --gradio --gpu 0  system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj system.geometry.shape_init_params=0.6 data.blender_generate=true

[zzzyuqing]

I apologize for getting back to you a bit late, as I missed your message and was a bit busy. Mainly while setting up the Conda environment, the GitHub repos I mentioned in my previous comment (which are actually being installed by running pip install -r requirements.txt) were creating issues. I even tried to install them separately, but there were some issues. I'll try to set up the Conda environment again today, and update the exact issue/error here.

Moreover, my main concern is regarding reproducing the exact inference results, as discussed earlier, and which I feel is due to some initialized value being passed (as stated by you earlier) or some other code-related issue (and NOT environment issue).

Can you please comment if the Inference for both the steps, i.e. PBR Material Generation and the output generation (3D textured model) is being done in 1 step via the command below, OR it requires running multiple commands?

python launch.py --config configs/dreammat.yaml --test --gradio --gpu 0  system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj system.geometry.shape_init_params=0.6 data.blender_generate=true

No，you should run this command for PBR material generation and output generation.

python launch.py --config configs/dreammat.yaml --train --gradio --gpu 0  system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj trainer.max_steps=3000 system.geometry.shape_init_params=0.6 data.blender_generate=true

[usamashami11]

No，you should run this command for PBR material generation and output generation.

python launch.py --config configs/dreammat.yaml --train --gradio --gpu 0  system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj trainer.max_steps=3000 system.geometry.shape_init_params=0.6 data.blender_generate=true

Okay, actually the flag --train was a bit confusing, and it was taking a bit longer to execute, which is why I thought it is training the model instead of inference. I'll try it and get back to you then. Thanks!

Just let me know in the meantime, that on average, how much time it should take to generate the PBR materials for 1 example, on a RTX 3090 (24 GB) GPU?

[zzzyuqing]

The training process, which pertains to the SDS procedure, indeed takes some time. Pre-rendering with Blender typically requires around 15 minutes, but this duration may vary depending on your CPU performance. When the pre-rendering is done, the generation process should be completed within 20 minutes on a RTX 3090 GPU.

[usamashami11]

Alright, thanks for the clarification! Right now, the command you shared, is working. I'll update you here, once outputs are generated. Moreover, I'll try to set up Conda environment as well, and let you know regarding exact issues.

[usamashami11]

Hi again, I still haven't been able to set up the Conda environment since I was busy, so please keep the issue open till then.
Moreover, I was able to generate the outputs for the provided examples, as shown in the paper.

Right now, I have 3 small questions, in which I need your assistance.

1. During inference, I'm using the command provided by you. Although the inference proceeds smoothly and generate results, there is a small warning/error shown in between, which highlights that the ControlNet weights file shall be a .safetensors file, but the provided weights file in Part 3 of section Preparation for inference are diffusion_pytorch_model.bin, hence it gives the File Not Found Error (image attached)

Do you think it will take less time, if the diffusion_pytorch_model.safetensors is provided, or inference time will stay the same?

2. Whenever I use a different prompt for the same object (to generate its variation) while keeping the data.blender_generate flag set to false, it throws an error, as the pre_render directory doesn't have all the required folders. Then I have to either copy the pre-render directory from the older output folder of the same object, to the new one OR infer it by setting the flag to true.
   What is the benefit of this flag disabling feature, if we can't even generate different variants of the same object by setting it as False?

3. I am generating textures for my custom objects. I really love your model, however, some of the results aren't up to the mark. I want to know what is the intuition behind setting the system.geometry.shape_init_params value to 0.6, 0.7 or 0.8, since it's different for each prompts in run_examples.sh? Moreover, please tell the same for system.guidance.cond_scale being changed from default value.
   My question is that what effect these params have on the output, and how do you set it for each example?

[zzzyuqing]

1. The storage format differs. When training your own ControlNet, you can use the .safetensors format for storage, but this does not affect usage.
2. The setting of data.blender_generate was designed for my convenience in tuning parameters, such as different iteration counts. If you want different text prompts all in the same folder without having to re-render, you can name the result folders after the mesh names, which eliminates the need for manual copying.
3. In most cases, you can directly use the default parameters: system.geometry.shape_init_params=1.0 and system.guidance.cond_scale=1.05. The parameter shape_init_params scales the size of the input mesh, while cond_scale is the weight in csd.

[usamashami11]

1. The storage format differs. When training your own ControlNet, you can use the .safetensors format for storage, but this does not affect usage.

Alright!

2. The setting of data.blender_generate was designed for my convenience in tuning parameters, such as different iteration counts. If you want different text prompts all in the same folder without having to re-render, you can name the result folders after the mesh names, which eliminates the need for manual copying.

Thanks for the tip! Will try to explore it further, and test it!

3. In most cases, you can directly use the default parameters: system.geometry.shape_init_params=1.0 and system.guidance.cond_scale=1.05. The parameter shape_init_params scales the size of the input mesh, while cond_scale is the weight in csd.

Thanks a lot for the insight, I was wondering today why the mesh size of output changed 😅, so it makes sense now! Will try to experiment with those as well!

[usamashami11]

Hi again, Yuqing,
I have queries from 2 perspectives, one is regarding outputs, and the other is regarding error regarding Conda env creation.

1. Regarding the outputs, I have been able to generate textures for some custom objects:

• Kindly tell if the generated texture isn't up to the mark, shall we experiment with cond_scale ?

• Regarding the system.geometry.shape_init_params, I tried all the values 0.7, 0.8 and even 1.0. It turned out that 0.7 or maybe 0.8 are the best options, as 1.0 resized the Mesh to quite large size and the PBR material didn't align with the Mesh.
  - Does the DreamMat generate its own mesh in addition to Texture, since the Output Mesh is a bit low-poly and Altering the Shape parameters, generated objects of varying sizes, as verified in blender?
  - Also, does it alter UV maps, because I tried wrapping the generating PBR materials over the original mesh, but it didn't fit well

• Is the code modular, in case someone wants to replace the underlying model. Since the Stable Diffusion, doesn't create that many accurate textures, for specific prompts, would it be possible to replace it with a better option, or give image at input (along with text) for conditioning the texture generation?

2. Shall I create a different issue for environment setup related issues (to avoid cluttering) or ask here? As stated in Model Inference Output Issue | Base Color not Showing #20 (comment), the main issues arise during installation of these repos. Moreover, there are some differences even in the order of installation of setup, is you see the Dockerfile and instructions for Conda env setup. I have tried to resolve some issues, but even installing each and every single repo causes issues of its own, which didn't occur much in case of Dockerfile (although I had to modify it as well)

I would love to hear your opinion on the above-mentioned questions, thanks!

[zzzyuqing]

About the quality of the generated textures not being "up to the mark," I'm not entirely sure what specific issues you're encountering. While I cannot guarantee that adjusting cond_scale will definitely improve the quality, since DreamMat isn't a perfect material generation scheme, I would be very interested in seeing some failure cases. These could provide valuable directions for future improvements.

The scaling adjustments we make are applied on a normalized mesh. The determination of scale primarily depends on the shape of the object to ensure it occupies a reasonable proportion of the overall scene. I appreciate your feedback on the parameters, and I agree that 0.7-0.8 seems suitable for most models.

We do not alter the topology of the input model; we only change the model's scale. For models without UVs, we use xatlas to generate UV mapping.

The underlying model can be replaced in the config file. For image-guided texture generation, please look forward to our work at SIGGRAPH Asia.

Regarding the environment setup issues, feel free to create a new issue for that. You might also find solutions in the closed issues if others have encountered similar problems.

Thank you for your questions, and please do not hesitate to contact me if you have more.

[usamashami11]

About the quality of the generated textures not being "up to the mark," I'm not entirely sure what specific issues you're encountering. While I cannot guarantee that adjusting cond_scale will definitely improve the quality, since DreamMat isn't a perfect material generation scheme, I would be very interested in seeing some failure cases. These could provide valuable directions for future improvements.

I agree that it isn't perfect, but it still is one of the best available model. I'll definitely share the failure cases. Mostly either the Generated Texture isn't up to the mark as the mesh is also a bit of low polygon (maybe due to normalization) and in some cases, it takes even more memory than 24 GB (for inference) e.g. cube/box shapes, giving Error: CUDA Out of Memory. Some other cases include swapping the front and back side textures, like on a clock, it generated face on the back side of it. Moreover, mostly it requires a lot of prompt engineering, but in some cases a simple text prompt suffices.

The scaling adjustments we make are applied on a normalized mesh. The determination of scale primarily depends on the shape of the object to ensure it occupies a reasonable proportion of the overall scene. I appreciate your feedback on the parameters, and I agree that 0.7-0.8 seems suitable for most models.
We do not alter the topology of the input model; we only change the model's scale. For models without UVs, we use xatlas to generate UV mapping.

Thanks for the clarification. And yes, I was considering to apply the generated PBR material to the original mesh, but it doesn't fit properly on that, which is why I asked.
I would like to know more about xatlas, are you using it in DreamMat for objects with no UV, or it's a separate study/experiment of yours?

The underlying model can be replaced in the config file. For image-guided texture generation, please look forward to our work at SIGGRAPH Asia.

I will check it for exploration and let you know for any queries. Upon some research, I believe it's an upcoming event, so is there any arXiV or pre-release of that work available?

=============================================

Regarding the environment setup issues, feel free to create a new issue for that. You might also find solutions in the closed issues if others have encountered similar problems.

About the Environment Issues, thankfully, after a tedious effort, I managed to set it up at last, successfully. There were some CUDA/GCC version incompatibility issues at my server/machine side, but I feel some further instructions may be added, in the provided guidelines for DreamMat too. I'll let you know, after documenting them, if I found any major change. And yes, I checked all the issues too, they did help to some extent, initially.

Thanks once again for the detailed answers to my queries!

[usamashami11]

Hi Yuqing, I'm awaiting your response to my previous comment.

In the meantime, I'll add my observations regarding conda environment setup.

The order for installation of files/packages/libraries which is mentioned in Dockerfile seems more appropriate and relevant, however, there are some things missing in that, which I'll try to add:

1. Create conda environment with required Python version (say 3.10.12)
   conda create -n DreamMat python=3.10.12

2. Ensure correct GCC, Cmake and CUDA Toolkit versions in your environment (since they create issues during installation of Tiny CUDA NN ) and then proceed for installation of packages.

3. Upgrade ninja package.
   pip install --upgrade pip setuptools ninja

4. Install PyTorch with Cuda-11.8.
   pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --index-url https://download.pytorch.org/whl/cu118

5. Install the complicated repositories first, one by one.

pip install git+https://github.com/KAIR-BAIR/nerfacc.git@v0.5.2
pip install git+https://github.com/NVlabs/tiny-cuda-nn.git#subdirectory=bindings/torch

6. Install requirements file. Make sure to comment out the already installed or separately installed repos/packages.
   pip install -r requirements.txt

7. If the torch version is changed (which it did in my case), repeat the step 4, again.

8. Install missing dependencies, set path for eigen3 and install raytracing package

apt-get update && apt-get install -y libeigen3-dev && apt-get install libxi6
export CPLUS_INCLUDE_PATH=/usr/include/eigen3
pip install git+https://github.com/ashawkey/raytracing.git

9. Install mediapipe package (since it was missing in requirements file)
   pip install mediapipe

10. Install Blender and set its path. The path command is changed a bit

export PATH=/path_to_blender/blender-3.2.2-linux-x64:${PATH}

[zzzyuqing]

Our geometry- and light-aware ControlNet is designed to encourage the generated textures to stay consistent with the current geometry. Therefore, it indeed might struggle with very low-polygon models, such as cubes. Additionally, if the front and back of a model have similar geometries, the multi-face issue might occur, which is a common problem with SDS methods. Regarding the Error: CUDA Out of Memory, you could try reducing the model scale or decreasing the number of light samples to mitigate the problem.

In DreamMat, we utilize xatlas to obtain UV mappings. For the generated textures, we have incorporated some post-processing techniques such as UV padding to address issues with seams.

Our subsequent work has not yet been released. We've been quite busy recently, and we expect to release it in November. You can find some existing research on Arxiv, such as FlexiTex (https://arxiv.org/pdf/2409.12431) and EASI-Tex (https://arxiv.org/abs/2405.17393).

Lastly, thank you very much for your comments and observations regarding conda environment setup.

[usamashami11]

Thank you for sharing your insights, and the relevant papers. I really appreciate it.
And I'm happy to help.

In the meanwhile, please comment on this, if you have any idea regarding this.
I am facing this error, while trying to generate texture for a custom box.

[INFO] Loaded Stable Diffusion!
Sanity Checking: |                                                                                                                                                                      | 0/? [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py:424: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=15` in the `DataLoader` to improve performance.
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py:424: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=15` in the `DataLoader` to improve performance.
Epoch 0: |                                                                                                                                                            | 20/? [00:04<00:00,  4.44it/s, v_num=3]../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7024,0,0], thread: [23,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7024,0,0], thread: [24,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7024,0,0], thread: [25,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
...
...

The box is more like a carton, as shown below.

Although the same code works for other objects, I'm not sure why such a box gives errors, although neither the GPU memory is overflowing nor the model itself seems that much complex, and the error generates before even the 3000 iterations complete (of training epoch during inference). Any suggestion would be helpful.

P.s. I'll open another issue since I want some assistance regarding fine-tuning as well. Thanks again!

[zzzyuqing]

Thank you for sharing your insights, and the relevant papers. I really appreciate it. And I'm happy to help.

In the meanwhile, please comment on this, if you have any idea regarding this. I am facing this error, while trying to generate texture for a custom box.

[INFO] Loaded Stable Diffusion!
Sanity Checking: |                                                                                                                                                                      | 0/? [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py:424: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=15` in the `DataLoader` to improve performance.
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py:424: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=15` in the `DataLoader` to improve performance.
Epoch 0: |                                                                                                                                                            | 20/? [00:04<00:00,  4.44it/s, v_num=3]../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7024,0,0], thread: [23,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7024,0,0], thread: [24,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7024,0,0], thread: [25,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
...
...

The box is more like a carton, as shown below.

Although the same code works for other objects, I'm not sure why such a box gives errors, although neither the GPU memory is overflowing nor the model itself seems that much complex, and the error generates before even the 3000 iterations complete (of training epoch during inference). Any suggestion would be helpful.

P.s. I'll open another issue since I want some assistance regarding fine-tuning as well. Thanks again!

Could you send me the .obj file of this model? I would like to try running the generation code to see the results.

[usamashami11]

Hi, I couldn't get back to you earlier.
I am attaching the box.obj file (sample 3D Mesh) with which I was having issues.
sample_mesh.zip

Moreover, I have some questions from you, regarding the generated output of DreamMat.

1. The rendered image, shown at inference, looks a lot better, as compared to when I display the generated .obj file along with generated PBR materials wrapped over it, in Blender. Why is that? Are you guys doing any post-processing or adding any environment light, or simply visualizing it in Blender?

2. To improve the DreamMat texture, are there any post-processing techniques recommended, in addition to what you mentioned in Appendix of Paper? Also, does fine-tuning improve the quality of generated textures?

3. I tried the generating texture over a pen using a similar prompt, as provided in the sample training_data, but it didn't perform well. I gave a 3D Mesh of a Pen and the prompt a sharpie pen with a white writing on it but there was no text on it, so I wanted to ask if it's able to generate some sort of text on the texture? And if not, fine-tuning the model on similar examples will enable it or not ? Let me know if you need any visual results of the same.

P.s. You may answer these questions at the moment and test the box issue too. Moreover, I do have some queries regarding fine-tuning, so I'll hopefully ask them in a separate issue, soon. Thanks again for your assistance!

[usamashami11]

Hi, any update on the highlighted matter? Also, I would appreciate your response on the queries I mentioned, in the two issues.

[zzzyuqing]

Regarding the array out-of-bounds issue you encountered, this is mainly due to a division by zero when normalizing the sampled ray directions, resulting in NaN. You can resolve this problem by modifying our code. Specifically, change

DreamMat/threestudio_dreammat/threestudio/models/materials/dreammat_material.py

Line 443 in 92afaa6

directions = directions / directions.norm(p=2, dim=-1, keepdim=True) # Normalize the vectors

to:

directions = directions / (directions.norm(p=2, dim=-1, keepdim=True) + 1e-6)

This modification prevents the division by zero error while ensuring correct normalization of the vectors. I used the model you provided and ran it with the following command python launch.py --config configs/dreammat.yaml --train --gradio --gpu 0 system.prompt_processor.prompt="a wooden box" system.geometry.shape_init=mesh:load/shapes/objs/box.obj trainer.max_steps=4000 system.geometry.shape_init_params=0.6 data.blender_generate=false. The result is as follows:

The albedo map we have stored is in linear space, so it is necessary to select the correct color space when rendering in Blender, otherwise the result may appear too dark. Additionally, different rendering engines use different methods of correction for the final rendering results, which may also lead to some variations.

Currently, our method cannot generate clear and reasonable text on textures; early image generation models also had this issue. You might consider replacing the base stable diffusion model and training your own controlnet.

[usamashami11]

Regarding the array out-of-bounds issue you encountered, this is mainly due to a division by zero when normalizing the sampled ray directions, resulting in NaN. You can resolve this problem by modifying our code. Specifically, change

DreamMat/threestudio_dreammat/threestudio/models/materials/dreammat_material.py

Line 443 in 92afaa6

directions = directions / directions.norm(p=2, dim=-1, keepdim=True) # Normalize the vectors

to:

directions = directions / (directions.norm(p=2, dim=-1, keepdim=True) + 1e-6)

This modification prevents the division by zero error while ensuring correct normalization of the vectors....

Thank you for the tip! I'll check it for all such cases.

...I used the model you provided and ran it with the following command python launch.py --config configs/dreammat.yaml --train --gradio --gpu 0 system.prompt_processor.prompt="a wooden box" system.geometry.shape_init=mesh:load/shapes/objs/box.obj trainer.max_steps=4000 system.geometry.shape_init_params=0.6 data.blender_generate=false.

In this command, did you generate conditional maps by setting data.blender_generate=true, at least for the first time, for the same box, or nope ?

The albedo map we have stored is in linear space, so it is necessary to select the correct color space when rendering in Blender, otherwise the result may appear too dark. Additionally, different rendering engines use different methods of correction for the final rendering results, which may also lead to some variations.

Does the Linear setting apply for Roughness and Metallic too?

Currently, our method cannot generate clear and reasonable text on textures; early image generation models also had this issue. You might consider replacing the base stable diffusion model and training your own controlnet.

Alright. I was looking forward to it, but I have some doubts, as stated in the Issue #25, so, I request your valuable comments in that as well (whenever convenient for you). And thanks again. I really appreciate your support!

[zzzyuqing]

1. Initially set data.blender_generate=true.

2. Please note that both the roughness and metallic parameters are in linear space.

[usamashami11]

Got it, thanks!
However, I did check after changing the directions = directions / (directions.norm(p=2, dim=-1, keepdim=True) + 1e-6) but it still gave me tensor dimension error. But I will experiment extensively, by changing the added value, and let you know in case I'm unable to resolve the same error. Your comment did give me a hint what was causing the issue, since I was unable to find it earlier.

[usamashami11]

Just two small queries:

• If we train the ControlNet as it is, using the training code/steps which you provided, will it Fine-tune the existing ControlNet (provided by you) or Train the ControlNet from scratch, or both can be done depending upon weights path?
  And what are the minimum GPU requirements for this Training, since the inference only consumes 24GB usually. Your 8 V100 GPUs are mentioned but I'm not sure if they were of 16 GB each or 32 GB?

• How to generate the testing data, as shown in the testing directory of dataset_example, for the teapot example? I think you generated it using the same blender scripts for training data, but with 1 Light Map (environment), and 1 viewpoint angle, right?
  

Your assistance will be appreciated, Thanks!