Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

why it stop in first trian step when i train it in my dataset #7

Closed
FENGShuanglang opened this issue Sep 26, 2018 · 5 comments
Closed

Comments

@FENGShuanglang
Copy link

image

@HasnainRaz
Copy link
Owner

The training is happening, the output is printed every 50 steps, so it might take some time until you see the next step, you can modify the train function to remove this 50 step output condition as needed.

@FENGShuanglang
Copy link
Author

yes,you are right!,but i think it is so slow for me ,It takes about 20 seconds to process a picture,I wonder if it is normal?I have used GPU

@HasnainRaz
Copy link
Owner

HasnainRaz commented Sep 26, 2018

The tiramisu is a deep network, and it's memory consumption is quadratic because of how tensorflow computes gradients. Also, your images are high resolution. Combined all these facts, it will be slow. You can either try reducing the dense blocks, number of layers or your image size to make it faster.

If you have access to a better GPU (V100s), that should also help, but I realize not everyone does.

@FENGShuanglang
Copy link
Author

OK,thank you very much ,but I wonder Why does the loss not converge? I am doing multi-class segmentation.
image

@HasnainRaz
Copy link
Owner

60 iterations is too low, the Tiramisu is known to be difficult to optimize, see, try training it longer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants