Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"float division by zero" exception during notebook-example on anaconda/windows #59

Closed
Daiver opened this issue Jan 16, 2019 · 4 comments
Assignees
Labels
bug Something isn't working

Comments

@Daiver
Copy link

Daiver commented Jan 16, 2019

Hi guys, thank you for great project.

I tried to run notebook-example over my Anaconda environment and faced with "float division by zero exception"
This is a code which causes exception: https://gist.github.com/Daiver/b4f9115a9e33a1ca233d0defbabee6d9 (basically notebook-example copy-pasted inside one .py file)

This is stacktrace:

C:\Users\Daiver\Anaconda3\python.exe C:/Users/Daiver/PycharmProjects/untitled/main.py
Python version 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
Catalyst version: 0.6
Files already downloaded and verified
Files already downloaded and verified
0 * Epoch (train):   0% 1/1563 [00:00<15:06,  1.72it/s, base/batch_time=0.01562, base/data_time=0.01562, base/sample_per_second=2048.43759, loss=2.32288, lr=0.00100, momentum=0.90000, precision01=3.12500, precision03=18.75000, precision05=56.25000]Traceback (most recent call last):
  File "C:/Users/Daiver/PycharmProjects/untitled/main.py", line 113, in <module>
    main()
  File "C:/Users/Daiver/PycharmProjects/untitled/main.py", line 109, in main
    epochs=n_epochs, verbose=True)
  File "C:\Users\Daiver\PycharmProjects\untitled\catalyst\dl\runner.py", line 210, in train
    verbose=verbose
  File "C:\Users\Daiver\PycharmProjects\untitled\catalyst\dl\runner.py", line 159, in run
    self.run_event(callbacks=callbacks, event="on_batch_end")
  File "C:\Users\Daiver\PycharmProjects\untitled\catalyst\dl\runner.py", line 92, in run_event
    getattr(self.state, f"{event}_pre")(state=self.state)
  File "C:\Users\Daiver\PycharmProjects\untitled\catalyst\dl\state.py", line 203, in on_batch_end_pre
    state.batch_size / elapsed_time
ZeroDivisionError: float division by zero


Process finished with exit code 1

It can be fixed by adding zero check on elapsed_time but i have no idea, why elapsed_time is zero

My python/catalyst versions

Python version 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
Catalyst version: 0.6

Catalyst was installed by clonning current repo (master branch, last commit 892d5e5 "Merge pull request #56 from dbrainio/master")

@Daiver Daiver changed the title "float division by zero" exception during notebook-example on windows "float division by zero" exception during notebook-example on anaconda/windows Jan 16, 2019
@TezRomacH TezRomacH added the bug Something isn't working label Jan 16, 2019
@TezRomacH
Copy link
Contributor

Hi, Davier. Thanks for reporting!

Currently, Catalyst is in alpha and we do a lot of breaking changes right now. Sorry for that!
Within a couple of weeks, we plan to release the stable version of Catalyst with a great number of new features!

Right now, you may add a zero-checker while we fixing it 😄

@Daiver
Copy link
Author

Daiver commented Jan 16, 2019

Hi @TezRomacH! It's totally ok.
Can't wait for documentation, by the way :)

@Scitator
Copy link
Member

Hi,

Good catch! Looks like we need some additional Windows testing, cause all code works like a charm on Linux and MacOS. Just to be sure, I have tested your code with:

Python 3.6.4 :: Anaconda custom (64-bit)
torch.__version__==1.0.0
catalyst == 892d5e5

It's a bit weird when batch elapsed_time is == 0, but as a hotfix you can try to:

  • turn off GPU
  • increase batch size
  • use large dataset, like finetune example

Somehow you process your batch so quick, that even python cannot make it out :)

@Daiver
Copy link
Author

Daiver commented Jan 17, 2019

Just checked example on ubuntu machine (with much better hardware, lol). Works perfectly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
No open projects
Catalyst.DL 19.03
  
Awaiting triage
Development

No branches or pull requests

3 participants