-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VADA or DIRT-T failed #1
Comments
If the folder is empty, then it sounds like either VADA failed to run or failed to save. Check the tensorboard log to see if the model ran at all. To debug VADA, run
this will print a progress bar. |
It sounds like it's running just fine? It takes a few hours for VADA to run to completion on a 1080Ti. |
How often is the code generating a checkpoint? I'm using a K80 (which is weaker than the 1080Ti), and indeed the program is still running seemingly fine. Yet after running the whole night, I'd expected that some results would have been stored to disk. Was that a incorrect assumption perhaps? |
Those configs are available in train.py. Based on your screenshot, it seems like you only went through 2054 mini-batch updates after a whole night; that sounds pretty slow. Training completes after 80k updates. |
It appears that the GPU card was not utilized. I updated drivers to CUDA 9. |
I'm a bit puzzled by what you described. Have you used tensorboard before? |
Hi again, |
Hi,
I got a vanilla copy of the master branch and downloaded the data using the 2 scripts in the repo.
Then I ran VADA, but got an error while executing DIRT-T following that.
The logs says the following. What do they tell me?
The folder 'checkpoints' only contain one folder 'model=dirtt_src=mnist_trg=svhn_nn=small_trim=5_dw=1e-02_bw=0e+00_sw=1e+00_tw=1e-02_dirt=00000_run=0000' and its empty. My guess is that VADA failed to complete. Am I right?
The text was updated successfully, but these errors were encountered: