Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to get the visualization result? #69

Closed
MrRace opened this issue Feb 13, 2019 · 22 comments

Comments

Projects
None yet
9 participants
@MrRace
Copy link

commented Feb 13, 2019

After running ludwig visualize --visualization learning_curves -ts results/experiment_run_4/training_statistics.json, it returns empty. Where to get the visualization results?Is it saved as a image in somewhere?

@gridcellcoder

This comment has been minimized.

Copy link

commented Feb 13, 2019

According to the docs it should be -tes not -ts and in the README it is --training_stats when it should be --training_statistics results/training_stats.json

try with -tes someone also needs to correct the README

@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Feb 13, 2019

Actually, I may have inverted tes and trs. Will change that. Anyway, i guess the point is a different one: when you run the visualize commands, a window is opened and you can save the image from there, the motivation being that in some cases, when you have a lot of classes for instance, graphs need to be resized a bit to be readable before saving them. If no window is opening, then please report on your configuration and please make a reproducible example so that I can look into it.

@davideprimiceri

This comment has been minimized.

Copy link

commented Feb 13, 2019

Hi, I have the same problem working on Google Colab, in fact after running !ludwig visualize --visualization learning_curves --training_statistics /content/results/experiment_run_0/training_statistics.json I do not get the visualization, but the cell is empty. Is there a way to get the visualization not in a new window, but save it directly in a file?

@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Feb 13, 2019

@davideprimiceri can you please share a colab that reproduces the issue?

@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Feb 13, 2019

Oh so you are running the visualize command from the colab, you are not calling the functions themselves. I'm not sure how that would work, as if you call matplotlib functions by running the proces, they are unlikely to be piped into the cell output. Do you have experience in doing something like this (calling scripts that open mtplotlib windows and having them rendered in colab cells)?

This is making me think that probably I should add an option to visualize that allows for saving the images to disk instead of opening windows. That would solve your problem as you would be able to open the saved images in a cell, and also @MrRace 's problem (if if I don't know yet what his problem was in detail).

@dshlai

This comment has been minimized.

Copy link

commented Feb 13, 2019

Hi, since visualize is using matplotlib internally, wouldn’t using inline plot in Colab or Juypter also work?

@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Feb 14, 2019

@dshlai I don't have enough experience with either to be sure about it. Will look into it and try to figure it out.

@dsmmr

This comment has been minimized.

Copy link

commented Feb 14, 2019

I also find the visualize command a bit bulky and non-intuitive. A more "intelligent" variant would be nice, where the visualize command is just called with the results folder of the experiment and returned is a choice which plots can be made after possible results files were found by ludwig. Would this be feasible?

For example in this case I do not know if I used it wrong or another file is expected:

(test) D:\ludwig\titanic\results\experiment_run_1>ludwig visualize --visualization confusion_matrix --predictions prediction_statistics.json
No prediction_statistics provided

(test) D:\ludwig\titanic\results\experiment_run_1>dir
14.02.2019  14:47    <DIR>          .
14.02.2019  14:47    <DIR>          ..
14.02.2019  14:35             5.768 description.json
14.02.2019  14:47    <DIR>          model
14.02.2019  14:47            20.401 prediction_statistics.json
14.02.2019  14:47             1.373 Survived_predictions.csv
14.02.2019  14:47               308 Survived_predictions.npy
14.02.2019  14:47             2.309 Survived_probabilities.csv
14.02.2019  14:47               848 Survived_probabilities.npy
14.02.2019  14:47            88.804 training_statistics.json
@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Feb 14, 2019

@dsmmr you should provide --prediction_statistics not --predictions.

@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Feb 14, 2019

@MrRace can you please provide details about your environment and a reproducible example, otherwise it's really difficult for us to help you.

@MrRace

This comment has been minimized.

Copy link
Author

commented Feb 15, 2019

@MrRace can you please provide details about your environment and a reproducible example, otherwise it's really difficult for us to help you.

What environment? I just install the packages in requirements.txt. By the way I run the command in linux sysytem. Run ludwig visualize --visualization....... these is no any window opened. Therefore what commands should I use to get the environment you mentioned?

@MrRace

This comment has been minimized.

Copy link
Author

commented Feb 15, 2019

According to the docs it should be -tes not -ts and in the README it is --training_stats when it should be --training_statistics results/training_stats.json

try with -tes someone also needs to correct the README

I use ludwig visualize --help to check the arguments, and it do not have a argument like -tes . It exist difference between Document and Command.

@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Feb 15, 2019

@MrRace, I meant what operative system, what version of python and so on. I'm trying to find a way to replicate your issue to fix it. Can you attach your training_statistics.json or send it to me privately?

@MrRace

This comment has been minimized.

Copy link
Author

commented Feb 15, 2019

@w4nderlust ,Here is my system message:Linux version 4.15.0-45-generic (buildd@lcy01-amd64-027) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)),and I use python 3.5. The attachment is the file I use to do the visualization.

training_statistics.zip

@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Feb 20, 2019

Looked at your training statistics. There seem to be nothing wrong with it. On the same configuration i was able to obtain a visualization of the learning curves. I don't actually know what is going wrong here. You don't get any error, the program closes correctly, but you just don't see the window, right?
Maybe one thing you can try is to update matplotlib in your virtualenv to the latest version and see if that solves the issue, but it's kinda of a hail mary, without an error I have a hard time replicating the problem and figuring out a solution. If also this doesn't work, I would suggest you just wait for the addition of an --output_directory parameter to visualize which is in the list of future enhancement. Or if you actually want, you can try to contribute, it should be pretty straightforward.

@dehdari

This comment has been minimized.

Copy link

commented Mar 4, 2019

Similar problem for me on MacOS, where nothing was appearing. It turns out that my Matplotlib backend was displaying the plots in the background, behind every other window on my desktop.

Still, the ability to save images to disk would be nice (and more plots automatically in Tensorboard).

@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Mar 5, 2019

@dehdari from the window that opens you can save them (there's a button at the bottom). The rationale to have them in the window is that before saving you can resize (which is useful if you have many classes or many models to compare, otherwise labels would overlap and the plot not be readable). Anyway, will add the functionality ho save the plots directly.
Which other plots would you be interested to have in the TensorBoard? Consider it is only used to keep track of measures at the batch level during training.

@gobzer

This comment has been minimized.

Copy link

commented Mar 12, 2019

I hacked the ludwig/utils/visualization_utils.py to save the graphs.
Before each
plt.show()
I put a
plt.savefig(title+'.png')
It's stupid, but works
saves the graphs in current directory

@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Mar 13, 2019

@gobzer it's not stupid, it's the way it should be done :)
I would be really glad if you could add a save_figures, save_path and figure_extention parameters to the visualize.py module command line interface and pass those parameters through to compose the path savefig is going to save the file to, and finally make a PR. it would be much appreciated and it would be was I was planning to doanyway.

@carlogrisetti

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

In the meantime, pull request #332 should fix the naming mismatch in the parameters, both in the arguments passed to ludwig visualize, and in the web user guide

@w4nderlust

This comment has been minimized.

Copy link
Collaborator

commented Jun 29, 2019

#403 addresses this.

@w4nderlust w4nderlust closed this Jun 29, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.