Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📡 📉 Adding Tensorboard Tracker #416

Merged
merged 41 commits into from Jun 2, 2021

Conversation

sbonner0
Copy link
Contributor

@sbonner0 sbonner0 commented May 3, 2021

This PR closes #383

Adding tensorboard as a results tracker.

What you need to test:

  1. pip install tensorboard
  2. start the tensorboard process with the default logging path: tensorboard --logdir=.data/pykeen/logs/tensorboard/

Issues:

  • I am not happy with the current naming scheme for the experiment log dir.
  • The params are current saved as text within the tb log. There is a add_hparams function, but I don't think it is quite what is needed here.
  • It would be great to distinguish between train and eval parameters when logging. It might also be worth considering if all the metrics actually need to be logged to tensorboard.
  • Tensorboard has great support for visualizing embeddings using the add_embedding function - would be great if the final embeddings (or a subset of them more realistically) could be added.

Tasks:

  • add documentation and examples
  • update setup.cfg with tensorboard deps

@Rodrigo-A-Pereira
Copy link
Contributor

Pytorch's tensorboard has an add_scalars() function, wich allows to pass multiple scalars at once instead of having to do it on a for cycle with add_scalar(). Maybe it could be used here.

Documentation: https://pytorch.org/docs/stable/tensorboard.html

@sbonner0
Copy link
Contributor Author

sbonner0 commented May 3, 2021

Pytorch's tensorboard has an add_scalars() function, wich allows to pass multiple scalars at once instead of having to do it on a for cycle with add_scalar(). Maybe it could be used here.

Documentation: https://pytorch.org/docs/stable/tensorboard.html

Thanks! This is totally an option, I just didn't use it here as I was unsure of a sensible main tag. Ideally it would be great to have train/eval separation perhaps using the main tag here.

@sbonner0 sbonner0 marked this pull request as draft May 3, 2021 16:43
sbonner0 and others added 3 commits May 4, 2021 19:13
Using joinpath instead of /

Co-authored-by: Charles Tapley Hoyt <cthoyt@gmail.com>
This commit does the following:
1. Imports TYPE_CHECKING from typing. This is always false during Python runtime, but set to true when a type checker like MyPy is being used to statically analyze the code
2. Uses a conditional to wrap importing the tensorboard stuff at the top. this means that during actual runtime, this is never imported
3. Make a type annotation for instance variables in the class. Specifically, this is for the SummaryWriter variable. Note that the thing that's stored in this variable is the class itself, so to refer to the class and not it being an instance of the class, we wrap the annotation with Type[]. Further, note that the annotation is done as a string. This makes it a lazy type annotation, which is good because it won't explode because it points to a package that will not be imported.

Note: this is independent from my other comment, where I said this isn't really a good name for an instance variable.
better name for the summary writer

Co-authored-by: Charles Tapley Hoyt <cthoyt@gmail.com>
@cthoyt
Copy link
Member

cthoyt commented May 11, 2021

@PyKEEN-bot test please

@sbonner0
Copy link
Contributor Author

I am having issues with changes the setup.cfg passing the precommit checks. I had wanted to add tensorboard to [options.extras_require] as follows:

 tensorboard =
    tensorboard

However this causes the check to fail complaining of trailing white space.

@cthoyt cthoyt added the 🐺 Tracker Related to results trackers, like MLflow label May 16, 2021
@cthoyt cthoyt added this to the PyKEEN v1.5.0 milestone May 16, 2021
@sbonner0
Copy link
Contributor Author

I think this might be ok for an initial release of the tensorboard tracker.

There is a bunch of extra stuff I think that could be done to improve and expand upon the experience however these would not affect core functionality and could be done at a later date. Stuff I would like to add includes embedding visualization and better separation between train/test metrics.

@cthoyt
Copy link
Member

cthoyt commented May 26, 2021

@sbonner0 okay, then we're going to send the @PyKEEN-bot to test! It will give some feedback that you'll need to take care of (code style, doc style, etc.). If some of the feedback doesn't make sense, just tag me and I will explain what needs to be done

@cthoyt cthoyt marked this pull request as ready for review May 27, 2021 13:17
@cthoyt
Copy link
Member

cthoyt commented May 27, 2021

@sbonner0 there's also the tradition of prefixing all pull requests' names in PyKEEN with two emojis. Please feel free to pick the ones you'd like :)

@sbonner0
Copy link
Contributor Author

@sbonner0 there's also the tradition of prefixing all pull requests' names in PyKEEN with two emojis. Please feel free to pick the ones you'd like :)

Such responsibility! Is there some rule for choosing appropriate emojis?

@cthoyt
Copy link
Member

cthoyt commented May 27, 2021

@sbonner0 there's also the tradition of prefixing all pull requests' names in PyKEEN with two emojis. Please feel free to pick the ones you'd like :)

Such responsibility! Is there some rule for choosing appropriate emojis?

Nope, just do what feels right

@sbonner0 sbonner0 changed the title Adding Tensorboard Tracker 📡 📉 Adding Tensorboard Tracker May 27, 2021
@cthoyt
Copy link
Member

cthoyt commented Jun 1, 2021

@sbonner0 please let me know about the tags (see #416 (comment)) and we can finish this up and merge it

@sbonner0
Copy link
Contributor Author

sbonner0 commented Jun 2, 2021

@sbonner0 please let me know about the tags (see #416 (comment)) and we can finish this up and merge it

Hi @cthoyt - tags have now been removed - unused hangover from having used mlflow as the template for the TB tracker.

@cthoyt
Copy link
Member

cthoyt commented Jun 2, 2021

@PyKEEN-bot test please!

@cthoyt
Copy link
Member

cthoyt commented Jun 2, 2021

@PyKEEN-bot test again

@cthoyt cthoyt merged commit 811fa2b into pykeen:master Jun 2, 2021
@cthoyt
Copy link
Member

cthoyt commented Jun 2, 2021

@sbonner0 thank you for your contribution! I know this took a long time but it's much appreciated

@sbonner0
Copy link
Contributor Author

sbonner0 commented Jun 2, 2021

@cthoyt Sorry it took me ages to finish off! Super happy to finally be a contributor though. I think there are some ways we could improve the TB tracker so I'll open appropriate issues and get working on a PR

@cthoyt
Copy link
Member

cthoyt commented Jun 3, 2021

We're gearing up to make a release soon, so feel free to make new issues or a new PR as you battle test this! Thanks again :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💎 New Component 🐺 Tracker Related to results trackers, like MLflow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Tensorboard Tracker
5 participants