Cannot control wandb metadata when running examples/seq2seq/finetune.py

[sshleifer]

There is no easy way to control the wandb project_name kwarg.
How can we facilitate this without massive number of command line args?

[borisdayma]

You should be able to do it with environment variables. See these docs
Let me know if that solves the issue.

The reason of the current implementation (and auto init of wandb) was to be able to instrument all example scripts without having to add custom wandb code.

There is documentation on W&B related to transformers that probably need to be updated to add this detail. We should probably also find a place to add documentation related to W&B integration in transformers repo. Let me know if I can be of any help.

[sshleifer]

You could make examples/wandb.md with information and link to it from examples/README.md? Or just link to this in examples/README.md

For the lightning WandbLogger integration, does $WANDB_PROJECT take precedence over passing WandbLogger(project='project_name')?

[borisdayma]

Thanks, that's a great idea!

For the lightning integration, any parameter you pass explicitly in WandbLogger should take precedence.

[borisdayma]

@sshleifer it seems that the pytorch-lightning integration is commonly used so I also added a reference to it in my PR.
Just curious, is it because you cannot do distributed computing with Trainer on multiple machines?

Would it make sense to add an easier integration on lightning_base with a similar logic to Trainer and TFTrainer, ie by using WandbLogger on generic_train whenever wandb is installed and logged in (and ignore it otherwise)?

[sshleifer]

As long as we can avoid
(1) generating a lot of logs when I run the unittests on my local machine
(2) enabling gradient watching by default

That sounds like a good change to me. @patil-suraj what do you think? do you use wandb?

[patil-suraj]

Yes, adding wandb integration on lightning_base makes sense to me given wandb is enabled by default in Trainer and TFTraner , this will enable wandb logging for run_pl_ner and run_pl_glue

With Trainer I can disable logging when I'm testing using WANDB_DISABLED env variable and gradient watching by setting
WANDB_WATCH .

env variables should avoid excessive command line args

[borisdayma]

Thanks for the feedback @sshleifer and @patil-suraj
I can try to propose something similar for lightning_base. I'll wait for PR #5607 to be closed (feel free to comment if you think it's missing details) and I'll update the README accordingly when adding this functionality.

[sshleifer]

Awesome. I had no idea that PR existed. Tag me next time and sorry for being so slow.