Replies: 5 comments 5 replies
-
|
I have also tried doing this this one actually seems to work the best so far? I know that it only uses on GPU. The main issue appears to be that since it is running immediately after the Futzing with it more: and then I still get the same error: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @turian! There's several ways to do this. Some more flexible, some more automatically managed. Since you mentioned here that this is your preference, I'll follow-up on this example. import os
import torch
from pytorch_lightning import Trainer
from pytorch_lightning.demos.boring_classes import BoringModel
class MyModel(BoringModel):
def on_validation_epoch_end(self):
if self.trainer.sanity_checking: # optional skip
return
print("Start predicting!")
for i, batch in enumerate(self.predict_dataloader()):
batch = self.transfer_batch_to_device(batch, self.device, dataloader_idx=0)
out = self.predict_step(batch, i)
print(i, out)
def configure_optimizers(self):
return torch.optim.SGD(self.layer.parameters(), lr=0.1)
def run():
model = MyModel()
trainer = Trainer(
default_root_dir=os.getcwd(),
accelerator="auto",
limit_train_batches=1,
limit_val_batches=1,
max_epochs=1,
enable_model_summary=False,
)
trainer.fit(model)
if __name__ == "__main__":
run()You should be able to expand this snippet to support a datamodule or doing this through a callback without issues. If you prefer to continue using a different pattern (loops customization, ...), tell me and we can discuss it. |
Beta Was this translation helpful? Give feedback.
-
|
@carmocca Thank you for the response. Is this the more flexible or self-managed approach? Just curious what's the more flexible approach down the road. So I have been unable to get your approach working when I replace BoringModel with something that uses a datamodule :( Here are two things that don't work: With your approach of extending MyModel, but having a BoringModel that doesn't have dataloaders defined and instead receives a datamodule, how do you find the datamodule or predict_dataloader when you're in fit? With a callback, I am not really sure if I should be passing the datamodule into |
Beta Was this translation helpful? Give feedback.
-
|
This is the fully self-managed approach. Here's the incantation you are looking for import os
from pytorch_lightning import Trainer
from pytorch_lightning.callbacks import Callback
from pytorch_lightning.demos.boring_classes import BoringDataModule, BoringModel
class PredictAfterValidationCallback(Callback):
def setup(self, trainer, pl_module, stage):
if stage in ("fit", "validate"):
# setup the predict data even for fit/validate, as we will call it during `on_validation_epoch_end`
trainer.datamodule.setup("predict")
def on_validation_epoch_end(self, trainer, pl_module):
if trainer.sanity_checking: # optional skip
return
print("Start predicting!")
# this will fetch the dataloader from the LM or LDM
predict_dataloader = trainer._data_connector._predict_dataloader_source.dataloader()
for i, batch in enumerate(predict_dataloader):
# this will move the batch to device using the LM or LDM
batch = pl_module._apply_batch_transfer_handler(batch)
# manually run the predict step
out = pl_module.predict_step(batch, i)
print(i, out)
class MyModel(BoringModel):
...
def run():
model = MyModel()
datamodule = BoringDataModule()
trainer = Trainer(
default_root_dir=os.getcwd(),
accelerator="auto",
limit_train_batches=1,
limit_val_batches=1,
max_epochs=1,
enable_model_summary=False,
callbacks=PredictAfterValidationCallback(),
)
trainer.fit(model, datamodule=datamodule)
if __name__ == "__main__":
run()It reaches deep into the framework to reliably get the predict dataloader and to transfer the batch to device. This is only so it chooses between the different potential sources of data, but you could simplify it with the assumption that you have a single dataloader and that only one of the LM or LDM might override predict_dataloader = trainer.datamodule.predict_dataloader()
for i, batch in enumerate(predict_dataloader):
batch = pl_module.transfer_batch_to_device(batch, trainer.strategy.root_device, 0)We'll be coming up with more approaches to simplify this and give you more control down the road. |
Beta Was this translation helpful? Give feedback.
-
|
This is great, thank you! Closing. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @carmocca 👋 I have been doing something similar for a while and updating lightning broke a couple of things on my side 😅 (had been a while since I updated to be honest). On your callback the |
Beta Was this translation helpful? Give feedback.
-
|
Instead of using a Callback, and trying to use loops as per #10393: I add this code suggested by @tchaton and @carmocca after creating my trainer: However, So the above isn't calling with the appropriate dataset :\ |
Beta Was this translation helpful? Give feedback.
-
|
I verified before that This actually causes a crash after Epoch 0 finishes right as Epoch 1 starts with the following error: (I still get no predictions being performed.) |
Beta Was this translation helpful? Give feedback.
-
|
Because the predict_dataloaders are None, I try to fix the loop as follows: In this case, the code recommended by @carmocca and @tchaton doesn't work either. I now have a predict_dataloader, but the predict_loop doesn't read anything and ends instantly. (I have a print statement in each prediction_step() which is not triggering.) |
Beta Was this translation helpful? Give feedback.
-
|
@turian did you solve this, I made it partway there on a logging issue, but got stopped a similar spot as you |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I've been trying to figure this out for nearly two days straight and it's very frustrating :(
Periodically, during training, I want to run the model on a bunch of data that has no labels, and log that. Let's just say that's my predict data and I'll do it at the end of validation, to keep it simple.
I'm trying several callbacks and none works:
and I get:
Or if I try to write the predict loop myself:
then I get the following error:
which is even more confusing and gnarly.
All I want is to run my model on a particular unlabelled dataset and log the predictions periodically. But I really can't figure this out for the life of me. Help!
Related:
Beta Was this translation helpful? Give feedback.
All reactions