extremely slow training with multiple GPUs

[pamparana34]

I am training a model with lightning where I am attempting to use all the GPUs on my system (4 in total).

My trainer is run as:

model = MyModel(hparams)
if torch.cuda.is_available():
    trainer = Trainer(gpus=-1)
else:
    trainer = Trainer()
trainer.fit(model)

My model is defined as follows:

class SiameseNet(pl.LightningModule):
    """
    Implement a siamese network as a feature extractor withh Lightning module
    """
    def __init__(self,
                 hparams):
        """
        Build the network
        """
        super(SiameseNet, self).__init__()
        self.net = self._build_net()
        self.hparams = hparams
        self.train_data_path = hparams.get('train_data_path', None)
        self.test_data_path = hparams.get('test_data_path', None)
        self.val_data_path = hparams.get('val_data_path', None)
        self.train_dataset = None
        self.val_dataset = None
        self.test_dataset = None

        self.lossfn = TripletLoss(margin=1.0)

    def forward_once(self, x):
        output = self.net(x)
        output = torch.squeeze(output)
        return output

    def forward(self, input1, input2, input3=None):
        output1 = self.forward_once(input1)
        output2 = self.forward_once(input2)

        if input3 is not None:
            output3 = self.forward_once(input3)
            return output1, output2, output3

        return output1, output2

    @staticmethod
    def _build_net():
        net = nn.Sequential(
            nn.Conv2d(3, 32,kernel_size=3,stride=2),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(32),

            nn.Conv2d(32, 64, kernel_size=3, stride=2),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(64),

            nn.Conv2d(64, 128, kernel_size=3, stride=2),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(128),

            nn.Conv2d(128, 256, kernel_size=1, stride=2),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(256),

            nn.Conv2d(256, 256, kernel_size=1, stride=2),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(256),

            nn.Conv2d(256, 512, kernel_size=3, stride=2),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(512),

            nn.Conv2d(512, 1024, kernel_size=1, stride=1),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(1024))

        return net

    def prepare_data(self):
        transform = torchvision.transforms.Compose([
            torchvision.transforms.Resize((128, 128)),
            torchvision.transforms.ColorJitter(hue=.05, saturation=.05),
            torchvision.transforms.RandomHorizontalFlip(),
            torchvision.transforms.RandomRotation(20, resample=PIL.Image.BILINEAR),
            torchvision.transforms.ToTensor()
        ])

        if self.train_data_path:
            train_folder_dataset = dset.ImageFolder(root=self.train_data_path)
            self.train_dataset = SiameseTriplet(image_folder_dataset=train_folder_dataset,
                                                transform=transform)
        if self.val_data_path:
            val_folder_dataset = dset.ImageFolder(root=self.val_data_path)
            self.val_dataset = SiameseTriplet(image_folder_dataset=val_folder_dataset)

        if self.test_data_path:
            test_folder_dataset = dset.ImageFolder(root=self.test_data_path)
            self.test_dataset = SiameseTriplet(image_folder_dataset=test_folder_dataset)

    def training_step(self, batch, batch_idx):
        anchor, positive, negative = batch
        anchor_out, positive_out, negative_out = self.forward(anchor, positive, negative)
        loss_val = self.lossfn(anchor_out, positive_out, negative_out)
        return {'loss': loss_val}

    def configure_optimizers(self):
        optimizer = optim.Adam(self.parameters(), lr=self.hparams.get('learning_rate', 0.001))
        scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10)
        return [optimizer], [scheduler]

    @pl.data_loader
    def train_dataloader(self):
        if self.train_dataset:
            return DataLoader(self.train_dataset,
                              self.hparams.get('batch_size', 64),
                              num_workers=12)
        return None

When I try and run it, it seems the beginning of the epoch hangs for like 10 minutes to get data into the model and after that the progress is very sluggish.

I also get these messages in the beginning. Not sure if it is of concern

GPU available: True, used: True
No environment variable for node rank defined. Set as 0.
CUDA_VISIBLE_DEVICES: [0,1,2,3]
MASTER_ADDR environment variable is not defined. Set as localhost
initializing proc_rank 0 world 4
MASTER_ADDR environment variable is not defined. Set as localhost
initializing proc_rank 1 world 4
MASTER_ADDR environment variable is not defined. Set as localhost
initializing proc_rank 2 world 4
MASTER_ADDR environment variable is not defined. Set as localhost
initializing proc_rank 3 world 4

It basically hangs with this:

Epoch 1:   0%|                                                                                          | 0/172 [00:00<?, ?it/s]

During this time, looking at GPU utilisation it seems:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:05:00.0 Off |                  N/A |
| 48%   79C    P2    90W / 250W |   4527MiB / 11176MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:06:00.0 Off |                  N/A |
| 45%   76C    P2    85W / 250W |   1636MiB / 11178MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  Off  | 00000000:09:00.0 Off |                  N/A |
| 45%   76C    P2    79W / 250W |   1626MiB / 11178MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 108...  Off  | 00000000:0A:00.0 Off |                  N/A |
| 32%   65C    P2    79W / 250W |   2689MiB / 11178MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1584      C   /home/pd/.conda/envs/alchera37/bin/python    801MiB |
|    0     10714      C   /home/pd/.conda/envs/alchera37/bin/python    543MiB |
|    0     28957      C   /home/pd/.conda/envs/alchera37/bin/python   1047MiB |
|    0     30880      C   /home/pd/.conda/envs/alchera37/bin/python   1063MiB |
|    0     32266      C   /home/pd/.conda/envs/alchera37/bin/python   1063MiB |
|    1     10733      C   /home/pd/.conda/envs/alchera37/bin/python    543MiB |
|    1     28972      C   /home/pd/.conda/envs/alchera37/bin/python   1063MiB |
|    2     10789      C   /home/pd/.conda/envs/alchera37/bin/python    543MiB |
|    2     32297      C   /home/pd/.conda/envs/alchera37/bin/python   1063MiB |
|    3     10807      C   /home/pd/.conda/envs/alchera37/bin/python    543MiB |
|    3     29006      C   /home/pd/.conda/envs/alchera37/bin/python   1063MiB |
|    3     30967      C   /home/pd/.conda/envs/alchera37/bin/python   1063MiB |
+-----------------------------------------------------------------------------+

So, it seems that getting the data into the GPU is quite slow even though everything looks maxed out.

And when it does eventually start the epoch after ~30 minutes, it seems to give similar performance as my CPU on MacBook Pro. I am really not sure if I am doing somethingvery wrong here in how I am using PL.

[williamFalcon]

update to master. will reopen if not fixed

[pamparana34]

Unfortunately, this seems to be even worst with the current master.

First it gives the error:

Traceback (most recent call last):
  File "/home/pd/Altrack/altrack/apps/train_siamnet.py", line 3, in <module>
    from ..models.siamnet import SiameseNet
ImportError: attempted relative import with no known parent package

which is incorrect. This did not happen with the previous version.

And then it gets stuck after this line (at least no console output)

initializing ddp: LOCAL_RANK: 0/3 WORLD_SIZE:4

[Borda]

@pamparana34 mind chek the recent 0.9 from master?

[awaelchli]

And then it gets stuck after this line (at least no console output)

This looks very familiar, and I am sure I fixed this problem in #2997, please try again with the latest version. Regarding the relative import error, you probably just launched the script in the wrong directory, but anyway I recommend to use absolute imports. Please let me know if the upgrade fixes your problem, thanks.