[09/19 03:56:27] libai INFO: Rank of current process: 0. World size: 8 [09/19 03:56:27] libai INFO: Command line arguments: Namespace(config_file='configs/swinv2_imagenet.py', eval_only=False, fast_dev_run=False, opts=[], resume=False) [09/19 03:56:37] libai INFO: Rank of current process: 0. World size: 8 [09/19 03:56:37] libai INFO: Command line arguments: Namespace(config_file='configs/swinv2_imagenet.py', eval_only=False, fast_dev_run=False, opts=[], resume=False) [09/19 03:56:37] libai INFO: Contents of args.config_file=configs/swinv2_imagenet.py: from libai.config import LazyCall from .common.models.swinv2.swinv2_tiny_patch4_window8_256 import model from .common.models.graph import graph from .common.train import train from .common.optim import optim from .common.data.imagenet import dataloader from flowvision import transforms from flowvision.data import Mixup from flowvision.loss.cross_entropy import SoftTargetCrossEntropy from flowvision.transforms import InterpolationMode from flowvision.transforms.functional import str_to_interp_mode from flowvision.data.constants import (  IMAGENET_DEFAULT_MEAN,  IMAGENET_DEFAULT_STD, ) from flowvision.data.auto_augment import rand_augment_transform from flowvision.data.random_erasing import RandomErasing # Refine data path to imagenet dataloader.train.dataset[0].root = "/data/ImageNet/extract/" dataloader.test[0].dataset.root = "/data/ImageNet/extract/" # Add Mixup Func dataloader.train.mixup_func = LazyCall(Mixup)(  mixup_alpha=0.8,  cutmix_alpha=1.0,  prob=1.0,  switch_prob=0.5,  mode="batch",  num_classes=1000, ) dataloader.train.dataset[0].transform = LazyCall(transforms.Compose)(  transforms=[  LazyCall(transforms.RandomResizedCrop)(  size=256,  scale=(0.08, 1.0),  ratio=(3.0 / 4.0, 4.0 / 3.0),  interpolation=InterpolationMode.BICUBIC,  ),  LazyCall(transforms.RandomHorizontalFlip)(p=0.5),  LazyCall(rand_augment_transform)(  config_str="rand-m9-mstd0.5-inc1",  hparams=dict(  translate_const=int(256 * 0.45),  img_mean=tuple([min(255, round(255 * x)) for x in IMAGENET_DEFAULT_MEAN]),  interpolation=str_to_interp_mode("bicubic"),  ),  ),  LazyCall(transforms.ToTensor)(),  LazyCall(transforms.Normalize)(  mean=IMAGENET_DEFAULT_MEAN,  std=IMAGENET_DEFAULT_STD,  ),  LazyCall(RandomErasing)(  probability=0.25,  mode="pixel",  max_count=1,  num_splits=0,  device="cpu",  ),  ] ) dataloader.test[0].dataset.transform = LazyCall(transforms.Compose)(  transforms=[  LazyCall(transforms.Resize)(  size=256,  interpolation=InterpolationMode.BICUBIC,  ),  LazyCall(transforms.CenterCrop)(  size=256,  ),  LazyCall(transforms.ToTensor)(),  LazyCall(transforms.Normalize)(  mean=IMAGENET_DEFAULT_MEAN,  std=IMAGENET_DEFAULT_STD,  ),  ] ) # Refine model cfg for vit training on imagenet model.cfg.num_classes = 1000 model.cfg.loss_func = SoftTargetCrossEntropy() # Refine optimizer cfg for vit model optim.lr = 1e-3 # The pytorch version is 1024 as the total batch size, 1e-3 as the learning rate optim.eps = 1e-8 optim.weight_decay = 0.05 def check_keywords_in_name(name, keywords=()):  isin = False  for keyword in keywords:  if keyword in name:  isin = True  return isin def set_weight_decay(model, skip_list=(), skip_keywords=()):  has_decay = []  no_decay = []  for name, param in model.named_parameters():  if not param.requires_grad:  continue # frozen weights  if (  len(param.shape) == 1  or name.endswith(".bias")  or (name in skip_list)  or check_keywords_in_name(name, skip_keywords)  ):  no_decay.append(param)  else:  has_decay.append(param)  return [{"params": has_decay}, {"params": no_decay, "weight_decay": 0.0}] optim.params = LazyCall(set_weight_decay)(  model=model,  skip_list=("absolute_pos_embed"),  skip_keywords=("cpb_mlp", "logit_scale", "relative_position_bias_table"), ) # Refine train cfg for vit model train.train_micro_batch_size = 128 train.test_micro_batch_size = 128 train.train_epoch = 300 train.warmup_ratio = 20 / 300 train.eval_period = 1562 train.log_period = 100 graph.enabled = True train.dist.pipeline_num_layers = sum(model.cfg.depths) train.output_dir = "./commit_7a3a_swinv2" train.rdma_enabled = True # Scheduler train.scheduler.warmup_factor = 0.001 train.scheduler.alpha = 0.01 train.scheduler.warmup_method = "linear" # Set fp16 ON train.amp.enabled = True [09/19 03:56:37] lb.config.lazy WARNING: The config contains objects that cannot serialize to a valid yaml. ./commit_7a3a_swinv2/config.yaml is human-readable but cannot be loaded. [09/19 03:56:37] lb.config.lazy WARNING: Config is saved using cloudpickle at ./commit_7a3a_swinv2/config.yaml.pkl. [09/19 03:56:37] libai INFO: Full config saved to ./commit_7a3a_swinv2/config.yaml [09/19 03:56:37] lb.engine.default INFO: > compiling dataset index builder ... [09/19 03:56:38] lb.engine.default INFO: >>> done with dataset index builder. Compilation time: 0.074 seconds [09/19 03:56:38] lb.engine.default INFO: >>> done with compiling. Compilation time: 0.075 seconds [09/19 03:56:38] lb.engine.default INFO: Prepare training, validating, testing set [09/19 03:56:41] lb.engine.default INFO: Prepare testing set [09/19 03:56:41] lb.engine.default INFO: Auto-scaling the config to train.train_iter=375342, train.warmup_iter=25023 [09/19 03:56:51] lb.engine.default INFO: Model: SwinTransformerV2( (patch_embed): PatchEmbed( (proj): Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4)) (norm): LayerNorm((96,), eps=1e-05, elementwise_affine=True) ) (pos_drop): Dropout(p=0.0, inplace=False) (layers): ModuleList( (0): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=3, bias=False, parallel=data) ) (qkv): Linear1D(in_features=96, out_features=288, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=96, out_features=96, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): Identity() (norm2): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=96, out_features=384, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=384, out_features=96, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=3, bias=False, parallel=data) ) (qkv): Linear1D(in_features=96, out_features=288, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=96, out_features=96, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=96, out_features=384, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=384, out_features=96, bias=True, parallel=row) ) ) ) (downsample): PatchMerging( (reduction): Linear1D(in_features=384, out_features=192, bias=False, parallel=data) (norm): LayerNorm((192,), eps=1e-05, elementwise_affine=True) ) ) (1): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=6, bias=False, parallel=data) ) (qkv): Linear1D(in_features=192, out_features=576, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=192, out_features=192, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=192, out_features=768, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=768, out_features=192, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=6, bias=False, parallel=data) ) (qkv): Linear1D(in_features=192, out_features=576, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=192, out_features=192, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=192, out_features=768, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=768, out_features=192, bias=True, parallel=row) ) ) ) (downsample): PatchMerging( (reduction): Linear1D(in_features=768, out_features=384, bias=False, parallel=data) (norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True) ) ) (2): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (2): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (3): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (4): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (5): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) ) (downsample): PatchMerging( (reduction): Linear1D(in_features=1536, out_features=768, bias=False, parallel=data) (norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) ) ) (3): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=24, bias=False, parallel=data) ) (qkv): Linear1D(in_features=768, out_features=2304, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=768, out_features=768, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=768, out_features=3072, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=3072, out_features=768, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=24, bias=False, parallel=data) ) (qkv): Linear1D(in_features=768, out_features=2304, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=768, out_features=768, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=768, out_features=3072, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=3072, out_features=768, bias=True, parallel=row) ) ) ) ) ) (norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (avgpool): AdaptiveAvgPool1d() (head): Linear1D(in_features=768, out_features=1000, bias=True, parallel=data) (loss_func): SoftTargetCrossEntropy() ) [09/19 03:56:51] lb.engine.trainer INFO: Starting training from iteration 0 [09/19 03:56:53] lb.models.utils.graph_base INFO: Start compling the train graph which may take some time. Please wait for a moment ... [09/19 03:56:53] lb.engine.trainer ERROR: Exception during training: Traceback (most recent call last): File "/home/shaoshitong/libai/libai/engine/trainer.py", line 146, in train self.run_step() File "/home/shaoshitong/libai/libai/engine/default.py", line 477, in run_step self._trainer.run_step(self.get_batch, self.cfg.train.input_placement_device) File "/home/shaoshitong/libai/libai/engine/trainer.py", line 337, in run_step loss_dict = self.graph(**data) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/graph.py", line 224, in __call__ self._compile(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/graph.py", line 779, in _compile _, eager_outputs = self.build_graph(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/graph.py", line 798, in build_graph outputs = self.__build_graph(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/graph.py", line 902, in __build_graph outputs = self.build(*lazy_args, **lazy_kwargs) File "/home/shaoshitong/libai/libai/models/utils/graph_base.py", line 105, in build loss_dict = self.model(**kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 248, in __call__ result = self.__block_forward(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 280, in __block_forward result = self._origin.__class__.forward(self, *args, **kwargs) File "/home/shaoshitong/libai/libai/models/swin_transformer_v2.py", line 815, in forward x = self.forward_features(images) File "/home/shaoshitong/libai/libai/models/swin_transformer_v2.py", line 794, in forward_features x = layer(x) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 248, in __call__ result = self.__block_forward(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 280, in __block_forward result = self._origin.__class__.forward(self, *args, **kwargs) File "/home/shaoshitong/libai/libai/models/swin_transformer_v2.py", line 567, in forward x = blk(x) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 248, in __call__ result = self.__block_forward(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 280, in __block_forward result = self._origin.__class__.forward(self, *args, **kwargs) File "/home/shaoshitong/libai/libai/models/swin_transformer_v2.py", line 423, in forward attn_windows = self.attn(x_windows, mask=self.attn_mask) # nW*B, window_size*window_size, C File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 248, in __call__ result = self.__block_forward(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 280, in __block_forward result = self._origin.__class__.forward(self, *args, **kwargs) File "/home/shaoshitong/libai/libai/models/swin_transformer_v2.py", line 246, in forward relative_position_bias_table = self.cpb_mlp(self.relative_coords_table.to_global( NotImplementedError: The type cast_to_global has not been supported in LazyInterpreter::Apply. [09/19 03:56:53] lb.engine.hooks INFO: Total training time: 0:00:01 (0:00:00 on hooks) [09/19 03:58:53] libai INFO: Rank of current process: 0. World size: 8 [09/19 03:58:53] libai INFO: Command line arguments: Namespace(config_file='configs/swinv2_imagenet.py', eval_only=False, fast_dev_run=False, opts=[], resume=False) [09/19 03:58:53] libai INFO: Contents of args.config_file=configs/swinv2_imagenet.py: from libai.config import LazyCall from .common.models.swinv2.swinv2_tiny_patch4_window8_256 import model from .common.models.graph import graph from .common.train import train from .common.optim import optim from .common.data.imagenet import dataloader from flowvision import transforms from flowvision.data import Mixup from flowvision.loss.cross_entropy import SoftTargetCrossEntropy from flowvision.transforms import InterpolationMode from flowvision.transforms.functional import str_to_interp_mode from flowvision.data.constants import (  IMAGENET_DEFAULT_MEAN,  IMAGENET_DEFAULT_STD, ) from flowvision.data.auto_augment import rand_augment_transform from flowvision.data.random_erasing import RandomErasing # Refine data path to imagenet dataloader.train.dataset[0].root = "/data/ImageNet/extract/" dataloader.test[0].dataset.root = "/data/ImageNet/extract/" # Add Mixup Func dataloader.train.mixup_func = LazyCall(Mixup)(  mixup_alpha=0.8,  cutmix_alpha=1.0,  prob=1.0,  switch_prob=0.5,  mode="batch",  num_classes=1000, ) dataloader.train.dataset[0].transform = LazyCall(transforms.Compose)(  transforms=[  LazyCall(transforms.RandomResizedCrop)(  size=256,  scale=(0.08, 1.0),  ratio=(3.0 / 4.0, 4.0 / 3.0),  interpolation=InterpolationMode.BICUBIC,  ),  LazyCall(transforms.RandomHorizontalFlip)(p=0.5),  LazyCall(rand_augment_transform)(  config_str="rand-m9-mstd0.5-inc1",  hparams=dict(  translate_const=int(256 * 0.45),  img_mean=tuple([min(255, round(255 * x)) for x in IMAGENET_DEFAULT_MEAN]),  interpolation=str_to_interp_mode("bicubic"),  ),  ),  LazyCall(transforms.ToTensor)(),  LazyCall(transforms.Normalize)(  mean=IMAGENET_DEFAULT_MEAN,  std=IMAGENET_DEFAULT_STD,  ),  LazyCall(RandomErasing)(  probability=0.25,  mode="pixel",  max_count=1,  num_splits=0,  device="cpu",  ),  ] ) dataloader.test[0].dataset.transform = LazyCall(transforms.Compose)(  transforms=[  LazyCall(transforms.Resize)(  size=256,  interpolation=InterpolationMode.BICUBIC,  ),  LazyCall(transforms.CenterCrop)(  size=256,  ),  LazyCall(transforms.ToTensor)(),  LazyCall(transforms.Normalize)(  mean=IMAGENET_DEFAULT_MEAN,  std=IMAGENET_DEFAULT_STD,  ),  ] ) # Refine model cfg for vit training on imagenet model.cfg.num_classes = 1000 model.cfg.loss_func = SoftTargetCrossEntropy() # Refine optimizer cfg for vit model optim.lr = 1e-3 # The pytorch version is 1024 as the total batch size, 1e-3 as the learning rate optim.eps = 1e-8 optim.weight_decay = 0.05 def check_keywords_in_name(name, keywords=()):  isin = False  for keyword in keywords:  if keyword in name:  isin = True  return isin def set_weight_decay(model, skip_list=(), skip_keywords=()):  has_decay = []  no_decay = []  for name, param in model.named_parameters():  if not param.requires_grad:  continue # frozen weights  if (  len(param.shape) == 1  or name.endswith(".bias")  or (name in skip_list)  or check_keywords_in_name(name, skip_keywords)  ):  no_decay.append(param)  else:  has_decay.append(param)  return [{"params": has_decay}, {"params": no_decay, "weight_decay": 0.0}] optim.params = LazyCall(set_weight_decay)(  model=model,  skip_list=("absolute_pos_embed"),  skip_keywords=("cpb_mlp", "logit_scale", "relative_position_bias_table"), ) # Refine train cfg for vit model train.train_micro_batch_size = 128 train.test_micro_batch_size = 128 train.train_epoch = 300 train.warmup_ratio = 20 / 300 train.eval_period = 1562 train.log_period = 100 graph.enabled = True train.dist.pipeline_num_layers = sum(model.cfg.depths) train.output_dir = "./commit_7a3a_swinv2" train.rdma_enabled = True # Scheduler train.scheduler.warmup_factor = 0.001 train.scheduler.alpha = 0.01 train.scheduler.warmup_method = "linear" # Set fp16 ON train.amp.enabled = True [09/19 03:58:53] lb.config.lazy WARNING: The config contains objects that cannot serialize to a valid yaml. ./commit_7a3a_swinv2/config.yaml is human-readable but cannot be loaded. [09/19 03:58:53] lb.config.lazy WARNING: Config is saved using cloudpickle at ./commit_7a3a_swinv2/config.yaml.pkl. [09/19 03:58:53] libai INFO: Full config saved to ./commit_7a3a_swinv2/config.yaml [09/19 03:58:53] lb.engine.default INFO: > compiling dataset index builder ... [09/19 03:58:53] lb.engine.default INFO: >>> done with dataset index builder. Compilation time: 0.069 seconds [09/19 03:58:53] lb.engine.default INFO: >>> done with compiling. Compilation time: 0.070 seconds [09/19 03:58:53] lb.engine.default INFO: Prepare training, validating, testing set [09/19 03:58:56] lb.engine.default INFO: Prepare testing set [09/19 03:58:57] lb.engine.default INFO: Auto-scaling the config to train.train_iter=375342, train.warmup_iter=25023 [09/19 03:59:06] lb.engine.default INFO: Model: SwinTransformerV2( (patch_embed): PatchEmbed( (proj): Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4)) (norm): LayerNorm((96,), eps=1e-05, elementwise_affine=True) ) (pos_drop): Dropout(p=0.0, inplace=False) (layers): ModuleList( (0): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=3, bias=False, parallel=data) ) (qkv): Linear1D(in_features=96, out_features=288, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=96, out_features=96, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): Identity() (norm2): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=96, out_features=384, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=384, out_features=96, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=3, bias=False, parallel=data) ) (qkv): Linear1D(in_features=96, out_features=288, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=96, out_features=96, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=96, out_features=384, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=384, out_features=96, bias=True, parallel=row) ) ) ) (downsample): PatchMerging( (reduction): Linear1D(in_features=384, out_features=192, bias=False, parallel=data) (norm): LayerNorm((192,), eps=1e-05, elementwise_affine=True) ) ) (1): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=6, bias=False, parallel=data) ) (qkv): Linear1D(in_features=192, out_features=576, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=192, out_features=192, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=192, out_features=768, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=768, out_features=192, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=6, bias=False, parallel=data) ) (qkv): Linear1D(in_features=192, out_features=576, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=192, out_features=192, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=192, out_features=768, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=768, out_features=192, bias=True, parallel=row) ) ) ) (downsample): PatchMerging( (reduction): Linear1D(in_features=768, out_features=384, bias=False, parallel=data) (norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True) ) ) (2): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (2): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (3): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (4): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (5): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) ) (downsample): PatchMerging( (reduction): Linear1D(in_features=1536, out_features=768, bias=False, parallel=data) (norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) ) ) (3): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=24, bias=False, parallel=data) ) (qkv): Linear1D(in_features=768, out_features=2304, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=768, out_features=768, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=768, out_features=3072, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=3072, out_features=768, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=24, bias=False, parallel=data) ) (qkv): Linear1D(in_features=768, out_features=2304, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=768, out_features=768, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=768, out_features=3072, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=3072, out_features=768, bias=True, parallel=row) ) ) ) ) ) (norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (avgpool): AdaptiveAvgPool1d() (head): Linear1D(in_features=768, out_features=1000, bias=True, parallel=data) (loss_func): SoftTargetCrossEntropy() ) [09/19 03:59:06] lb.engine.trainer INFO: Starting training from iteration 0 [09/19 03:59:08] lb.models.utils.graph_base INFO: Start compling the train graph which may take some time. Please wait for a moment ... [09/19 03:59:08] lb.engine.trainer ERROR: Exception during training: Traceback (most recent call last): File "/home/shaoshitong/libai/libai/engine/trainer.py", line 146, in train self.run_step() File "/home/shaoshitong/libai/libai/engine/default.py", line 477, in run_step self._trainer.run_step(self.get_batch, self.cfg.train.input_placement_device) File "/home/shaoshitong/libai/libai/engine/trainer.py", line 337, in run_step loss_dict = self.graph(**data) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/graph.py", line 224, in __call__ self._compile(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/graph.py", line 779, in _compile _, eager_outputs = self.build_graph(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/graph.py", line 798, in build_graph outputs = self.__build_graph(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/graph.py", line 902, in __build_graph outputs = self.build(*lazy_args, **lazy_kwargs) File "/home/shaoshitong/libai/libai/models/utils/graph_base.py", line 105, in build loss_dict = self.model(**kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 248, in __call__ result = self.__block_forward(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 280, in __block_forward result = self._origin.__class__.forward(self, *args, **kwargs) File "/home/shaoshitong/libai/libai/models/swin_transformer_v2.py", line 812, in forward x = self.forward_features(images) File "/home/shaoshitong/libai/libai/models/swin_transformer_v2.py", line 791, in forward_features x = layer(x) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 248, in __call__ result = self.__block_forward(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 280, in __block_forward result = self._origin.__class__.forward(self, *args, **kwargs) File "/home/shaoshitong/libai/libai/models/swin_transformer_v2.py", line 564, in forward x = blk(x) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 248, in __call__ result = self.__block_forward(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 280, in __block_forward result = self._origin.__class__.forward(self, *args, **kwargs) File "/home/shaoshitong/libai/libai/models/swin_transformer_v2.py", line 420, in forward attn_windows = self.attn(x_windows, mask=self.attn_mask) # nW*B, window_size*window_size, C File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 248, in __call__ result = self.__block_forward(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 280, in __block_forward result = self._origin.__class__.forward(self, *args, **kwargs) File "/home/shaoshitong/libai/libai/models/swin_transformer_v2.py", line 246, in forward relative_position_bias_table = self.cpb_mlp(self.relative_coords_table).view( File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 248, in __call__ result = self.__block_forward(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 280, in __block_forward result = self._origin.__class__.forward(self, *args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/utils/container.py", line 99, in forward input = module(input) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 248, in __call__ result = self.__block_forward(*args, **kwargs) File "/home/shaoshitong/oneflow/python/oneflow/nn/graph/block.py", line 280, in __block_forward result = self._origin.__class__.forward(self, *args, **kwargs) File "/home/shaoshitong/libai/libai/layers/linear.py", line 126, in forward x = x.to_global(grad_sbp=x.sbp) RuntimeError: RuntimeError : Local tensor has no sbp property. sbp is the description in the oneflow distributed case, you can refer to https://docs.oneflow.org/master/parallelism/03_global_tensor.html; For example, create a global tensor like this : 'x = oneflow.tensor((2,3, placement=oneflow.placement("cuda", {0: 0}), sbp=oneflow.sbp.broadcast))', then 'x.sbp' is 'oneflow.sbp.broadcast' [09/19 03:59:08] lb.engine.hooks INFO: Total training time: 0:00:01 (0:00:00 on hooks) [09/19 04:04:12] libai INFO: Rank of current process: 0. World size: 8 [09/19 04:04:12] libai INFO: Command line arguments: Namespace(config_file='configs/swinv2_imagenet.py', eval_only=False, fast_dev_run=False, opts=[], resume=False) [09/19 04:04:12] libai INFO: Contents of args.config_file=configs/swinv2_imagenet.py: from libai.config import LazyCall from .common.models.swinv2.swinv2_tiny_patch4_window8_256 import model from .common.models.graph import graph from .common.train import train from .common.optim import optim from .common.data.imagenet import dataloader from flowvision import transforms from flowvision.data import Mixup from flowvision.loss.cross_entropy import SoftTargetCrossEntropy from flowvision.transforms import InterpolationMode from flowvision.transforms.functional import str_to_interp_mode from flowvision.data.constants import (  IMAGENET_DEFAULT_MEAN,  IMAGENET_DEFAULT_STD, ) from flowvision.data.auto_augment import rand_augment_transform from flowvision.data.random_erasing import RandomErasing # Refine data path to imagenet dataloader.train.dataset[0].root = "/data/ImageNet/extract/" dataloader.test[0].dataset.root = "/data/ImageNet/extract/" # Add Mixup Func dataloader.train.mixup_func = LazyCall(Mixup)(  mixup_alpha=0.8,  cutmix_alpha=1.0,  prob=1.0,  switch_prob=0.5,  mode="batch",  num_classes=1000, ) dataloader.train.dataset[0].transform = LazyCall(transforms.Compose)(  transforms=[  LazyCall(transforms.RandomResizedCrop)(  size=256,  scale=(0.08, 1.0),  ratio=(3.0 / 4.0, 4.0 / 3.0),  interpolation=InterpolationMode.BICUBIC,  ),  LazyCall(transforms.RandomHorizontalFlip)(p=0.5),  LazyCall(rand_augment_transform)(  config_str="rand-m9-mstd0.5-inc1",  hparams=dict(  translate_const=int(256 * 0.45),  img_mean=tuple([min(255, round(255 * x)) for x in IMAGENET_DEFAULT_MEAN]),  interpolation=str_to_interp_mode("bicubic"),  ),  ),  LazyCall(transforms.ToTensor)(),  LazyCall(transforms.Normalize)(  mean=IMAGENET_DEFAULT_MEAN,  std=IMAGENET_DEFAULT_STD,  ),  LazyCall(RandomErasing)(  probability=0.25,  mode="pixel",  max_count=1,  num_splits=0,  device="cpu",  ),  ] ) dataloader.test[0].dataset.transform = LazyCall(transforms.Compose)(  transforms=[  LazyCall(transforms.Resize)(  size=256,  interpolation=InterpolationMode.BICUBIC,  ),  LazyCall(transforms.CenterCrop)(  size=256,  ),  LazyCall(transforms.ToTensor)(),  LazyCall(transforms.Normalize)(  mean=IMAGENET_DEFAULT_MEAN,  std=IMAGENET_DEFAULT_STD,  ),  ] ) # Refine model cfg for vit training on imagenet model.cfg.num_classes = 1000 model.cfg.loss_func = SoftTargetCrossEntropy() # Refine optimizer cfg for vit model optim.lr = 1e-3 # The pytorch version is 1024 as the total batch size, 1e-3 as the learning rate optim.eps = 1e-8 optim.weight_decay = 0.05 def check_keywords_in_name(name, keywords=()):  isin = False  for keyword in keywords:  if keyword in name:  isin = True  return isin def set_weight_decay(model, skip_list=(), skip_keywords=()):  has_decay = []  no_decay = []  for name, param in model.named_parameters():  if not param.requires_grad:  continue # frozen weights  if (  len(param.shape) == 1  or name.endswith(".bias")  or (name in skip_list)  or check_keywords_in_name(name, skip_keywords)  ):  no_decay.append(param)  else:  has_decay.append(param)  return [{"params": has_decay}, {"params": no_decay, "weight_decay": 0.0}] optim.params = LazyCall(set_weight_decay)(  model=model,  skip_list=("absolute_pos_embed"),  skip_keywords=("cpb_mlp", "logit_scale", "relative_position_bias_table"), ) # Refine train cfg for vit model train.train_micro_batch_size = 128 train.test_micro_batch_size = 128 train.train_epoch = 300 train.warmup_ratio = 20 / 300 train.eval_period = 1562 train.log_period = 100 graph.enabled = False train.dist.pipeline_num_layers = sum(model.cfg.depths) train.output_dir = "./commit_7a3a_swinv2" train.rdma_enabled = True # Scheduler train.scheduler.warmup_factor = 0.001 train.scheduler.alpha = 0.01 train.scheduler.warmup_method = "linear" # Set fp16 ON train.amp.enabled = True [09/19 04:04:12] lb.config.lazy WARNING: The config contains objects that cannot serialize to a valid yaml. ./commit_7a3a_swinv2/config.yaml is human-readable but cannot be loaded. [09/19 04:04:12] lb.config.lazy WARNING: Config is saved using cloudpickle at ./commit_7a3a_swinv2/config.yaml.pkl. [09/19 04:04:12] libai INFO: Full config saved to ./commit_7a3a_swinv2/config.yaml [09/19 04:04:12] lb.engine.default INFO: > compiling dataset index builder ... [09/19 04:04:12] lb.engine.default INFO: >>> done with dataset index builder. Compilation time: 0.071 seconds [09/19 04:04:12] lb.engine.default INFO: >>> done with compiling. Compilation time: 0.072 seconds [09/19 04:04:12] lb.engine.default INFO: Prepare training, validating, testing set [09/19 04:04:15] lb.engine.default INFO: Prepare testing set [09/19 04:04:16] lb.engine.default INFO: Auto-scaling the config to train.train_iter=375342, train.warmup_iter=25023 [09/19 04:04:26] lb.engine.default INFO: Model: SwinTransformerV2( (patch_embed): PatchEmbed( (proj): Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4)) (norm): LayerNorm((96,), eps=1e-05, elementwise_affine=True) ) (pos_drop): Dropout(p=0.0, inplace=False) (layers): ModuleList( (0): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=3, bias=False, parallel=data) ) (qkv): Linear1D(in_features=96, out_features=288, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=96, out_features=96, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): Identity() (norm2): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=96, out_features=384, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=384, out_features=96, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=3, bias=False, parallel=data) ) (qkv): Linear1D(in_features=96, out_features=288, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=96, out_features=96, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((96,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=96, out_features=384, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=384, out_features=96, bias=True, parallel=row) ) ) ) (downsample): PatchMerging( (reduction): Linear1D(in_features=384, out_features=192, bias=False, parallel=data) (norm): LayerNorm((192,), eps=1e-05, elementwise_affine=True) ) ) (1): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=6, bias=False, parallel=data) ) (qkv): Linear1D(in_features=192, out_features=576, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=192, out_features=192, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=192, out_features=768, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=768, out_features=192, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=6, bias=False, parallel=data) ) (qkv): Linear1D(in_features=192, out_features=576, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=192, out_features=192, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((192,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=192, out_features=768, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=768, out_features=192, bias=True, parallel=row) ) ) ) (downsample): PatchMerging( (reduction): Linear1D(in_features=768, out_features=384, bias=False, parallel=data) (norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True) ) ) (2): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (2): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (3): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (4): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) (5): SwinTransformerBlock( (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=12, bias=False, parallel=data) ) (qkv): Linear1D(in_features=384, out_features=1152, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row) ) ) ) (downsample): PatchMerging( (reduction): Linear1D(in_features=1536, out_features=768, bias=False, parallel=data) (norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) ) ) (3): BasicLayer( (blocks): ModuleList( (0): SwinTransformerBlock( (norm1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=24, bias=False, parallel=data) ) (qkv): Linear1D(in_features=768, out_features=2304, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=768, out_features=768, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=768, out_features=3072, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=3072, out_features=768, bias=True, parallel=row) ) ) (1): SwinTransformerBlock( (norm1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( (cpb_mlp): Sequential( (0): Linear1D(in_features=2, out_features=512, bias=True, parallel=data) (1): ReLU(inplace=True) (2): Linear1D(in_features=512, out_features=24, bias=False, parallel=data) ) (qkv): Linear1D(in_features=768, out_features=2304, bias=False, parallel=data) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear1D(in_features=768, out_features=768, bias=True, parallel=data) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath() (norm2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0 (dense_h_to_4h): Linear1D(in_features=768, out_features=3072, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=3072, out_features=768, bias=True, parallel=row) ) ) ) ) ) (norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (avgpool): AdaptiveAvgPool1d() (head): Linear1D(in_features=768, out_features=1000, bias=True, parallel=data) (loss_func): SoftTargetCrossEntropy() ) [09/19 04:04:26] lb.engine.trainer INFO: Starting training from iteration 0 [09/19 04:05:26] lb.utils.events INFO: eta: 2 days, 6:43:44 iteration: 99/375342 consumed_samples: 102400 total_loss: 0.8695 time: 0.5248 s/iter data_time: 0.0339 s/iter total_throughput: 1951.12 samples/s lr: 4.91e-06 [09/19 04:06:19] lb.utils.events INFO: eta: 2 days, 6:47:42 iteration: 199/375342 consumed_samples: 204800 total_loss: 0.8676 time: 0.5257 s/iter data_time: 0.0355 s/iter total_throughput: 1947.76 samples/s lr: 8.86e-06 [09/19 04:07:12] lb.utils.events INFO: eta: 2 days, 6:49:00 iteration: 299/375342 consumed_samples: 307200 total_loss: 0.864 time: 0.5262 s/iter data_time: 0.0359 s/iter total_throughput: 1945.97 samples/s lr: 1.28e-05 [09/19 04:08:04] lb.utils.events INFO: eta: 2 days, 6:48:38 iteration: 399/375342 consumed_samples: 409600 total_loss: 0.8615 time: 0.5265 s/iter data_time: 0.0356 s/iter total_throughput: 1944.94 samples/s lr: 1.68e-05 [09/19 04:08:57] lb.utils.events INFO: eta: 2 days, 6:49:57 iteration: 499/375342 consumed_samples: 512000 total_loss: 0.8594 time: 0.5267 s/iter data_time: 0.0375 s/iter total_throughput: 1944.02 samples/s lr: 2.07e-05 [09/19 04:09:50] lb.utils.events INFO: eta: 2 days, 6:50:55 iteration: 599/375342 consumed_samples: 614400 total_loss: 0.8577 time: 0.5270 s/iter data_time: 0.0362 s/iter total_throughput: 1943.16 samples/s lr: 2.47e-05 [09/19 04:10:43] lb.utils.events INFO: eta: 2 days, 6:51:38 iteration: 699/375342 consumed_samples: 716800 total_loss: 0.8561 time: 0.5272 s/iter data_time: 0.0355 s/iter total_throughput: 1942.31 samples/s lr: 2.86e-05 [09/19 04:11:36] lb.utils.events INFO: eta: 2 days, 6:51:32 iteration: 799/375342 consumed_samples: 819200 total_loss: 0.8549 time: 0.5274 s/iter data_time: 0.0376 s/iter total_throughput: 1941.50 samples/s lr: 3.26e-05 [09/19 04:12:29] lb.utils.events INFO: eta: 2 days, 6:52:11 iteration: 899/375342 consumed_samples: 921600 total_loss: 0.8521 time: 0.5277 s/iter data_time: 0.0376 s/iter total_throughput: 1940.53 samples/s lr: 3.65e-05 [09/19 04:13:22] lb.utils.events INFO: eta: 2 days, 6:52:32 iteration: 999/375342 consumed_samples: 1024000 total_loss: 0.8493 time: 0.5280 s/iter data_time: 0.0369 s/iter total_throughput: 1939.49 samples/s lr: 4.05e-05 [09/19 04:14:15] lb.utils.events INFO: eta: 2 days, 6:55:44 iteration: 1099/375342 consumed_samples: 1126400 total_loss: 0.8461 time: 0.5282 s/iter data_time: 0.0389 s/iter total_throughput: 1938.78 samples/s lr: 4.44e-05 [09/19 04:15:08] lb.utils.events INFO: eta: 2 days, 6:56:37 iteration: 1199/375342 consumed_samples: 1228800 total_loss: 0.8411 time: 0.5283 s/iter data_time: 0.0390 s/iter total_throughput: 1938.26 samples/s lr: 4.83e-05 [09/19 04:16:01] lb.utils.events INFO: eta: 2 days, 6:57:28 iteration: 1299/375342 consumed_samples: 1331200 total_loss: 0.8337 time: 0.5285 s/iter data_time: 0.0379 s/iter total_throughput: 1937.68 samples/s lr: 5.23e-05 [09/19 04:16:54] lb.utils.events INFO: eta: 2 days, 6:58:18 iteration: 1399/375342 consumed_samples: 1433600 total_loss: 0.8245 time: 0.5286 s/iter data_time: 0.0380 s/iter total_throughput: 1937.08 samples/s lr: 5.62e-05 [09/19 04:17:47] lb.utils.events INFO: eta: 2 days, 7:00:18 iteration: 1499/375342 consumed_samples: 1536000 total_loss: 0.8193 time: 0.5288 s/iter data_time: 0.0391 s/iter total_throughput: 1936.31 samples/s lr: 6.02e-05 [09/19 04:18:41] lb.utils.events INFO: eta: 2 days, 7:01:37 iteration: 1599/375342 consumed_samples: 1638400 total_loss: 0.8179 time: 0.5291 s/iter data_time: 0.0403 s/iter total_throughput: 1935.38 samples/s lr: 6.41e-05 [09/19 04:19:34] lb.utils.events INFO: eta: 2 days, 7:04:09 iteration: 1699/375342 consumed_samples: 1740800 total_loss: 0.8173 time: 0.5294 s/iter data_time: 0.0406 s/iter total_throughput: 1934.42 samples/s lr: 6.81e-05 [09/19 04:20:27] lb.utils.events INFO: eta: 2 days, 7:06:25 iteration: 1799/375342 consumed_samples: 1843200 total_loss: 0.8139 time: 0.5296 s/iter data_time: 0.0390 s/iter total_throughput: 1933.57 samples/s lr: 7.20e-05 [09/19 04:21:21] lb.utils.events INFO: eta: 2 days, 7:07:38 iteration: 1899/375342 consumed_samples: 1945600 total_loss: 0.811 time: 0.5298 s/iter data_time: 0.0416 s/iter total_throughput: 1932.72 samples/s lr: 7.60e-05 [09/19 04:22:14] lb.utils.events INFO: eta: 2 days, 7:08:57 iteration: 1999/375342 consumed_samples: 2048000 total_loss: 0.8067 time: 0.5301 s/iter data_time: 0.0414 s/iter total_throughput: 1931.81 samples/s lr: 7.99e-05 [09/19 04:23:08] lb.utils.events INFO: eta: 2 days, 7:10:10 iteration: 2099/375342 consumed_samples: 2150400 total_loss: 0.7993 time: 0.5303 s/iter data_time: 0.0417 s/iter total_throughput: 1930.96 samples/s lr: 8.39e-05 [09/19 04:24:01] lb.utils.events INFO: eta: 2 days, 7:12:14 iteration: 2199/375342 consumed_samples: 2252800 total_loss: 0.7959 time: 0.5305 s/iter data_time: 0.0416 s/iter total_throughput: 1930.26 samples/s lr: 8.78e-05 [09/19 04:24:55] lb.utils.events INFO: eta: 2 days, 7:14:52 iteration: 2299/375342 consumed_samples: 2355200 total_loss: 0.7943 time: 0.5307 s/iter data_time: 0.0419 s/iter total_throughput: 1929.54 samples/s lr: 9.18e-05 [09/19 04:25:48] lb.utils.events INFO: eta: 2 days, 7:16:46 iteration: 2399/375342 consumed_samples: 2457600 total_loss: 0.7924 time: 0.5309 s/iter data_time: 0.0424 s/iter total_throughput: 1928.82 samples/s lr: 9.57e-05 [09/19 04:26:42] lb.utils.events INFO: eta: 2 days, 7:18:57 iteration: 2499/375342 consumed_samples: 2560000 total_loss: 0.79 time: 0.5311 s/iter data_time: 0.0414 s/iter total_throughput: 1928.10 samples/s lr: 9.97e-05 [09/19 04:27:35] lb.utils.events INFO: eta: 2 days, 7:19:45 iteration: 2599/375342 consumed_samples: 2662400 total_loss: 0.7872 time: 0.5313 s/iter data_time: 0.0431 s/iter total_throughput: 1927.48 samples/s lr: 1.04e-04 [09/19 04:28:29] lb.utils.events INFO: eta: 2 days, 7:19:57 iteration: 2699/375342 consumed_samples: 2764800 total_loss: 0.7828 time: 0.5314 s/iter data_time: 0.0431 s/iter total_throughput: 1926.81 samples/s lr: 1.08e-04 [09/19 04:29:23] lb.utils.events INFO: eta: 2 days, 7:20:56 iteration: 2799/375342 consumed_samples: 2867200 total_loss: 0.7804 time: 0.5316 s/iter data_time: 0.0429 s/iter total_throughput: 1926.18 samples/s lr: 1.12e-04 [09/19 04:30:16] lb.utils.events INFO: eta: 2 days, 7:22:14 iteration: 2899/375342 consumed_samples: 2969600 total_loss: 0.7795 time: 0.5318 s/iter data_time: 0.0434 s/iter total_throughput: 1925.55 samples/s lr: 1.15e-04 [09/19 04:31:10] lb.utils.events INFO: eta: 2 days, 7:22:27 iteration: 2999/375342 consumed_samples: 3072000 total_loss: 0.7762 time: 0.5320 s/iter data_time: 0.0447 s/iter total_throughput: 1924.91 samples/s lr: 1.19e-04 [09/19 04:32:04] lb.utils.events INFO: eta: 2 days, 7:23:30 iteration: 3099/375342 consumed_samples: 3174400 total_loss: 0.7742 time: 0.5321 s/iter data_time: 0.0440 s/iter total_throughput: 1924.31 samples/s lr: 1.23e-04 [09/19 04:32:58] lb.utils.events INFO: eta: 2 days, 7:23:42 iteration: 3199/375342 consumed_samples: 3276800 total_loss: 0.7728 time: 0.5323 s/iter data_time: 0.0430 s/iter total_throughput: 1923.71 samples/s lr: 1.27e-04 [09/19 04:33:51] lb.utils.events INFO: eta: 2 days, 7:24:53 iteration: 3299/375342 consumed_samples: 3379200 total_loss: 0.7677 time: 0.5325 s/iter data_time: 0.0459 s/iter total_throughput: 1923.08 samples/s lr: 1.31e-04 [09/19 04:34:45] lb.utils.events INFO: eta: 2 days, 7:25:30 iteration: 3399/375342 consumed_samples: 3481600 total_loss: 0.7654 time: 0.5326 s/iter data_time: 0.0454 s/iter total_throughput: 1922.60 samples/s lr: 1.35e-04 [09/19 04:35:39] lb.utils.events INFO: eta: 2 days, 7:26:29 iteration: 3499/375342 consumed_samples: 3584000 total_loss: 0.7661 time: 0.5328 s/iter data_time: 0.0485 s/iter total_throughput: 1921.81 samples/s lr: 1.39e-04 [09/19 04:36:33] lb.utils.events INFO: eta: 2 days, 7:27:30 iteration: 3599/375342 consumed_samples: 3686400 total_loss: 0.7632 time: 0.5330 s/iter data_time: 0.0468 s/iter total_throughput: 1921.31 samples/s lr: 1.43e-04 [09/19 04:37:27] lb.utils.events INFO: eta: 2 days, 7:27:38 iteration: 3699/375342 consumed_samples: 3788800 total_loss: 0.7588 time: 0.5331 s/iter data_time: 0.0463 s/iter total_throughput: 1920.82 samples/s lr: 1.47e-04 [09/19 04:38:21] lb.utils.events INFO: eta: 2 days, 7:27:41 iteration: 3799/375342 consumed_samples: 3891200 total_loss: 0.7554 time: 0.5332 s/iter data_time: 0.0467 s/iter total_throughput: 1920.31 samples/s lr: 1.51e-04 [09/19 04:39:15] lb.utils.events INFO: eta: 2 days, 7:27:41 iteration: 3899/375342 consumed_samples: 3993600 total_loss: 0.7543 time: 0.5334 s/iter data_time: 0.0452 s/iter total_throughput: 1919.81 samples/s lr: 1.55e-04 [09/19 04:40:09] lb.utils.events INFO: eta: 2 days, 7:27:16 iteration: 3999/375342 consumed_samples: 4096000 total_loss: 0.7564 time: 0.5335 s/iter data_time: 0.0483 s/iter total_throughput: 1919.27 samples/s lr: 1.59e-04 [09/19 04:41:03] lb.utils.events INFO: eta: 2 days, 7:28:46 iteration: 4099/375342 consumed_samples: 4198400 total_loss: 0.7551 time: 0.5337 s/iter data_time: 0.0487 s/iter total_throughput: 1918.70 samples/s lr: 1.63e-04 [09/19 04:41:57] lb.utils.events INFO: eta: 2 days, 7:31:09 iteration: 4199/375342 consumed_samples: 4300800 total_loss: 0.7504 time: 0.5339 s/iter data_time: 0.0488 s/iter total_throughput: 1918.10 samples/s lr: 1.67e-04 [09/19 04:42:51] lb.utils.events INFO: eta: 2 days, 7:32:39 iteration: 4299/375342 consumed_samples: 4403200 total_loss: 0.7429 time: 0.5340 s/iter data_time: 0.0479 s/iter total_throughput: 1917.52 samples/s lr: 1.71e-04 [09/19 04:43:45] lb.utils.events INFO: eta: 2 days, 7:34:35 iteration: 4399/375342 consumed_samples: 4505600 total_loss: 0.7429 time: 0.5342 s/iter data_time: 0.0487 s/iter total_throughput: 1916.94 samples/s lr: 1.75e-04 [09/19 04:44:39] lb.utils.events INFO: eta: 2 days, 7:35:33 iteration: 4499/375342 consumed_samples: 4608000 total_loss: 0.7466 time: 0.5344 s/iter data_time: 0.0498 s/iter total_throughput: 1916.33 samples/s lr: 1.79e-04 [09/19 04:45:33] lb.utils.events INFO: eta: 2 days, 7:36:09 iteration: 4599/375342 consumed_samples: 4710400 total_loss: 0.7408 time: 0.5345 s/iter data_time: 0.0490 s/iter total_throughput: 1915.82 samples/s lr: 1.83e-04 [09/19 04:46:27] lb.utils.events INFO: eta: 2 days, 7:36:11 iteration: 4699/375342 consumed_samples: 4812800 total_loss: 0.7394 time: 0.5346 s/iter data_time: 0.0499 s/iter total_throughput: 1915.37 samples/s lr: 1.87e-04 [09/19 04:47:21] lb.utils.events INFO: eta: 2 days, 7:36:49 iteration: 4799/375342 consumed_samples: 4915200 total_loss: 0.7398 time: 0.5348 s/iter data_time: 0.0478 s/iter total_throughput: 1914.90 samples/s lr: 1.91e-04 [09/19 04:48:15] lb.utils.events INFO: eta: 2 days, 7:37:15 iteration: 4899/375342 consumed_samples: 5017600 total_loss: 0.7327 time: 0.5349 s/iter data_time: 0.0473 s/iter total_throughput: 1914.47 samples/s lr: 1.94e-04 [09/19 04:49:10] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0004999 [09/19 04:49:10] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 04:49:10] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 04:49:15] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0682 s/iter. Inference: 0.2394 s/iter. Eval: 0.0023 s/iter. Total: 0.3099 s/iter. ETA=0:00:11 [09/19 04:49:20] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0866 s/iter. Inference: 0.2446 s/iter. Eval: 0.0023 s/iter. Total: 0.3339 s/iter. ETA=0:00:07 [09/19 04:49:25] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0829 s/iter. Inference: 0.2428 s/iter. Eval: 0.0025 s/iter. Total: 0.3285 s/iter. ETA=0:00:01 [09/19 04:49:27] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 04:49:27] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.316569 (0.000286 s / iter per device, on 8 devices) [09/19 04:49:27] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000213 s / iter per device, on 8 devices) [09/19 04:49:27] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 04:49:27] lb.evaluation.utils INFO: copypaste: Acc@1=12.778 [09/19 04:49:27] lb.evaluation.utils INFO: copypaste: Acc@5=30.226 [09/19 04:49:27] lb.engine.hooks INFO: Saved first model at 12.77800 @ 4999 steps [09/19 04:49:27] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 04:49:28] lb.utils.events INFO: eta: 2 days, 7:36:44 iteration: 4999/375342 consumed_samples: 5120000 total_loss: 0.7314 time: 0.5350 s/iter data_time: 0.0483 s/iter total_throughput: 1914.02 samples/s lr: 1.98e-04 [09/19 04:50:21] lb.utils.events INFO: eta: 2 days, 7:31:55 iteration: 5099/375342 consumed_samples: 5222400 total_loss: 0.7314 time: 0.5349 s/iter data_time: 0.0380 s/iter total_throughput: 1914.35 samples/s lr: 2.02e-04 [09/19 04:51:14] lb.utils.events INFO: eta: 2 days, 7:26:37 iteration: 5199/375342 consumed_samples: 5324800 total_loss: 0.7308 time: 0.5348 s/iter data_time: 0.0362 s/iter total_throughput: 1914.65 samples/s lr: 2.06e-04 [09/19 04:52:07] lb.utils.events INFO: eta: 2 days, 7:18:31 iteration: 5299/375342 consumed_samples: 5427200 total_loss: 0.729 time: 0.5347 s/iter data_time: 0.0393 s/iter total_throughput: 1914.94 samples/s lr: 2.10e-04 [09/19 04:53:00] lb.utils.events INFO: eta: 2 days, 7:10:12 iteration: 5399/375342 consumed_samples: 5529600 total_loss: 0.7244 time: 0.5347 s/iter data_time: 0.0346 s/iter total_throughput: 1915.19 samples/s lr: 2.14e-04 [09/19 04:53:53] lb.utils.events INFO: eta: 2 days, 7:00:35 iteration: 5499/375342 consumed_samples: 5632000 total_loss: 0.7196 time: 0.5346 s/iter data_time: 0.0377 s/iter total_throughput: 1915.41 samples/s lr: 2.18e-04 [09/19 04:54:46] lb.utils.events INFO: eta: 2 days, 6:52:36 iteration: 5599/375342 consumed_samples: 5734400 total_loss: 0.7162 time: 0.5345 s/iter data_time: 0.0362 s/iter total_throughput: 1915.63 samples/s lr: 2.22e-04 [09/19 04:55:40] lb.utils.events INFO: eta: 2 days, 6:45:26 iteration: 5699/375342 consumed_samples: 5836800 total_loss: 0.7136 time: 0.5345 s/iter data_time: 0.0380 s/iter total_throughput: 1915.83 samples/s lr: 2.26e-04 [09/19 04:56:33] lb.utils.events INFO: eta: 2 days, 6:39:12 iteration: 5799/375342 consumed_samples: 5939200 total_loss: 0.7136 time: 0.5344 s/iter data_time: 0.0388 s/iter total_throughput: 1916.07 samples/s lr: 2.30e-04 [09/19 04:57:26] lb.utils.events INFO: eta: 2 days, 6:34:08 iteration: 5899/375342 consumed_samples: 6041600 total_loss: 0.7136 time: 0.5344 s/iter data_time: 0.0361 s/iter total_throughput: 1916.26 samples/s lr: 2.34e-04 [09/19 04:58:19] lb.utils.events INFO: eta: 2 days, 6:29:26 iteration: 5999/375342 consumed_samples: 6144000 total_loss: 0.7103 time: 0.5343 s/iter data_time: 0.0397 s/iter total_throughput: 1916.47 samples/s lr: 2.38e-04 [09/19 04:59:12] lb.utils.events INFO: eta: 2 days, 6:30:01 iteration: 6099/375342 consumed_samples: 6246400 total_loss: 0.7046 time: 0.5343 s/iter data_time: 0.0350 s/iter total_throughput: 1916.65 samples/s lr: 2.42e-04 [09/19 05:00:05] lb.utils.events INFO: eta: 2 days, 6:29:01 iteration: 6199/375342 consumed_samples: 6348800 total_loss: 0.7052 time: 0.5342 s/iter data_time: 0.0389 s/iter total_throughput: 1916.86 samples/s lr: 2.46e-04 [09/19 05:00:58] lb.utils.events INFO: eta: 2 days, 6:28:43 iteration: 6299/375342 consumed_samples: 6451200 total_loss: 0.7067 time: 0.5342 s/iter data_time: 0.0397 s/iter total_throughput: 1917.04 samples/s lr: 2.50e-04 [09/19 05:01:51] lb.utils.events INFO: eta: 2 days, 6:26:23 iteration: 6399/375342 consumed_samples: 6553600 total_loss: 0.7019 time: 0.5341 s/iter data_time: 0.0378 s/iter total_throughput: 1917.26 samples/s lr: 2.54e-04 [09/19 05:02:44] lb.utils.events INFO: eta: 2 days, 6:24:52 iteration: 6499/375342 consumed_samples: 6656000 total_loss: 0.6992 time: 0.5340 s/iter data_time: 0.0372 s/iter total_throughput: 1917.47 samples/s lr: 2.58e-04 [09/19 05:03:38] lb.utils.events INFO: eta: 2 days, 6:23:55 iteration: 6599/375342 consumed_samples: 6758400 total_loss: 0.6963 time: 0.5340 s/iter data_time: 0.0355 s/iter total_throughput: 1917.64 samples/s lr: 2.62e-04 [09/19 05:04:31] lb.utils.events INFO: eta: 2 days, 6:21:56 iteration: 6699/375342 consumed_samples: 6860800 total_loss: 0.6915 time: 0.5339 s/iter data_time: 0.0368 s/iter total_throughput: 1917.83 samples/s lr: 2.66e-04 [09/19 05:05:24] lb.utils.events INFO: eta: 2 days, 6:19:36 iteration: 6799/375342 consumed_samples: 6963200 total_loss: 0.6928 time: 0.5339 s/iter data_time: 0.0371 s/iter total_throughput: 1918.04 samples/s lr: 2.69e-04 [09/19 05:06:17] lb.utils.events INFO: eta: 2 days, 6:17:53 iteration: 6899/375342 consumed_samples: 7065600 total_loss: 0.6909 time: 0.5338 s/iter data_time: 0.0420 s/iter total_throughput: 1918.14 samples/s lr: 2.73e-04 [09/19 05:07:10] lb.utils.events INFO: eta: 2 days, 6:16:59 iteration: 6999/375342 consumed_samples: 7168000 total_loss: 0.6901 time: 0.5338 s/iter data_time: 0.0355 s/iter total_throughput: 1918.32 samples/s lr: 2.77e-04 [09/19 05:08:03] lb.utils.events INFO: eta: 2 days, 6:15:53 iteration: 7099/375342 consumed_samples: 7270400 total_loss: 0.6884 time: 0.5337 s/iter data_time: 0.0362 s/iter total_throughput: 1918.51 samples/s lr: 2.81e-04 [09/19 05:08:56] lb.utils.events INFO: eta: 2 days, 6:15:05 iteration: 7199/375342 consumed_samples: 7372800 total_loss: 0.6853 time: 0.5337 s/iter data_time: 0.0352 s/iter total_throughput: 1918.67 samples/s lr: 2.85e-04 [09/19 05:09:49] lb.utils.events INFO: eta: 2 days, 6:14:18 iteration: 7299/375342 consumed_samples: 7475200 total_loss: 0.6845 time: 0.5337 s/iter data_time: 0.0355 s/iter total_throughput: 1918.80 samples/s lr: 2.89e-04 [09/19 05:10:42] lb.utils.events INFO: eta: 2 days, 6:14:38 iteration: 7399/375342 consumed_samples: 7577600 total_loss: 0.6812 time: 0.5336 s/iter data_time: 0.0365 s/iter total_throughput: 1918.97 samples/s lr: 2.93e-04 [09/19 05:11:35] lb.utils.events INFO: eta: 2 days, 6:13:49 iteration: 7499/375342 consumed_samples: 7680000 total_loss: 0.6799 time: 0.5336 s/iter data_time: 0.0311 s/iter total_throughput: 1919.12 samples/s lr: 2.97e-04 [09/19 05:12:28] lb.utils.events INFO: eta: 2 days, 6:13:15 iteration: 7599/375342 consumed_samples: 7782400 total_loss: 0.6825 time: 0.5335 s/iter data_time: 0.0310 s/iter total_throughput: 1919.27 samples/s lr: 3.01e-04 [09/19 05:13:21] lb.utils.events INFO: eta: 2 days, 6:13:51 iteration: 7699/375342 consumed_samples: 7884800 total_loss: 0.6713 time: 0.5335 s/iter data_time: 0.0326 s/iter total_throughput: 1919.38 samples/s lr: 3.05e-04 [09/19 05:14:15] lb.utils.events INFO: eta: 2 days, 6:15:26 iteration: 7799/375342 consumed_samples: 7987200 total_loss: 0.6686 time: 0.5335 s/iter data_time: 0.0389 s/iter total_throughput: 1919.44 samples/s lr: 3.09e-04 [09/19 05:15:08] lb.utils.events INFO: eta: 2 days, 6:16:39 iteration: 7899/375342 consumed_samples: 8089600 total_loss: 0.6657 time: 0.5335 s/iter data_time: 0.0358 s/iter total_throughput: 1919.48 samples/s lr: 3.13e-04 [09/19 05:16:01] lb.utils.events INFO: eta: 2 days, 6:17:12 iteration: 7999/375342 consumed_samples: 8192000 total_loss: 0.6678 time: 0.5335 s/iter data_time: 0.0366 s/iter total_throughput: 1919.52 samples/s lr: 3.17e-04 [09/19 05:16:54] lb.utils.events INFO: eta: 2 days, 6:17:26 iteration: 8099/375342 consumed_samples: 8294400 total_loss: 0.6706 time: 0.5334 s/iter data_time: 0.0387 s/iter total_throughput: 1919.59 samples/s lr: 3.21e-04 [09/19 05:17:48] lb.utils.events INFO: eta: 2 days, 6:17:03 iteration: 8199/375342 consumed_samples: 8396800 total_loss: 0.6634 time: 0.5334 s/iter data_time: 0.0352 s/iter total_throughput: 1919.66 samples/s lr: 3.25e-04 [09/19 05:18:41] lb.utils.events INFO: eta: 2 days, 6:16:38 iteration: 8299/375342 consumed_samples: 8499200 total_loss: 0.6642 time: 0.5334 s/iter data_time: 0.0363 s/iter total_throughput: 1919.70 samples/s lr: 3.29e-04 [09/19 05:19:34] lb.utils.events INFO: eta: 2 days, 6:16:17 iteration: 8399/375342 consumed_samples: 8601600 total_loss: 0.6619 time: 0.5334 s/iter data_time: 0.0380 s/iter total_throughput: 1919.76 samples/s lr: 3.33e-04 [09/19 05:20:27] lb.utils.events INFO: eta: 2 days, 6:15:45 iteration: 8499/375342 consumed_samples: 8704000 total_loss: 0.6593 time: 0.5334 s/iter data_time: 0.0406 s/iter total_throughput: 1919.78 samples/s lr: 3.37e-04 [09/19 05:21:21] lb.utils.events INFO: eta: 2 days, 6:16:47 iteration: 8599/375342 consumed_samples: 8806400 total_loss: 0.6547 time: 0.5334 s/iter data_time: 0.0361 s/iter total_throughput: 1919.84 samples/s lr: 3.41e-04 [09/19 05:22:14] lb.utils.events INFO: eta: 2 days, 6:16:16 iteration: 8699/375342 consumed_samples: 8908800 total_loss: 0.6555 time: 0.5334 s/iter data_time: 0.0366 s/iter total_throughput: 1919.91 samples/s lr: 3.45e-04 [09/19 05:23:07] lb.utils.events INFO: eta: 2 days, 6:15:13 iteration: 8799/375342 consumed_samples: 9011200 total_loss: 0.6588 time: 0.5334 s/iter data_time: 0.0384 s/iter total_throughput: 1919.93 samples/s lr: 3.48e-04 [09/19 05:24:00] lb.utils.events INFO: eta: 2 days, 6:14:07 iteration: 8899/375342 consumed_samples: 9113600 total_loss: 0.6574 time: 0.5333 s/iter data_time: 0.0397 s/iter total_throughput: 1919.99 samples/s lr: 3.52e-04 [09/19 05:24:54] lb.utils.events INFO: eta: 2 days, 6:12:56 iteration: 8999/375342 consumed_samples: 9216000 total_loss: 0.6543 time: 0.5333 s/iter data_time: 0.0394 s/iter total_throughput: 1920.04 samples/s lr: 3.56e-04 [09/19 05:25:47] lb.utils.events INFO: eta: 2 days, 6:12:44 iteration: 9099/375342 consumed_samples: 9318400 total_loss: 0.6529 time: 0.5333 s/iter data_time: 0.0351 s/iter total_throughput: 1920.06 samples/s lr: 3.60e-04 [09/19 05:26:40] lb.utils.events INFO: eta: 2 days, 6:11:55 iteration: 9199/375342 consumed_samples: 9420800 total_loss: 0.6538 time: 0.5333 s/iter data_time: 0.0408 s/iter total_throughput: 1920.08 samples/s lr: 3.64e-04 [09/19 05:27:33] lb.utils.events INFO: eta: 2 days, 6:10:35 iteration: 9299/375342 consumed_samples: 9523200 total_loss: 0.6546 time: 0.5333 s/iter data_time: 0.0389 s/iter total_throughput: 1920.16 samples/s lr: 3.68e-04 [09/19 05:28:26] lb.utils.events INFO: eta: 2 days, 6:09:48 iteration: 9399/375342 consumed_samples: 9625600 total_loss: 0.6495 time: 0.5333 s/iter data_time: 0.0359 s/iter total_throughput: 1920.24 samples/s lr: 3.72e-04 [09/19 05:29:20] lb.utils.events INFO: eta: 2 days, 6:08:47 iteration: 9499/375342 consumed_samples: 9728000 total_loss: 0.6485 time: 0.5333 s/iter data_time: 0.0361 s/iter total_throughput: 1920.29 samples/s lr: 3.76e-04 [09/19 05:30:13] lb.utils.events INFO: eta: 2 days, 6:06:49 iteration: 9599/375342 consumed_samples: 9830400 total_loss: 0.6499 time: 0.5332 s/iter data_time: 0.0378 s/iter total_throughput: 1920.36 samples/s lr: 3.80e-04 [09/19 05:31:06] lb.utils.events INFO: eta: 2 days, 6:05:48 iteration: 9699/375342 consumed_samples: 9932800 total_loss: 0.6481 time: 0.5332 s/iter data_time: 0.0403 s/iter total_throughput: 1920.43 samples/s lr: 3.84e-04 [09/19 05:31:59] lb.utils.events INFO: eta: 2 days, 6:03:13 iteration: 9799/375342 consumed_samples: 10035200 total_loss: 0.6456 time: 0.5332 s/iter data_time: 0.0383 s/iter total_throughput: 1920.52 samples/s lr: 3.88e-04 [09/19 05:32:52] lb.utils.events INFO: eta: 2 days, 6:01:45 iteration: 9899/375342 consumed_samples: 10137600 total_loss: 0.6438 time: 0.5332 s/iter data_time: 0.0387 s/iter total_throughput: 1920.57 samples/s lr: 3.92e-04 [09/19 05:33:45] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0009999 [09/19 05:33:46] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 05:33:46] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 05:33:51] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0719 s/iter. Inference: 0.2411 s/iter. Eval: 0.0024 s/iter. Total: 0.3154 s/iter. ETA=0:00:11 [09/19 05:33:56] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0901 s/iter. Inference: 0.2448 s/iter. Eval: 0.0026 s/iter. Total: 0.3378 s/iter. ETA=0:00:07 [09/19 05:34:01] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0851 s/iter. Inference: 0.2445 s/iter. Eval: 0.0027 s/iter. Total: 0.3326 s/iter. ETA=0:00:01 [09/19 05:34:03] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 05:34:03] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.285173 (0.000286 s / iter per device, on 8 devices) [09/19 05:34:03] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000215 s / iter per device, on 8 devices) [09/19 05:34:03] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 05:34:03] lb.evaluation.utils INFO: copypaste: Acc@1=33.464 [09/19 05:34:03] lb.evaluation.utils INFO: copypaste: Acc@5=58.355999999999995 [09/19 05:34:03] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 33.46400, better than last best score 12.77800 @ iteration 4999. [09/19 05:34:03] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 05:34:04] lb.utils.events INFO: eta: 2 days, 6:00:07 iteration: 9999/375342 consumed_samples: 10240000 total_loss: 0.6407 time: 0.5332 s/iter data_time: 0.0356 s/iter total_throughput: 1920.62 samples/s lr: 3.96e-04 [09/19 05:34:57] lb.utils.events INFO: eta: 2 days, 5:57:52 iteration: 10099/375342 consumed_samples: 10342400 total_loss: 0.6378 time: 0.5331 s/iter data_time: 0.0365 s/iter total_throughput: 1920.72 samples/s lr: 4.00e-04 [09/19 05:35:50] lb.utils.events INFO: eta: 2 days, 5:54:58 iteration: 10199/375342 consumed_samples: 10444800 total_loss: 0.6367 time: 0.5331 s/iter data_time: 0.0361 s/iter total_throughput: 1920.83 samples/s lr: 4.04e-04 [09/19 05:36:43] lb.utils.events INFO: eta: 2 days, 5:52:27 iteration: 10299/375342 consumed_samples: 10547200 total_loss: 0.636 time: 0.5331 s/iter data_time: 0.0375 s/iter total_throughput: 1920.94 samples/s lr: 4.08e-04 [09/19 05:37:36] lb.utils.events INFO: eta: 2 days, 5:50:39 iteration: 10399/375342 consumed_samples: 10649600 total_loss: 0.6327 time: 0.5331 s/iter data_time: 0.0392 s/iter total_throughput: 1920.95 samples/s lr: 4.12e-04 [09/19 05:38:29] lb.utils.events INFO: eta: 2 days, 5:48:52 iteration: 10499/375342 consumed_samples: 10752000 total_loss: 0.6321 time: 0.5330 s/iter data_time: 0.0325 s/iter total_throughput: 1921.09 samples/s lr: 4.16e-04 [09/19 05:39:22] lb.utils.events INFO: eta: 2 days, 5:47:05 iteration: 10599/375342 consumed_samples: 10854400 total_loss: 0.6282 time: 0.5330 s/iter data_time: 0.0353 s/iter total_throughput: 1921.22 samples/s lr: 4.20e-04 [09/19 05:40:15] lb.utils.events INFO: eta: 2 days, 5:45:13 iteration: 10699/375342 consumed_samples: 10956800 total_loss: 0.6243 time: 0.5330 s/iter data_time: 0.0375 s/iter total_throughput: 1921.32 samples/s lr: 4.24e-04 [09/19 05:41:08] lb.utils.events INFO: eta: 2 days, 5:44:22 iteration: 10799/375342 consumed_samples: 11059200 total_loss: 0.6275 time: 0.5329 s/iter data_time: 0.0389 s/iter total_throughput: 1921.40 samples/s lr: 4.27e-04 [09/19 05:42:01] lb.utils.events INFO: eta: 2 days, 5:42:30 iteration: 10899/375342 consumed_samples: 11161600 total_loss: 0.6332 time: 0.5329 s/iter data_time: 0.0364 s/iter total_throughput: 1921.51 samples/s lr: 4.31e-04 [09/19 05:42:54] lb.utils.events INFO: eta: 2 days, 5:40:26 iteration: 10999/375342 consumed_samples: 11264000 total_loss: 0.6295 time: 0.5329 s/iter data_time: 0.0357 s/iter total_throughput: 1921.60 samples/s lr: 4.35e-04 [09/19 05:43:47] lb.utils.events INFO: eta: 2 days, 5:39:33 iteration: 11099/375342 consumed_samples: 11366400 total_loss: 0.6307 time: 0.5329 s/iter data_time: 0.0356 s/iter total_throughput: 1921.69 samples/s lr: 4.39e-04 [09/19 05:44:40] lb.utils.events INFO: eta: 2 days, 5:38:40 iteration: 11199/375342 consumed_samples: 11468800 total_loss: 0.6247 time: 0.5328 s/iter data_time: 0.0311 s/iter total_throughput: 1921.76 samples/s lr: 4.43e-04 [09/19 05:45:33] lb.utils.events INFO: eta: 2 days, 5:37:56 iteration: 11299/375342 consumed_samples: 11571200 total_loss: 0.6162 time: 0.5328 s/iter data_time: 0.0347 s/iter total_throughput: 1921.85 samples/s lr: 4.47e-04 [09/19 05:46:26] lb.utils.events INFO: eta: 2 days, 5:38:33 iteration: 11399/375342 consumed_samples: 11673600 total_loss: 0.6228 time: 0.5328 s/iter data_time: 0.0343 s/iter total_throughput: 1921.95 samples/s lr: 4.51e-04 [09/19 05:47:19] lb.utils.events INFO: eta: 2 days, 5:39:37 iteration: 11499/375342 consumed_samples: 11776000 total_loss: 0.6228 time: 0.5328 s/iter data_time: 0.0356 s/iter total_throughput: 1921.99 samples/s lr: 4.55e-04 [09/19 05:48:13] lb.utils.events INFO: eta: 2 days, 5:39:28 iteration: 11599/375342 consumed_samples: 11878400 total_loss: 0.6231 time: 0.5328 s/iter data_time: 0.0345 s/iter total_throughput: 1922.04 samples/s lr: 4.59e-04 [09/19 05:49:06] lb.utils.events INFO: eta: 2 days, 5:39:59 iteration: 11699/375342 consumed_samples: 11980800 total_loss: 0.6184 time: 0.5328 s/iter data_time: 0.0322 s/iter total_throughput: 1922.07 samples/s lr: 4.63e-04 [09/19 05:49:59] lb.utils.events INFO: eta: 2 days, 5:39:42 iteration: 11799/375342 consumed_samples: 12083200 total_loss: 0.6216 time: 0.5327 s/iter data_time: 0.0359 s/iter total_throughput: 1922.11 samples/s lr: 4.67e-04 [09/19 05:50:52] lb.utils.events INFO: eta: 2 days, 5:39:46 iteration: 11899/375342 consumed_samples: 12185600 total_loss: 0.6207 time: 0.5327 s/iter data_time: 0.0346 s/iter total_throughput: 1922.12 samples/s lr: 4.71e-04 [09/19 05:51:45] lb.utils.events INFO: eta: 2 days, 5:39:01 iteration: 11999/375342 consumed_samples: 12288000 total_loss: 0.6154 time: 0.5327 s/iter data_time: 0.0396 s/iter total_throughput: 1922.18 samples/s lr: 4.75e-04 [09/19 05:52:39] lb.utils.events INFO: eta: 2 days, 5:39:54 iteration: 12099/375342 consumed_samples: 12390400 total_loss: 0.6118 time: 0.5327 s/iter data_time: 0.0356 s/iter total_throughput: 1922.17 samples/s lr: 4.79e-04 [09/19 05:53:32] lb.utils.events INFO: eta: 2 days, 5:40:33 iteration: 12199/375342 consumed_samples: 12492800 total_loss: 0.6082 time: 0.5327 s/iter data_time: 0.0361 s/iter total_throughput: 1922.17 samples/s lr: 4.83e-04 [09/19 05:54:25] lb.utils.events INFO: eta: 2 days, 5:41:39 iteration: 12299/375342 consumed_samples: 12595200 total_loss: 0.6079 time: 0.5327 s/iter data_time: 0.0370 s/iter total_throughput: 1922.17 samples/s lr: 4.87e-04 [09/19 05:55:19] lb.utils.events INFO: eta: 2 days, 5:41:48 iteration: 12399/375342 consumed_samples: 12697600 total_loss: 0.61 time: 0.5327 s/iter data_time: 0.0382 s/iter total_throughput: 1922.17 samples/s lr: 4.91e-04 [09/19 05:56:12] lb.utils.events INFO: eta: 2 days, 5:41:14 iteration: 12499/375342 consumed_samples: 12800000 total_loss: 0.6102 time: 0.5327 s/iter data_time: 0.0349 s/iter total_throughput: 1922.18 samples/s lr: 4.95e-04 [09/19 05:57:05] lb.utils.events INFO: eta: 2 days, 5:40:09 iteration: 12599/375342 consumed_samples: 12902400 total_loss: 0.602 time: 0.5327 s/iter data_time: 0.0399 s/iter total_throughput: 1922.21 samples/s lr: 4.99e-04 [09/19 05:57:58] lb.utils.events INFO: eta: 2 days, 5:39:15 iteration: 12699/375342 consumed_samples: 13004800 total_loss: 0.6018 time: 0.5327 s/iter data_time: 0.0379 s/iter total_throughput: 1922.22 samples/s lr: 5.02e-04 [09/19 05:58:51] lb.utils.events INFO: eta: 2 days, 5:38:40 iteration: 12799/375342 consumed_samples: 13107200 total_loss: 0.6057 time: 0.5327 s/iter data_time: 0.0396 s/iter total_throughput: 1922.23 samples/s lr: 5.06e-04 [09/19 05:59:45] lb.utils.events INFO: eta: 2 days, 5:38:30 iteration: 12899/375342 consumed_samples: 13209600 total_loss: 0.6043 time: 0.5327 s/iter data_time: 0.0397 s/iter total_throughput: 1922.22 samples/s lr: 5.10e-04 [09/19 06:00:38] lb.utils.events INFO: eta: 2 days, 5:38:36 iteration: 12999/375342 consumed_samples: 13312000 total_loss: 0.5973 time: 0.5327 s/iter data_time: 0.0370 s/iter total_throughput: 1922.25 samples/s lr: 5.14e-04 [09/19 06:01:31] lb.utils.events INFO: eta: 2 days, 5:37:10 iteration: 13099/375342 consumed_samples: 13414400 total_loss: 0.5967 time: 0.5327 s/iter data_time: 0.0379 s/iter total_throughput: 1922.27 samples/s lr: 5.18e-04 [09/19 06:02:24] lb.utils.events INFO: eta: 2 days, 5:34:18 iteration: 13199/375342 consumed_samples: 13516800 total_loss: 0.6051 time: 0.5327 s/iter data_time: 0.0384 s/iter total_throughput: 1922.32 samples/s lr: 5.22e-04 [09/19 06:03:17] lb.utils.events INFO: eta: 2 days, 5:32:41 iteration: 13299/375342 consumed_samples: 13619200 total_loss: 0.6048 time: 0.5327 s/iter data_time: 0.0373 s/iter total_throughput: 1922.37 samples/s lr: 5.26e-04 [09/19 06:04:11] lb.utils.events INFO: eta: 2 days, 5:30:51 iteration: 13399/375342 consumed_samples: 13721600 total_loss: 0.6 time: 0.5327 s/iter data_time: 0.0375 s/iter total_throughput: 1922.38 samples/s lr: 5.30e-04 [09/19 06:05:04] lb.utils.events INFO: eta: 2 days, 5:29:29 iteration: 13499/375342 consumed_samples: 13824000 total_loss: 0.5933 time: 0.5327 s/iter data_time: 0.0369 s/iter total_throughput: 1922.38 samples/s lr: 5.34e-04 [09/19 06:05:57] lb.utils.events INFO: eta: 2 days, 5:28:31 iteration: 13599/375342 consumed_samples: 13926400 total_loss: 0.5932 time: 0.5327 s/iter data_time: 0.0364 s/iter total_throughput: 1922.40 samples/s lr: 5.38e-04 [09/19 06:06:50] lb.utils.events INFO: eta: 2 days, 5:26:47 iteration: 13699/375342 consumed_samples: 14028800 total_loss: 0.5947 time: 0.5327 s/iter data_time: 0.0396 s/iter total_throughput: 1922.42 samples/s lr: 5.42e-04 [09/19 06:07:44] lb.utils.events INFO: eta: 2 days, 5:25:28 iteration: 13799/375342 consumed_samples: 14131200 total_loss: 0.591 time: 0.5327 s/iter data_time: 0.0397 s/iter total_throughput: 1922.36 samples/s lr: 5.46e-04 [09/19 06:08:37] lb.utils.events INFO: eta: 2 days, 5:24:35 iteration: 13899/375342 consumed_samples: 14233600 total_loss: 0.5909 time: 0.5327 s/iter data_time: 0.0353 s/iter total_throughput: 1922.36 samples/s lr: 5.50e-04 [09/19 06:09:30] lb.utils.events INFO: eta: 2 days, 5:23:04 iteration: 13999/375342 consumed_samples: 14336000 total_loss: 0.5879 time: 0.5327 s/iter data_time: 0.0324 s/iter total_throughput: 1922.41 samples/s lr: 5.54e-04 [09/19 06:10:23] lb.utils.events INFO: eta: 2 days, 5:22:27 iteration: 14099/375342 consumed_samples: 14438400 total_loss: 0.5886 time: 0.5327 s/iter data_time: 0.0309 s/iter total_throughput: 1922.44 samples/s lr: 5.58e-04 [09/19 06:11:17] lb.utils.events INFO: eta: 2 days, 5:22:04 iteration: 14199/375342 consumed_samples: 14540800 total_loss: 0.5847 time: 0.5327 s/iter data_time: 0.0320 s/iter total_throughput: 1922.46 samples/s lr: 5.62e-04 [09/19 06:12:10] lb.utils.events INFO: eta: 2 days, 5:21:06 iteration: 14299/375342 consumed_samples: 14643200 total_loss: 0.5836 time: 0.5326 s/iter data_time: 0.0357 s/iter total_throughput: 1922.50 samples/s lr: 5.66e-04 [09/19 06:13:03] lb.utils.events INFO: eta: 2 days, 5:19:47 iteration: 14399/375342 consumed_samples: 14745600 total_loss: 0.5852 time: 0.5326 s/iter data_time: 0.0366 s/iter total_throughput: 1922.54 samples/s lr: 5.70e-04 [09/19 06:13:56] lb.utils.events INFO: eta: 2 days, 5:18:00 iteration: 14499/375342 consumed_samples: 14848000 total_loss: 0.5801 time: 0.5326 s/iter data_time: 0.0337 s/iter total_throughput: 1922.61 samples/s lr: 5.74e-04 [09/19 06:14:49] lb.utils.events INFO: eta: 2 days, 5:17:00 iteration: 14599/375342 consumed_samples: 14950400 total_loss: 0.5924 time: 0.5326 s/iter data_time: 0.0297 s/iter total_throughput: 1922.66 samples/s lr: 5.78e-04 [09/19 06:15:42] lb.utils.events INFO: eta: 2 days, 5:15:47 iteration: 14699/375342 consumed_samples: 15052800 total_loss: 0.5951 time: 0.5326 s/iter data_time: 0.0301 s/iter total_throughput: 1922.72 samples/s lr: 5.81e-04 [09/19 06:16:35] lb.utils.events INFO: eta: 2 days, 5:13:54 iteration: 14799/375342 consumed_samples: 15155200 total_loss: 0.5883 time: 0.5326 s/iter data_time: 0.0265 s/iter total_throughput: 1922.77 samples/s lr: 5.85e-04 [09/19 06:17:28] lb.utils.events INFO: eta: 2 days, 5:10:50 iteration: 14899/375342 consumed_samples: 15257600 total_loss: 0.5827 time: 0.5325 s/iter data_time: 0.0284 s/iter total_throughput: 1922.84 samples/s lr: 5.89e-04 [09/19 06:18:21] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0014999 [09/19 06:18:22] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 06:18:22] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 06:18:27] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0555 s/iter. Inference: 0.2481 s/iter. Eval: 0.0023 s/iter. Total: 0.3059 s/iter. ETA=0:00:11 [09/19 06:18:32] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0641 s/iter. Inference: 0.2619 s/iter. Eval: 0.0022 s/iter. Total: 0.3286 s/iter. ETA=0:00:07 [09/19 06:18:37] lb.evaluation.evaluator INFO: Inference done 41984/50000. Dataloading: 0.0727 s/iter. Inference: 0.2564 s/iter. Eval: 0.0022 s/iter. Total: 0.3316 s/iter. ETA=0:00:02 [09/19 06:18:39] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 06:18:39] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.307926 (0.000286 s / iter per device, on 8 devices) [09/19 06:18:39] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/19 06:18:39] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 06:18:39] lb.evaluation.utils INFO: copypaste: Acc@1=46.016 [09/19 06:18:39] lb.evaluation.utils INFO: copypaste: Acc@5=71.452 [09/19 06:18:39] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 46.01600, better than last best score 33.46400 @ iteration 9999. [09/19 06:18:39] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 06:18:40] lb.utils.events INFO: eta: 2 days, 5:09:38 iteration: 14999/375342 consumed_samples: 15360000 total_loss: 0.5893 time: 0.5325 s/iter data_time: 0.0305 s/iter total_throughput: 1922.83 samples/s lr: 5.93e-04 [09/19 06:19:33] lb.utils.events INFO: eta: 2 days, 5:09:31 iteration: 15099/375342 consumed_samples: 15462400 total_loss: 0.5881 time: 0.5326 s/iter data_time: 0.0336 s/iter total_throughput: 1922.80 samples/s lr: 5.97e-04 [09/19 06:20:27] lb.utils.events INFO: eta: 2 days, 5:09:31 iteration: 15199/375342 consumed_samples: 15564800 total_loss: 0.5791 time: 0.5326 s/iter data_time: 0.0324 s/iter total_throughput: 1922.75 samples/s lr: 6.01e-04 [09/19 06:21:20] lb.utils.events INFO: eta: 2 days, 5:11:23 iteration: 15299/375342 consumed_samples: 15667200 total_loss: 0.5807 time: 0.5326 s/iter data_time: 0.0301 s/iter total_throughput: 1922.70 samples/s lr: 6.05e-04 [09/19 06:22:13] lb.utils.events INFO: eta: 2 days, 5:11:12 iteration: 15399/375342 consumed_samples: 15769600 total_loss: 0.5788 time: 0.5326 s/iter data_time: 0.0310 s/iter total_throughput: 1922.71 samples/s lr: 6.09e-04 [09/19 06:23:07] lb.utils.events INFO: eta: 2 days, 5:13:33 iteration: 15499/375342 consumed_samples: 15872000 total_loss: 0.5784 time: 0.5326 s/iter data_time: 0.0346 s/iter total_throughput: 1922.69 samples/s lr: 6.13e-04 [09/19 06:24:00] lb.utils.events INFO: eta: 2 days, 5:14:27 iteration: 15599/375342 consumed_samples: 15974400 total_loss: 0.5773 time: 0.5326 s/iter data_time: 0.0349 s/iter total_throughput: 1922.65 samples/s lr: 6.17e-04 [09/19 06:24:54] lb.utils.events INFO: eta: 2 days, 5:15:26 iteration: 15699/375342 consumed_samples: 16076800 total_loss: 0.5745 time: 0.5326 s/iter data_time: 0.0371 s/iter total_throughput: 1922.63 samples/s lr: 6.21e-04 [09/19 06:25:47] lb.utils.events INFO: eta: 2 days, 5:17:52 iteration: 15799/375342 consumed_samples: 16179200 total_loss: 0.575 time: 0.5326 s/iter data_time: 0.0384 s/iter total_throughput: 1922.57 samples/s lr: 6.25e-04 [09/19 06:26:41] lb.utils.events INFO: eta: 2 days, 5:19:40 iteration: 15899/375342 consumed_samples: 16281600 total_loss: 0.5767 time: 0.5326 s/iter data_time: 0.0390 s/iter total_throughput: 1922.51 samples/s lr: 6.29e-04 [09/19 06:27:34] lb.utils.events INFO: eta: 2 days, 5:20:30 iteration: 15999/375342 consumed_samples: 16384000 total_loss: 0.5798 time: 0.5327 s/iter data_time: 0.0401 s/iter total_throughput: 1922.46 samples/s lr: 6.33e-04 [09/19 06:28:28] lb.utils.events INFO: eta: 2 days, 5:20:54 iteration: 16099/375342 consumed_samples: 16486400 total_loss: 0.575 time: 0.5327 s/iter data_time: 0.0385 s/iter total_throughput: 1922.43 samples/s lr: 6.37e-04 [09/19 06:29:21] lb.utils.events INFO: eta: 2 days, 5:20:51 iteration: 16199/375342 consumed_samples: 16588800 total_loss: 0.5822 time: 0.5327 s/iter data_time: 0.0410 s/iter total_throughput: 1922.38 samples/s lr: 6.41e-04 [09/19 06:30:15] lb.utils.events INFO: eta: 2 days, 5:19:14 iteration: 16299/375342 consumed_samples: 16691200 total_loss: 0.581 time: 0.5327 s/iter data_time: 0.0421 s/iter total_throughput: 1922.33 samples/s lr: 6.45e-04 [09/19 06:31:08] lb.utils.events INFO: eta: 2 days, 5:19:49 iteration: 16399/375342 consumed_samples: 16793600 total_loss: 0.5714 time: 0.5327 s/iter data_time: 0.0384 s/iter total_throughput: 1922.29 samples/s lr: 6.49e-04 [09/19 06:32:01] lb.utils.events INFO: eta: 2 days, 5:19:00 iteration: 16499/375342 consumed_samples: 16896000 total_loss: 0.562 time: 0.5327 s/iter data_time: 0.0397 s/iter total_throughput: 1922.27 samples/s lr: 6.53e-04 [09/19 06:32:55] lb.utils.events INFO: eta: 2 days, 5:18:36 iteration: 16599/375342 consumed_samples: 16998400 total_loss: 0.5715 time: 0.5327 s/iter data_time: 0.0402 s/iter total_throughput: 1922.23 samples/s lr: 6.57e-04 [09/19 06:33:48] lb.utils.events INFO: eta: 2 days, 5:17:47 iteration: 16699/375342 consumed_samples: 17100800 total_loss: 0.5737 time: 0.5327 s/iter data_time: 0.0403 s/iter total_throughput: 1922.21 samples/s lr: 6.60e-04 [09/19 06:34:42] lb.utils.events INFO: eta: 2 days, 5:16:07 iteration: 16799/375342 consumed_samples: 17203200 total_loss: 0.5667 time: 0.5327 s/iter data_time: 0.0403 s/iter total_throughput: 1922.19 samples/s lr: 6.64e-04 [09/19 06:35:35] lb.utils.events INFO: eta: 2 days, 5:13:59 iteration: 16899/375342 consumed_samples: 17305600 total_loss: 0.5649 time: 0.5327 s/iter data_time: 0.0383 s/iter total_throughput: 1922.17 samples/s lr: 6.68e-04 [09/19 06:36:28] lb.utils.events INFO: eta: 2 days, 5:11:29 iteration: 16999/375342 consumed_samples: 17408000 total_loss: 0.5628 time: 0.5327 s/iter data_time: 0.0389 s/iter total_throughput: 1922.17 samples/s lr: 6.72e-04 [09/19 06:37:22] lb.utils.events INFO: eta: 2 days, 5:09:59 iteration: 17099/375342 consumed_samples: 17510400 total_loss: 0.5579 time: 0.5327 s/iter data_time: 0.0387 s/iter total_throughput: 1922.16 samples/s lr: 6.76e-04 [09/19 06:38:15] lb.utils.events INFO: eta: 2 days, 5:06:45 iteration: 17199/375342 consumed_samples: 17612800 total_loss: 0.558 time: 0.5327 s/iter data_time: 0.0404 s/iter total_throughput: 1922.17 samples/s lr: 6.80e-04 [09/19 06:39:08] lb.utils.events INFO: eta: 2 days, 5:05:02 iteration: 17299/375342 consumed_samples: 17715200 total_loss: 0.5652 time: 0.5327 s/iter data_time: 0.0411 s/iter total_throughput: 1922.12 samples/s lr: 6.84e-04 [09/19 06:40:02] lb.utils.events INFO: eta: 2 days, 5:04:16 iteration: 17399/375342 consumed_samples: 17817600 total_loss: 0.5707 time: 0.5327 s/iter data_time: 0.0388 s/iter total_throughput: 1922.11 samples/s lr: 6.88e-04 [09/19 06:40:55] lb.utils.events INFO: eta: 2 days, 5:03:23 iteration: 17499/375342 consumed_samples: 17920000 total_loss: 0.5674 time: 0.5328 s/iter data_time: 0.0379 s/iter total_throughput: 1922.09 samples/s lr: 6.92e-04 [09/19 06:41:49] lb.utils.events INFO: eta: 2 days, 5:02:07 iteration: 17599/375342 consumed_samples: 18022400 total_loss: 0.5635 time: 0.5328 s/iter data_time: 0.0379 s/iter total_throughput: 1922.07 samples/s lr: 6.96e-04 [09/19 06:42:42] lb.utils.events INFO: eta: 2 days, 5:01:07 iteration: 17699/375342 consumed_samples: 18124800 total_loss: 0.5696 time: 0.5328 s/iter data_time: 0.0392 s/iter total_throughput: 1922.07 samples/s lr: 7.00e-04 [09/19 06:43:35] lb.utils.events INFO: eta: 2 days, 5:01:09 iteration: 17799/375342 consumed_samples: 18227200 total_loss: 0.5608 time: 0.5328 s/iter data_time: 0.0409 s/iter total_throughput: 1922.07 samples/s lr: 7.04e-04 [09/19 06:44:29] lb.utils.events INFO: eta: 2 days, 5:01:18 iteration: 17899/375342 consumed_samples: 18329600 total_loss: 0.5543 time: 0.5328 s/iter data_time: 0.0408 s/iter total_throughput: 1922.05 samples/s lr: 7.08e-04 [09/19 06:45:22] lb.utils.events INFO: eta: 2 days, 5:02:34 iteration: 17999/375342 consumed_samples: 18432000 total_loss: 0.5564 time: 0.5328 s/iter data_time: 0.0362 s/iter total_throughput: 1922.03 samples/s lr: 7.12e-04 [09/19 06:46:15] lb.utils.events INFO: eta: 2 days, 5:02:14 iteration: 18099/375342 consumed_samples: 18534400 total_loss: 0.5476 time: 0.5328 s/iter data_time: 0.0367 s/iter total_throughput: 1922.02 samples/s lr: 7.16e-04 [09/19 06:47:09] lb.utils.events INFO: eta: 2 days, 5:03:18 iteration: 18199/375342 consumed_samples: 18636800 total_loss: 0.5496 time: 0.5328 s/iter data_time: 0.0412 s/iter total_throughput: 1922.00 samples/s lr: 7.20e-04 [09/19 06:48:02] lb.utils.events INFO: eta: 2 days, 5:03:14 iteration: 18299/375342 consumed_samples: 18739200 total_loss: 0.5546 time: 0.5328 s/iter data_time: 0.0358 s/iter total_throughput: 1921.98 samples/s lr: 7.24e-04 [09/19 06:48:55] lb.utils.events INFO: eta: 2 days, 5:01:46 iteration: 18399/375342 consumed_samples: 18841600 total_loss: 0.553 time: 0.5328 s/iter data_time: 0.0368 s/iter total_throughput: 1921.97 samples/s lr: 7.28e-04 [09/19 06:49:49] lb.utils.events INFO: eta: 2 days, 5:01:08 iteration: 18499/375342 consumed_samples: 18944000 total_loss: 0.5549 time: 0.5328 s/iter data_time: 0.0382 s/iter total_throughput: 1921.93 samples/s lr: 7.32e-04 [09/19 06:50:43] lb.utils.events INFO: eta: 2 days, 5:02:13 iteration: 18599/375342 consumed_samples: 19046400 total_loss: 0.556 time: 0.5328 s/iter data_time: 0.0403 s/iter total_throughput: 1921.86 samples/s lr: 7.35e-04 [09/19 06:51:36] lb.utils.events INFO: eta: 2 days, 5:02:24 iteration: 18699/375342 consumed_samples: 19148800 total_loss: 0.5516 time: 0.5328 s/iter data_time: 0.0389 s/iter total_throughput: 1921.80 samples/s lr: 7.39e-04 [09/19 06:52:30] lb.utils.events INFO: eta: 2 days, 5:02:04 iteration: 18799/375342 consumed_samples: 19251200 total_loss: 0.5505 time: 0.5328 s/iter data_time: 0.0396 s/iter total_throughput: 1921.74 samples/s lr: 7.43e-04 [09/19 06:53:24] lb.utils.events INFO: eta: 2 days, 5:02:04 iteration: 18899/375342 consumed_samples: 19353600 total_loss: 0.5505 time: 0.5329 s/iter data_time: 0.0408 s/iter total_throughput: 1921.67 samples/s lr: 7.47e-04 [09/19 06:54:17] lb.utils.events INFO: eta: 2 days, 5:01:47 iteration: 18999/375342 consumed_samples: 19456000 total_loss: 0.5555 time: 0.5329 s/iter data_time: 0.0367 s/iter total_throughput: 1921.64 samples/s lr: 7.51e-04 [09/19 06:55:11] lb.utils.events INFO: eta: 2 days, 5:02:07 iteration: 19099/375342 consumed_samples: 19558400 total_loss: 0.5552 time: 0.5329 s/iter data_time: 0.0393 s/iter total_throughput: 1921.59 samples/s lr: 7.55e-04 [09/19 06:56:04] lb.utils.events INFO: eta: 2 days, 5:01:36 iteration: 19199/375342 consumed_samples: 19660800 total_loss: 0.5509 time: 0.5329 s/iter data_time: 0.0396 s/iter total_throughput: 1921.54 samples/s lr: 7.59e-04 [09/19 06:56:58] lb.utils.events INFO: eta: 2 days, 5:00:57 iteration: 19299/375342 consumed_samples: 19763200 total_loss: 0.5474 time: 0.5329 s/iter data_time: 0.0405 s/iter total_throughput: 1921.50 samples/s lr: 7.63e-04 [09/19 06:57:51] lb.utils.events INFO: eta: 2 days, 5:00:09 iteration: 19399/375342 consumed_samples: 19865600 total_loss: 0.5524 time: 0.5329 s/iter data_time: 0.0411 s/iter total_throughput: 1921.45 samples/s lr: 7.67e-04 [09/19 06:58:45] lb.utils.events INFO: eta: 2 days, 4:58:48 iteration: 19499/375342 consumed_samples: 19968000 total_loss: 0.5517 time: 0.5329 s/iter data_time: 0.0412 s/iter total_throughput: 1921.42 samples/s lr: 7.71e-04 [09/19 06:59:38] lb.utils.events INFO: eta: 2 days, 4:57:21 iteration: 19599/375342 consumed_samples: 20070400 total_loss: 0.5467 time: 0.5329 s/iter data_time: 0.0396 s/iter total_throughput: 1921.38 samples/s lr: 7.75e-04 [09/19 07:00:32] lb.utils.events INFO: eta: 2 days, 4:56:21 iteration: 19699/375342 consumed_samples: 20172800 total_loss: 0.5546 time: 0.5330 s/iter data_time: 0.0415 s/iter total_throughput: 1921.35 samples/s lr: 7.79e-04 [09/19 07:01:25] lb.utils.events INFO: eta: 2 days, 4:54:48 iteration: 19799/375342 consumed_samples: 20275200 total_loss: 0.545 time: 0.5330 s/iter data_time: 0.0417 s/iter total_throughput: 1921.31 samples/s lr: 7.83e-04 [09/19 07:02:19] lb.utils.events INFO: eta: 2 days, 4:52:19 iteration: 19899/375342 consumed_samples: 20377600 total_loss: 0.5416 time: 0.5330 s/iter data_time: 0.0424 s/iter total_throughput: 1921.28 samples/s lr: 7.87e-04 [09/19 07:03:12] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0019999 [09/19 07:03:13] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 07:03:13] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 07:03:18] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0758 s/iter. Inference: 0.2535 s/iter. Eval: 0.0028 s/iter. Total: 0.3321 s/iter. ETA=0:00:12 [09/19 07:03:23] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0899 s/iter. Inference: 0.2505 s/iter. Eval: 0.0025 s/iter. Total: 0.3431 s/iter. ETA=0:00:07 [09/19 07:03:28] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0833 s/iter. Inference: 0.2504 s/iter. Eval: 0.0026 s/iter. Total: 0.3365 s/iter. ETA=0:00:02 [09/19 07:03:30] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 07:03:30] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.404860 (0.000288 s / iter per device, on 8 devices) [09/19 07:03:30] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000219 s / iter per device, on 8 devices) [09/19 07:03:30] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 07:03:30] lb.evaluation.utils INFO: copypaste: Acc@1=53.64600000000001 [09/19 07:03:30] lb.evaluation.utils INFO: copypaste: Acc@5=77.97800000000001 [09/19 07:03:30] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 53.64600, better than last best score 46.01600 @ iteration 14999. [09/19 07:03:30] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 07:03:31] lb.utils.events INFO: eta: 2 days, 4:49:50 iteration: 19999/375342 consumed_samples: 20480000 total_loss: 0.5515 time: 0.5330 s/iter data_time: 0.0410 s/iter total_throughput: 1921.26 samples/s lr: 7.91e-04 [09/19 07:04:24] lb.utils.events INFO: eta: 2 days, 4:47:40 iteration: 20099/375342 consumed_samples: 20582400 total_loss: 0.5474 time: 0.5330 s/iter data_time: 0.0403 s/iter total_throughput: 1921.25 samples/s lr: 7.95e-04 [09/19 07:05:17] lb.utils.events INFO: eta: 2 days, 4:45:02 iteration: 20199/375342 consumed_samples: 20684800 total_loss: 0.5434 time: 0.5330 s/iter data_time: 0.0406 s/iter total_throughput: 1921.24 samples/s lr: 7.99e-04 [09/19 07:06:11] lb.utils.events INFO: eta: 2 days, 4:43:11 iteration: 20299/375342 consumed_samples: 20787200 total_loss: 0.543 time: 0.5330 s/iter data_time: 0.0403 s/iter total_throughput: 1921.23 samples/s lr: 8.03e-04 [09/19 07:07:04] lb.utils.events INFO: eta: 2 days, 4:41:16 iteration: 20399/375342 consumed_samples: 20889600 total_loss: 0.5464 time: 0.5330 s/iter data_time: 0.0390 s/iter total_throughput: 1921.22 samples/s lr: 8.07e-04 [09/19 07:07:57] lb.utils.events INFO: eta: 2 days, 4:39:34 iteration: 20499/375342 consumed_samples: 20992000 total_loss: 0.5473 time: 0.5330 s/iter data_time: 0.0392 s/iter total_throughput: 1921.23 samples/s lr: 8.11e-04 [09/19 07:08:51] lb.utils.events INFO: eta: 2 days, 4:37:36 iteration: 20599/375342 consumed_samples: 21094400 total_loss: 0.5362 time: 0.5330 s/iter data_time: 0.0396 s/iter total_throughput: 1921.24 samples/s lr: 8.14e-04 [09/19 07:09:44] lb.utils.events INFO: eta: 2 days, 4:34:59 iteration: 20699/375342 consumed_samples: 21196800 total_loss: 0.5381 time: 0.5330 s/iter data_time: 0.0406 s/iter total_throughput: 1921.21 samples/s lr: 8.18e-04 [09/19 07:10:38] lb.utils.events INFO: eta: 2 days, 4:33:58 iteration: 20799/375342 consumed_samples: 21299200 total_loss: 0.5422 time: 0.5330 s/iter data_time: 0.0376 s/iter total_throughput: 1921.19 samples/s lr: 8.22e-04 [09/19 07:11:31] lb.utils.events INFO: eta: 2 days, 4:32:21 iteration: 20899/375342 consumed_samples: 21401600 total_loss: 0.5496 time: 0.5330 s/iter data_time: 0.0364 s/iter total_throughput: 1921.20 samples/s lr: 8.26e-04 [09/19 07:12:24] lb.utils.events INFO: eta: 2 days, 4:30:16 iteration: 20999/375342 consumed_samples: 21504000 total_loss: 0.5382 time: 0.5330 s/iter data_time: 0.0366 s/iter total_throughput: 1921.22 samples/s lr: 8.30e-04 [09/19 07:13:17] lb.utils.events INFO: eta: 2 days, 4:29:23 iteration: 21099/375342 consumed_samples: 21606400 total_loss: 0.5325 time: 0.5330 s/iter data_time: 0.0381 s/iter total_throughput: 1921.21 samples/s lr: 8.34e-04 [09/19 07:14:11] lb.utils.events INFO: eta: 2 days, 4:28:29 iteration: 21199/375342 consumed_samples: 21708800 total_loss: 0.5347 time: 0.5330 s/iter data_time: 0.0408 s/iter total_throughput: 1921.23 samples/s lr: 8.38e-04 [09/19 07:15:04] lb.utils.events INFO: eta: 2 days, 4:29:01 iteration: 21299/375342 consumed_samples: 21811200 total_loss: 0.5381 time: 0.5330 s/iter data_time: 0.0410 s/iter total_throughput: 1921.21 samples/s lr: 8.42e-04 [09/19 07:15:57] lb.utils.events INFO: eta: 2 days, 4:28:29 iteration: 21399/375342 consumed_samples: 21913600 total_loss: 0.5439 time: 0.5330 s/iter data_time: 0.0385 s/iter total_throughput: 1921.22 samples/s lr: 8.46e-04 [09/19 07:16:51] lb.utils.events INFO: eta: 2 days, 4:28:34 iteration: 21499/375342 consumed_samples: 22016000 total_loss: 0.5422 time: 0.5330 s/iter data_time: 0.0379 s/iter total_throughput: 1921.21 samples/s lr: 8.50e-04 [09/19 07:17:44] lb.utils.events INFO: eta: 2 days, 4:28:54 iteration: 21599/375342 consumed_samples: 22118400 total_loss: 0.5395 time: 0.5330 s/iter data_time: 0.0366 s/iter total_throughput: 1921.20 samples/s lr: 8.54e-04 [09/19 07:18:38] lb.utils.events INFO: eta: 2 days, 4:29:44 iteration: 21699/375342 consumed_samples: 22220800 total_loss: 0.5342 time: 0.5330 s/iter data_time: 0.0380 s/iter total_throughput: 1921.19 samples/s lr: 8.58e-04 [09/19 07:19:31] lb.utils.events INFO: eta: 2 days, 4:28:26 iteration: 21799/375342 consumed_samples: 22323200 total_loss: 0.532 time: 0.5330 s/iter data_time: 0.0396 s/iter total_throughput: 1921.17 samples/s lr: 8.62e-04 [09/19 07:20:24] lb.utils.events INFO: eta: 2 days, 4:27:28 iteration: 21899/375342 consumed_samples: 22425600 total_loss: 0.5312 time: 0.5330 s/iter data_time: 0.0331 s/iter total_throughput: 1921.17 samples/s lr: 8.66e-04 [09/19 07:21:18] lb.utils.events INFO: eta: 2 days, 4:28:20 iteration: 21999/375342 consumed_samples: 22528000 total_loss: 0.533 time: 0.5330 s/iter data_time: 0.0356 s/iter total_throughput: 1921.14 samples/s lr: 8.70e-04 [09/19 07:22:11] lb.utils.events INFO: eta: 2 days, 4:29:13 iteration: 22099/375342 consumed_samples: 22630400 total_loss: 0.5354 time: 0.5330 s/iter data_time: 0.0395 s/iter total_throughput: 1921.09 samples/s lr: 8.74e-04 [09/19 07:23:05] lb.utils.events INFO: eta: 2 days, 4:29:49 iteration: 22199/375342 consumed_samples: 22732800 total_loss: 0.5306 time: 0.5330 s/iter data_time: 0.0392 s/iter total_throughput: 1921.04 samples/s lr: 8.78e-04 [09/19 07:23:59] lb.utils.events INFO: eta: 2 days, 4:29:47 iteration: 22299/375342 consumed_samples: 22835200 total_loss: 0.5318 time: 0.5331 s/iter data_time: 0.0420 s/iter total_throughput: 1921.00 samples/s lr: 8.82e-04 [09/19 07:24:52] lb.utils.events INFO: eta: 2 days, 4:29:45 iteration: 22399/375342 consumed_samples: 22937600 total_loss: 0.5416 time: 0.5331 s/iter data_time: 0.0406 s/iter total_throughput: 1920.95 samples/s lr: 8.86e-04 [09/19 07:25:46] lb.utils.events INFO: eta: 2 days, 4:29:45 iteration: 22499/375342 consumed_samples: 23040000 total_loss: 0.5341 time: 0.5331 s/iter data_time: 0.0383 s/iter total_throughput: 1920.91 samples/s lr: 8.90e-04 [09/19 07:26:39] lb.utils.events INFO: eta: 2 days, 4:29:05 iteration: 22599/375342 consumed_samples: 23142400 total_loss: 0.5317 time: 0.5331 s/iter data_time: 0.0340 s/iter total_throughput: 1920.88 samples/s lr: 8.93e-04 [09/19 07:27:33] lb.utils.events INFO: eta: 2 days, 4:28:25 iteration: 22699/375342 consumed_samples: 23244800 total_loss: 0.5358 time: 0.5331 s/iter data_time: 0.0374 s/iter total_throughput: 1920.83 samples/s lr: 8.97e-04 [09/19 07:28:27] lb.utils.events INFO: eta: 2 days, 4:29:06 iteration: 22799/375342 consumed_samples: 23347200 total_loss: 0.5352 time: 0.5331 s/iter data_time: 0.0367 s/iter total_throughput: 1920.79 samples/s lr: 9.01e-04 [09/19 07:29:20] lb.utils.events INFO: eta: 2 days, 4:29:04 iteration: 22899/375342 consumed_samples: 23449600 total_loss: 0.5294 time: 0.5331 s/iter data_time: 0.0395 s/iter total_throughput: 1920.75 samples/s lr: 9.05e-04 [09/19 07:30:14] lb.utils.events INFO: eta: 2 days, 4:28:16 iteration: 22999/375342 consumed_samples: 23552000 total_loss: 0.5336 time: 0.5331 s/iter data_time: 0.0401 s/iter total_throughput: 1920.70 samples/s lr: 9.09e-04 [09/19 07:31:07] lb.utils.events INFO: eta: 2 days, 4:26:48 iteration: 23099/375342 consumed_samples: 23654400 total_loss: 0.5263 time: 0.5331 s/iter data_time: 0.0385 s/iter total_throughput: 1920.68 samples/s lr: 9.13e-04 [09/19 07:32:01] lb.utils.events INFO: eta: 2 days, 4:25:30 iteration: 23199/375342 consumed_samples: 23756800 total_loss: 0.5321 time: 0.5332 s/iter data_time: 0.0419 s/iter total_throughput: 1920.65 samples/s lr: 9.17e-04 [09/19 07:32:54] lb.utils.events INFO: eta: 2 days, 4:23:51 iteration: 23299/375342 consumed_samples: 23859200 total_loss: 0.5407 time: 0.5332 s/iter data_time: 0.0409 s/iter total_throughput: 1920.61 samples/s lr: 9.21e-04 [09/19 07:33:48] lb.utils.events INFO: eta: 2 days, 4:22:16 iteration: 23399/375342 consumed_samples: 23961600 total_loss: 0.5365 time: 0.5332 s/iter data_time: 0.0407 s/iter total_throughput: 1920.59 samples/s lr: 9.25e-04 [09/19 07:34:41] lb.utils.events INFO: eta: 2 days, 4:20:52 iteration: 23499/375342 consumed_samples: 24064000 total_loss: 0.5317 time: 0.5332 s/iter data_time: 0.0400 s/iter total_throughput: 1920.57 samples/s lr: 9.29e-04 [09/19 07:35:35] lb.utils.events INFO: eta: 2 days, 4:19:10 iteration: 23599/375342 consumed_samples: 24166400 total_loss: 0.5266 time: 0.5332 s/iter data_time: 0.0403 s/iter total_throughput: 1920.56 samples/s lr: 9.33e-04 [09/19 07:36:28] lb.utils.events INFO: eta: 2 days, 4:17:31 iteration: 23699/375342 consumed_samples: 24268800 total_loss: 0.5235 time: 0.5332 s/iter data_time: 0.0390 s/iter total_throughput: 1920.54 samples/s lr: 9.37e-04 [09/19 07:37:22] lb.utils.events INFO: eta: 2 days, 4:15:14 iteration: 23799/375342 consumed_samples: 24371200 total_loss: 0.5275 time: 0.5332 s/iter data_time: 0.0378 s/iter total_throughput: 1920.53 samples/s lr: 9.41e-04 [09/19 07:38:15] lb.utils.events INFO: eta: 2 days, 4:12:54 iteration: 23899/375342 consumed_samples: 24473600 total_loss: 0.5215 time: 0.5332 s/iter data_time: 0.0403 s/iter total_throughput: 1920.53 samples/s lr: 9.45e-04 [09/19 07:39:08] lb.utils.events INFO: eta: 2 days, 4:09:04 iteration: 23999/375342 consumed_samples: 24576000 total_loss: 0.5261 time: 0.5332 s/iter data_time: 0.0404 s/iter total_throughput: 1920.53 samples/s lr: 9.49e-04 [09/19 07:40:02] lb.utils.events INFO: eta: 2 days, 4:07:35 iteration: 24099/375342 consumed_samples: 24678400 total_loss: 0.532 time: 0.5332 s/iter data_time: 0.0395 s/iter total_throughput: 1920.54 samples/s lr: 9.53e-04 [09/19 07:40:55] lb.utils.events INFO: eta: 2 days, 4:06:24 iteration: 24199/375342 consumed_samples: 24780800 total_loss: 0.5192 time: 0.5332 s/iter data_time: 0.0385 s/iter total_throughput: 1920.50 samples/s lr: 9.57e-04 [09/19 07:41:49] lb.utils.events INFO: eta: 2 days, 4:05:12 iteration: 24299/375342 consumed_samples: 24883200 total_loss: 0.5231 time: 0.5332 s/iter data_time: 0.0406 s/iter total_throughput: 1920.49 samples/s lr: 9.61e-04 [09/19 07:42:42] lb.utils.events INFO: eta: 2 days, 4:04:15 iteration: 24399/375342 consumed_samples: 24985600 total_loss: 0.52 time: 0.5332 s/iter data_time: 0.0392 s/iter total_throughput: 1920.49 samples/s lr: 9.65e-04 [09/19 07:43:35] lb.utils.events INFO: eta: 2 days, 4:03:05 iteration: 24499/375342 consumed_samples: 25088000 total_loss: 0.5131 time: 0.5332 s/iter data_time: 0.0388 s/iter total_throughput: 1920.50 samples/s lr: 9.68e-04 [09/19 07:44:29] lb.utils.events INFO: eta: 2 days, 4:02:37 iteration: 24599/375342 consumed_samples: 25190400 total_loss: 0.5262 time: 0.5332 s/iter data_time: 0.0366 s/iter total_throughput: 1920.48 samples/s lr: 9.72e-04 [09/19 07:45:22] lb.utils.events INFO: eta: 2 days, 4:01:54 iteration: 24699/375342 consumed_samples: 25292800 total_loss: 0.5268 time: 0.5332 s/iter data_time: 0.0411 s/iter total_throughput: 1920.47 samples/s lr: 9.76e-04 [09/19 07:46:16] lb.utils.events INFO: eta: 2 days, 4:02:54 iteration: 24799/375342 consumed_samples: 25395200 total_loss: 0.5193 time: 0.5332 s/iter data_time: 0.0387 s/iter total_throughput: 1920.45 samples/s lr: 9.80e-04 [09/19 07:47:09] lb.utils.events INFO: eta: 2 days, 4:02:45 iteration: 24899/375342 consumed_samples: 25497600 total_loss: 0.5124 time: 0.5332 s/iter data_time: 0.0401 s/iter total_throughput: 1920.45 samples/s lr: 9.84e-04 [09/19 07:48:02] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0024999 [09/19 07:48:03] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 07:48:03] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 07:48:08] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0647 s/iter. Inference: 0.2549 s/iter. Eval: 0.0022 s/iter. Total: 0.3217 s/iter. ETA=0:00:11 [09/19 07:48:13] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0766 s/iter. Inference: 0.2558 s/iter. Eval: 0.0024 s/iter. Total: 0.3351 s/iter. ETA=0:00:07 [09/19 07:48:18] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0738 s/iter. Inference: 0.2521 s/iter. Eval: 0.0024 s/iter. Total: 0.3287 s/iter. ETA=0:00:01 [09/19 07:48:20] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 07:48:20] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.154127 (0.000283 s / iter per device, on 8 devices) [09/19 07:48:20] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000221 s / iter per device, on 8 devices) [09/19 07:48:20] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 07:48:20] lb.evaluation.utils INFO: copypaste: Acc@1=57.958 [09/19 07:48:20] lb.evaluation.utils INFO: copypaste: Acc@5=81.552 [09/19 07:48:20] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 57.95800, better than last best score 53.64600 @ iteration 19999. [09/19 07:48:20] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 07:48:21] lb.utils.events INFO: eta: 2 days, 4:03:07 iteration: 24999/375342 consumed_samples: 25600000 total_loss: 0.5276 time: 0.5332 s/iter data_time: 0.0347 s/iter total_throughput: 1920.46 samples/s lr: 9.88e-04 [09/19 07:49:14] lb.utils.events INFO: eta: 2 days, 4:03:42 iteration: 25099/375342 consumed_samples: 25702400 total_loss: 0.5284 time: 0.5332 s/iter data_time: 0.0373 s/iter total_throughput: 1920.44 samples/s lr: 9.89e-04 [09/19 07:50:08] lb.utils.events INFO: eta: 2 days, 4:05:26 iteration: 25199/375342 consumed_samples: 25804800 total_loss: 0.5274 time: 0.5332 s/iter data_time: 0.0414 s/iter total_throughput: 1920.41 samples/s lr: 9.89e-04 [09/19 07:51:01] lb.utils.events INFO: eta: 2 days, 4:05:17 iteration: 25299/375342 consumed_samples: 25907200 total_loss: 0.5268 time: 0.5332 s/iter data_time: 0.0397 s/iter total_throughput: 1920.38 samples/s lr: 9.89e-04 [09/19 07:51:55] lb.utils.events INFO: eta: 2 days, 4:04:42 iteration: 25399/375342 consumed_samples: 26009600 total_loss: 0.5283 time: 0.5332 s/iter data_time: 0.0379 s/iter total_throughput: 1920.37 samples/s lr: 9.89e-04 [09/19 07:52:48] lb.utils.events INFO: eta: 2 days, 4:04:42 iteration: 25499/375342 consumed_samples: 26112000 total_loss: 0.524 time: 0.5332 s/iter data_time: 0.0420 s/iter total_throughput: 1920.34 samples/s lr: 9.89e-04 [09/19 07:53:42] lb.utils.events INFO: eta: 2 days, 4:04:29 iteration: 25599/375342 consumed_samples: 26214400 total_loss: 0.516 time: 0.5333 s/iter data_time: 0.0397 s/iter total_throughput: 1920.29 samples/s lr: 9.89e-04 [09/19 07:54:35] lb.utils.events INFO: eta: 2 days, 4:03:49 iteration: 25699/375342 consumed_samples: 26316800 total_loss: 0.5184 time: 0.5333 s/iter data_time: 0.0400 s/iter total_throughput: 1920.25 samples/s lr: 9.89e-04 [09/19 07:55:29] lb.utils.events INFO: eta: 2 days, 4:03:08 iteration: 25799/375342 consumed_samples: 26419200 total_loss: 0.5189 time: 0.5333 s/iter data_time: 0.0405 s/iter total_throughput: 1920.20 samples/s lr: 9.89e-04 [09/19 07:56:23] lb.utils.events INFO: eta: 2 days, 4:03:02 iteration: 25899/375342 consumed_samples: 26521600 total_loss: 0.5128 time: 0.5333 s/iter data_time: 0.0396 s/iter total_throughput: 1920.16 samples/s lr: 9.88e-04 [09/19 07:57:17] lb.utils.events INFO: eta: 2 days, 4:04:52 iteration: 25999/375342 consumed_samples: 26624000 total_loss: 0.5178 time: 0.5333 s/iter data_time: 0.0404 s/iter total_throughput: 1920.11 samples/s lr: 9.88e-04 [09/19 07:58:10] lb.utils.events INFO: eta: 2 days, 4:05:04 iteration: 26099/375342 consumed_samples: 26726400 total_loss: 0.5211 time: 0.5333 s/iter data_time: 0.0397 s/iter total_throughput: 1920.07 samples/s lr: 9.88e-04 [09/19 07:59:04] lb.utils.events INFO: eta: 2 days, 4:03:36 iteration: 26199/375342 consumed_samples: 26828800 total_loss: 0.5233 time: 0.5333 s/iter data_time: 0.0398 s/iter total_throughput: 1920.04 samples/s lr: 9.88e-04 [09/19 07:59:57] lb.utils.events INFO: eta: 2 days, 4:03:03 iteration: 26299/375342 consumed_samples: 26931200 total_loss: 0.5157 time: 0.5333 s/iter data_time: 0.0403 s/iter total_throughput: 1919.99 samples/s lr: 9.88e-04 [09/19 08:00:51] lb.utils.events INFO: eta: 2 days, 4:02:19 iteration: 26399/375342 consumed_samples: 27033600 total_loss: 0.5086 time: 0.5333 s/iter data_time: 0.0404 s/iter total_throughput: 1919.96 samples/s lr: 9.88e-04 [09/19 08:01:45] lb.utils.events INFO: eta: 2 days, 4:01:15 iteration: 26499/375342 consumed_samples: 27136000 total_loss: 0.5075 time: 0.5334 s/iter data_time: 0.0410 s/iter total_throughput: 1919.93 samples/s lr: 9.88e-04 [09/19 08:02:38] lb.utils.events INFO: eta: 2 days, 3:58:57 iteration: 26599/375342 consumed_samples: 27238400 total_loss: 0.5097 time: 0.5334 s/iter data_time: 0.0415 s/iter total_throughput: 1919.90 samples/s lr: 9.88e-04 [09/19 08:03:32] lb.utils.events INFO: eta: 2 days, 3:58:19 iteration: 26699/375342 consumed_samples: 27340800 total_loss: 0.5162 time: 0.5334 s/iter data_time: 0.0419 s/iter total_throughput: 1919.85 samples/s lr: 9.88e-04 [09/19 08:04:26] lb.utils.events INFO: eta: 2 days, 3:57:19 iteration: 26799/375342 consumed_samples: 27443200 total_loss: 0.5204 time: 0.5334 s/iter data_time: 0.0423 s/iter total_throughput: 1919.81 samples/s lr: 9.88e-04 [09/19 08:05:19] lb.utils.events INFO: eta: 2 days, 3:55:48 iteration: 26899/375342 consumed_samples: 27545600 total_loss: 0.5171 time: 0.5334 s/iter data_time: 0.0433 s/iter total_throughput: 1919.78 samples/s lr: 9.88e-04 [09/19 08:06:13] lb.utils.events INFO: eta: 2 days, 3:53:51 iteration: 26999/375342 consumed_samples: 27648000 total_loss: 0.5154 time: 0.5334 s/iter data_time: 0.0432 s/iter total_throughput: 1919.75 samples/s lr: 9.87e-04 [09/19 08:07:06] lb.utils.events INFO: eta: 2 days, 3:52:53 iteration: 27099/375342 consumed_samples: 27750400 total_loss: 0.5161 time: 0.5334 s/iter data_time: 0.0413 s/iter total_throughput: 1919.72 samples/s lr: 9.87e-04 [09/19 08:08:00] lb.utils.events INFO: eta: 2 days, 3:51:16 iteration: 27199/375342 consumed_samples: 27852800 total_loss: 0.5145 time: 0.5334 s/iter data_time: 0.0405 s/iter total_throughput: 1919.69 samples/s lr: 9.87e-04 [09/19 08:08:54] lb.utils.events INFO: eta: 2 days, 3:49:43 iteration: 27299/375342 consumed_samples: 27955200 total_loss: 0.5116 time: 0.5334 s/iter data_time: 0.0417 s/iter total_throughput: 1919.66 samples/s lr: 9.87e-04 [09/19 08:09:47] lb.utils.events INFO: eta: 2 days, 3:48:44 iteration: 27399/375342 consumed_samples: 28057600 total_loss: 0.5152 time: 0.5334 s/iter data_time: 0.0401 s/iter total_throughput: 1919.63 samples/s lr: 9.87e-04 [09/19 08:10:41] lb.utils.events INFO: eta: 2 days, 3:47:49 iteration: 27499/375342 consumed_samples: 28160000 total_loss: 0.514 time: 0.5334 s/iter data_time: 0.0416 s/iter total_throughput: 1919.60 samples/s lr: 9.87e-04 [09/19 08:11:35] lb.utils.events INFO: eta: 2 days, 3:45:18 iteration: 27599/375342 consumed_samples: 28262400 total_loss: 0.5128 time: 0.5335 s/iter data_time: 0.0391 s/iter total_throughput: 1919.55 samples/s lr: 9.87e-04 [09/19 08:12:28] lb.utils.events INFO: eta: 2 days, 3:44:11 iteration: 27699/375342 consumed_samples: 28364800 total_loss: 0.5168 time: 0.5335 s/iter data_time: 0.0412 s/iter total_throughput: 1919.53 samples/s lr: 9.87e-04 [09/19 08:13:22] lb.utils.events INFO: eta: 2 days, 3:42:50 iteration: 27799/375342 consumed_samples: 28467200 total_loss: 0.518 time: 0.5335 s/iter data_time: 0.0441 s/iter total_throughput: 1919.51 samples/s lr: 9.87e-04 [09/19 08:14:15] lb.utils.events INFO: eta: 2 days, 3:43:38 iteration: 27899/375342 consumed_samples: 28569600 total_loss: 0.5121 time: 0.5335 s/iter data_time: 0.0415 s/iter total_throughput: 1919.48 samples/s lr: 9.87e-04 [09/19 08:15:09] lb.utils.events INFO: eta: 2 days, 3:43:22 iteration: 27999/375342 consumed_samples: 28672000 total_loss: 0.5082 time: 0.5335 s/iter data_time: 0.0391 s/iter total_throughput: 1919.45 samples/s lr: 9.86e-04 [09/19 08:16:03] lb.utils.events INFO: eta: 2 days, 3:42:44 iteration: 28099/375342 consumed_samples: 28774400 total_loss: 0.5118 time: 0.5335 s/iter data_time: 0.0416 s/iter total_throughput: 1919.41 samples/s lr: 9.86e-04 [09/19 08:16:56] lb.utils.events INFO: eta: 2 days, 3:43:20 iteration: 28199/375342 consumed_samples: 28876800 total_loss: 0.5153 time: 0.5335 s/iter data_time: 0.0425 s/iter total_throughput: 1919.37 samples/s lr: 9.86e-04 [09/19 08:17:50] lb.utils.events INFO: eta: 2 days, 3:43:17 iteration: 28299/375342 consumed_samples: 28979200 total_loss: 0.5125 time: 0.5335 s/iter data_time: 0.0402 s/iter total_throughput: 1919.33 samples/s lr: 9.86e-04 [09/19 08:18:43] lb.utils.events INFO: eta: 2 days, 3:43:35 iteration: 28399/375342 consumed_samples: 29081600 total_loss: 0.5017 time: 0.5335 s/iter data_time: 0.0388 s/iter total_throughput: 1919.31 samples/s lr: 9.86e-04 [09/19 08:19:37] lb.utils.events INFO: eta: 2 days, 3:42:24 iteration: 28499/375342 consumed_samples: 29184000 total_loss: 0.4994 time: 0.5335 s/iter data_time: 0.0430 s/iter total_throughput: 1919.29 samples/s lr: 9.86e-04 [09/19 08:20:31] lb.utils.events INFO: eta: 2 days, 3:44:27 iteration: 28599/375342 consumed_samples: 29286400 total_loss: 0.5041 time: 0.5335 s/iter data_time: 0.0420 s/iter total_throughput: 1919.25 samples/s lr: 9.86e-04 [09/19 08:21:24] lb.utils.events INFO: eta: 2 days, 3:42:51 iteration: 28699/375342 consumed_samples: 29388800 total_loss: 0.5073 time: 0.5336 s/iter data_time: 0.0420 s/iter total_throughput: 1919.22 samples/s lr: 9.86e-04 [09/19 08:22:18] lb.utils.events INFO: eta: 2 days, 3:42:13 iteration: 28799/375342 consumed_samples: 29491200 total_loss: 0.5085 time: 0.5336 s/iter data_time: 0.0415 s/iter total_throughput: 1919.17 samples/s lr: 9.86e-04 [09/19 08:23:12] lb.utils.events INFO: eta: 2 days, 3:40:27 iteration: 28899/375342 consumed_samples: 29593600 total_loss: 0.5081 time: 0.5336 s/iter data_time: 0.0382 s/iter total_throughput: 1919.15 samples/s lr: 9.86e-04 [09/19 08:24:05] lb.utils.events INFO: eta: 2 days, 3:40:07 iteration: 28999/375342 consumed_samples: 29696000 total_loss: 0.5018 time: 0.5336 s/iter data_time: 0.0386 s/iter total_throughput: 1919.11 samples/s lr: 9.85e-04 [09/19 08:24:59] lb.utils.events INFO: eta: 2 days, 3:40:19 iteration: 29099/375342 consumed_samples: 29798400 total_loss: 0.5047 time: 0.5336 s/iter data_time: 0.0403 s/iter total_throughput: 1919.05 samples/s lr: 9.85e-04 [09/19 08:25:53] lb.utils.events INFO: eta: 2 days, 3:39:57 iteration: 29199/375342 consumed_samples: 29900800 total_loss: 0.5082 time: 0.5336 s/iter data_time: 0.0397 s/iter total_throughput: 1919.00 samples/s lr: 9.85e-04 [09/19 08:26:47] lb.utils.events INFO: eta: 2 days, 3:39:30 iteration: 29299/375342 consumed_samples: 30003200 total_loss: 0.5099 time: 0.5336 s/iter data_time: 0.0414 s/iter total_throughput: 1918.94 samples/s lr: 9.85e-04 [09/19 08:27:41] lb.utils.events INFO: eta: 2 days, 3:39:24 iteration: 29399/375342 consumed_samples: 30105600 total_loss: 0.5014 time: 0.5336 s/iter data_time: 0.0427 s/iter total_throughput: 1918.88 samples/s lr: 9.85e-04 [09/19 08:28:34] lb.utils.events INFO: eta: 2 days, 3:39:40 iteration: 29499/375342 consumed_samples: 30208000 total_loss: 0.5074 time: 0.5337 s/iter data_time: 0.0409 s/iter total_throughput: 1918.84 samples/s lr: 9.85e-04 [09/19 08:29:28] lb.utils.events INFO: eta: 2 days, 3:38:49 iteration: 29599/375342 consumed_samples: 30310400 total_loss: 0.513 time: 0.5337 s/iter data_time: 0.0383 s/iter total_throughput: 1918.80 samples/s lr: 9.85e-04 [09/19 08:30:22] lb.utils.events INFO: eta: 2 days, 3:38:16 iteration: 29699/375342 consumed_samples: 30412800 total_loss: 0.5083 time: 0.5337 s/iter data_time: 0.0439 s/iter total_throughput: 1918.75 samples/s lr: 9.85e-04 [09/19 08:31:16] lb.utils.events INFO: eta: 2 days, 3:37:49 iteration: 29799/375342 consumed_samples: 30515200 total_loss: 0.5089 time: 0.5337 s/iter data_time: 0.0422 s/iter total_throughput: 1918.70 samples/s lr: 9.85e-04 [09/19 08:32:09] lb.utils.events INFO: eta: 2 days, 3:37:11 iteration: 29899/375342 consumed_samples: 30617600 total_loss: 0.5113 time: 0.5337 s/iter data_time: 0.0401 s/iter total_throughput: 1918.67 samples/s lr: 9.85e-04 [09/19 08:33:03] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0029999 [09/19 08:33:04] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 08:33:04] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 08:33:08] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0606 s/iter. Inference: 0.2498 s/iter. Eval: 0.0023 s/iter. Total: 0.3128 s/iter. ETA=0:00:11 [09/19 08:33:13] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0684 s/iter. Inference: 0.2628 s/iter. Eval: 0.0025 s/iter. Total: 0.3338 s/iter. ETA=0:00:07 [09/19 08:33:19] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0677 s/iter. Inference: 0.2586 s/iter. Eval: 0.0025 s/iter. Total: 0.3291 s/iter. ETA=0:00:01 [09/19 08:33:21] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 08:33:21] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.154795 (0.000283 s / iter per device, on 8 devices) [09/19 08:33:21] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000228 s / iter per device, on 8 devices) [09/19 08:33:21] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 08:33:21] lb.evaluation.utils INFO: copypaste: Acc@1=61.082 [09/19 08:33:21] lb.evaluation.utils INFO: copypaste: Acc@5=83.842 [09/19 08:33:21] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 61.08200, better than last best score 57.95800 @ iteration 24999. [09/19 08:33:21] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 08:33:21] lb.utils.events INFO: eta: 2 days, 3:36:17 iteration: 29999/375342 consumed_samples: 30720000 total_loss: 0.5061 time: 0.5337 s/iter data_time: 0.0413 s/iter total_throughput: 1918.63 samples/s lr: 9.84e-04 [09/19 08:34:15] lb.utils.events INFO: eta: 2 days, 3:34:36 iteration: 30099/375342 consumed_samples: 30822400 total_loss: 0.5087 time: 0.5337 s/iter data_time: 0.0426 s/iter total_throughput: 1918.59 samples/s lr: 9.84e-04 [09/19 08:35:09] lb.utils.events INFO: eta: 2 days, 3:32:44 iteration: 30199/375342 consumed_samples: 30924800 total_loss: 0.5077 time: 0.5337 s/iter data_time: 0.0432 s/iter total_throughput: 1918.55 samples/s lr: 9.84e-04 [09/19 08:36:02] lb.utils.events INFO: eta: 2 days, 3:31:24 iteration: 30299/375342 consumed_samples: 31027200 total_loss: 0.5084 time: 0.5337 s/iter data_time: 0.0443 s/iter total_throughput: 1918.51 samples/s lr: 9.84e-04 [09/19 08:36:56] lb.utils.events INFO: eta: 2 days, 3:28:53 iteration: 30399/375342 consumed_samples: 31129600 total_loss: 0.5037 time: 0.5338 s/iter data_time: 0.0424 s/iter total_throughput: 1918.48 samples/s lr: 9.84e-04 [09/19 08:37:50] lb.utils.events INFO: eta: 2 days, 3:27:37 iteration: 30499/375342 consumed_samples: 31232000 total_loss: 0.4992 time: 0.5338 s/iter data_time: 0.0427 s/iter total_throughput: 1918.45 samples/s lr: 9.84e-04 [09/19 08:38:43] lb.utils.events INFO: eta: 2 days, 3:25:25 iteration: 30599/375342 consumed_samples: 31334400 total_loss: 0.5074 time: 0.5338 s/iter data_time: 0.0426 s/iter total_throughput: 1918.43 samples/s lr: 9.84e-04 [09/19 08:39:37] lb.utils.events INFO: eta: 2 days, 3:23:25 iteration: 30699/375342 consumed_samples: 31436800 total_loss: 0.51 time: 0.5338 s/iter data_time: 0.0421 s/iter total_throughput: 1918.40 samples/s lr: 9.84e-04 [09/19 08:40:31] lb.utils.events INFO: eta: 2 days, 3:21:09 iteration: 30799/375342 consumed_samples: 31539200 total_loss: 0.5066 time: 0.5338 s/iter data_time: 0.0411 s/iter total_throughput: 1918.37 samples/s lr: 9.84e-04 [09/19 08:41:24] lb.utils.events INFO: eta: 2 days, 3:19:24 iteration: 30899/375342 consumed_samples: 31641600 total_loss: 0.5006 time: 0.5338 s/iter data_time: 0.0411 s/iter total_throughput: 1918.34 samples/s lr: 9.84e-04 [09/19 08:42:18] lb.utils.events INFO: eta: 2 days, 3:17:11 iteration: 30999/375342 consumed_samples: 31744000 total_loss: 0.4996 time: 0.5338 s/iter data_time: 0.0413 s/iter total_throughput: 1918.33 samples/s lr: 9.83e-04 [09/19 08:43:12] lb.utils.events INFO: eta: 2 days, 3:15:29 iteration: 31099/375342 consumed_samples: 31846400 total_loss: 0.5041 time: 0.5338 s/iter data_time: 0.0432 s/iter total_throughput: 1918.27 samples/s lr: 9.83e-04 [09/19 08:44:06] lb.utils.events INFO: eta: 2 days, 3:14:47 iteration: 31199/375342 consumed_samples: 31948800 total_loss: 0.5042 time: 0.5338 s/iter data_time: 0.0424 s/iter total_throughput: 1918.22 samples/s lr: 9.83e-04 [09/19 08:44:59] lb.utils.events INFO: eta: 2 days, 3:13:53 iteration: 31299/375342 consumed_samples: 32051200 total_loss: 0.4919 time: 0.5338 s/iter data_time: 0.0430 s/iter total_throughput: 1918.19 samples/s lr: 9.83e-04 [09/19 08:45:53] lb.utils.events INFO: eta: 2 days, 3:13:07 iteration: 31399/375342 consumed_samples: 32153600 total_loss: 0.4921 time: 0.5338 s/iter data_time: 0.0449 s/iter total_throughput: 1918.17 samples/s lr: 9.83e-04 [09/19 08:46:46] lb.utils.events INFO: eta: 2 days, 3:13:07 iteration: 31499/375342 consumed_samples: 32256000 total_loss: 0.4925 time: 0.5339 s/iter data_time: 0.0429 s/iter total_throughput: 1918.14 samples/s lr: 9.83e-04 [09/19 08:47:40] lb.utils.events INFO: eta: 2 days, 3:13:28 iteration: 31599/375342 consumed_samples: 32358400 total_loss: 0.4942 time: 0.5339 s/iter data_time: 0.0394 s/iter total_throughput: 1918.10 samples/s lr: 9.83e-04 [09/19 08:48:34] lb.utils.events INFO: eta: 2 days, 3:13:53 iteration: 31699/375342 consumed_samples: 32460800 total_loss: 0.4951 time: 0.5339 s/iter data_time: 0.0413 s/iter total_throughput: 1918.06 samples/s lr: 9.83e-04 [09/19 08:49:28] lb.utils.events INFO: eta: 2 days, 3:14:56 iteration: 31799/375342 consumed_samples: 32563200 total_loss: 0.4944 time: 0.5339 s/iter data_time: 0.0465 s/iter total_throughput: 1918.02 samples/s lr: 9.83e-04 [09/19 08:50:21] lb.utils.events INFO: eta: 2 days, 3:14:50 iteration: 31899/375342 consumed_samples: 32665600 total_loss: 0.5075 time: 0.5339 s/iter data_time: 0.0425 s/iter total_throughput: 1918.00 samples/s lr: 9.82e-04 [09/19 08:51:15] lb.utils.events INFO: eta: 2 days, 3:15:30 iteration: 31999/375342 consumed_samples: 32768000 total_loss: 0.4998 time: 0.5339 s/iter data_time: 0.0446 s/iter total_throughput: 1917.97 samples/s lr: 9.82e-04 [09/19 08:52:09] lb.utils.events INFO: eta: 2 days, 3:16:27 iteration: 32099/375342 consumed_samples: 32870400 total_loss: 0.488 time: 0.5339 s/iter data_time: 0.0415 s/iter total_throughput: 1917.93 samples/s lr: 9.82e-04 [09/19 08:53:03] lb.utils.events INFO: eta: 2 days, 3:15:25 iteration: 32199/375342 consumed_samples: 32972800 total_loss: 0.4958 time: 0.5339 s/iter data_time: 0.0421 s/iter total_throughput: 1917.89 samples/s lr: 9.82e-04 [09/19 08:53:56] lb.utils.events INFO: eta: 2 days, 3:14:44 iteration: 32299/375342 consumed_samples: 33075200 total_loss: 0.5055 time: 0.5339 s/iter data_time: 0.0395 s/iter total_throughput: 1917.87 samples/s lr: 9.82e-04 [09/19 08:54:50] lb.utils.events INFO: eta: 2 days, 3:13:52 iteration: 32399/375342 consumed_samples: 33177600 total_loss: 0.5023 time: 0.5339 s/iter data_time: 0.0400 s/iter total_throughput: 1917.84 samples/s lr: 9.82e-04 [09/19 08:55:44] lb.utils.events INFO: eta: 2 days, 3:12:54 iteration: 32499/375342 consumed_samples: 33280000 total_loss: 0.4977 time: 0.5339 s/iter data_time: 0.0424 s/iter total_throughput: 1917.80 samples/s lr: 9.82e-04 [09/19 08:56:37] lb.utils.events INFO: eta: 2 days, 3:12:05 iteration: 32599/375342 consumed_samples: 33382400 total_loss: 0.4924 time: 0.5340 s/iter data_time: 0.0414 s/iter total_throughput: 1917.76 samples/s lr: 9.82e-04 [09/19 08:57:31] lb.utils.events INFO: eta: 2 days, 3:11:23 iteration: 32699/375342 consumed_samples: 33484800 total_loss: 0.4927 time: 0.5340 s/iter data_time: 0.0448 s/iter total_throughput: 1917.71 samples/s lr: 9.82e-04 [09/19 08:58:25] lb.utils.events INFO: eta: 2 days, 3:10:51 iteration: 32799/375342 consumed_samples: 33587200 total_loss: 0.5022 time: 0.5340 s/iter data_time: 0.0411 s/iter total_throughput: 1917.67 samples/s lr: 9.81e-04 [09/19 08:59:19] lb.utils.events INFO: eta: 2 days, 3:09:35 iteration: 32899/375342 consumed_samples: 33689600 total_loss: 0.4953 time: 0.5340 s/iter data_time: 0.0433 s/iter total_throughput: 1917.62 samples/s lr: 9.81e-04 [09/19 09:00:13] lb.utils.events INFO: eta: 2 days, 3:09:28 iteration: 32999/375342 consumed_samples: 33792000 total_loss: 0.4842 time: 0.5340 s/iter data_time: 0.0433 s/iter total_throughput: 1917.58 samples/s lr: 9.81e-04 [09/19 09:01:07] lb.utils.events INFO: eta: 2 days, 3:08:41 iteration: 33099/375342 consumed_samples: 33894400 total_loss: 0.4939 time: 0.5340 s/iter data_time: 0.0406 s/iter total_throughput: 1917.54 samples/s lr: 9.81e-04 [09/19 09:02:00] lb.utils.events INFO: eta: 2 days, 3:07:52 iteration: 33199/375342 consumed_samples: 33996800 total_loss: 0.5006 time: 0.5340 s/iter data_time: 0.0421 s/iter total_throughput: 1917.50 samples/s lr: 9.81e-04 [09/19 09:02:54] lb.utils.events INFO: eta: 2 days, 3:07:50 iteration: 33299/375342 consumed_samples: 34099200 total_loss: 0.502 time: 0.5340 s/iter data_time: 0.0426 s/iter total_throughput: 1917.45 samples/s lr: 9.81e-04 [09/19 09:03:48] lb.utils.events INFO: eta: 2 days, 3:07:14 iteration: 33399/375342 consumed_samples: 34201600 total_loss: 0.5019 time: 0.5341 s/iter data_time: 0.0429 s/iter total_throughput: 1917.41 samples/s lr: 9.81e-04 [09/19 09:04:42] lb.utils.events INFO: eta: 2 days, 3:06:46 iteration: 33499/375342 consumed_samples: 34304000 total_loss: 0.488 time: 0.5341 s/iter data_time: 0.0432 s/iter total_throughput: 1917.37 samples/s lr: 9.81e-04 [09/19 09:05:36] lb.utils.events INFO: eta: 2 days, 3:05:18 iteration: 33599/375342 consumed_samples: 34406400 total_loss: 0.4874 time: 0.5341 s/iter data_time: 0.0438 s/iter total_throughput: 1917.33 samples/s lr: 9.81e-04 [09/19 09:06:29] lb.utils.events INFO: eta: 2 days, 3:03:04 iteration: 33699/375342 consumed_samples: 34508800 total_loss: 0.493 time: 0.5341 s/iter data_time: 0.0431 s/iter total_throughput: 1917.30 samples/s lr: 9.80e-04 [09/19 09:07:23] lb.utils.events INFO: eta: 2 days, 3:01:35 iteration: 33799/375342 consumed_samples: 34611200 total_loss: 0.4903 time: 0.5341 s/iter data_time: 0.0441 s/iter total_throughput: 1917.27 samples/s lr: 9.80e-04 [09/19 09:08:17] lb.utils.events INFO: eta: 2 days, 2:59:45 iteration: 33899/375342 consumed_samples: 34713600 total_loss: 0.491 time: 0.5341 s/iter data_time: 0.0437 s/iter total_throughput: 1917.25 samples/s lr: 9.80e-04 [09/19 09:09:10] lb.utils.events INFO: eta: 2 days, 2:57:28 iteration: 33999/375342 consumed_samples: 34816000 total_loss: 0.4889 time: 0.5341 s/iter data_time: 0.0432 s/iter total_throughput: 1917.22 samples/s lr: 9.80e-04 [09/19 09:10:04] lb.utils.events INFO: eta: 2 days, 2:55:51 iteration: 34099/375342 consumed_samples: 34918400 total_loss: 0.4816 time: 0.5341 s/iter data_time: 0.0433 s/iter total_throughput: 1917.21 samples/s lr: 9.80e-04 [09/19 09:10:58] lb.utils.events INFO: eta: 2 days, 2:53:57 iteration: 34199/375342 consumed_samples: 35020800 total_loss: 0.4859 time: 0.5341 s/iter data_time: 0.0407 s/iter total_throughput: 1917.18 samples/s lr: 9.80e-04 [09/19 09:11:51] lb.utils.events INFO: eta: 2 days, 2:50:54 iteration: 34299/375342 consumed_samples: 35123200 total_loss: 0.4955 time: 0.5341 s/iter data_time: 0.0427 s/iter total_throughput: 1917.16 samples/s lr: 9.80e-04 [09/19 09:12:45] lb.utils.events INFO: eta: 2 days, 2:48:58 iteration: 34399/375342 consumed_samples: 35225600 total_loss: 0.497 time: 0.5341 s/iter data_time: 0.0415 s/iter total_throughput: 1917.14 samples/s lr: 9.80e-04 [09/19 09:13:39] lb.utils.events INFO: eta: 2 days, 2:46:39 iteration: 34499/375342 consumed_samples: 35328000 total_loss: 0.4991 time: 0.5341 s/iter data_time: 0.0453 s/iter total_throughput: 1917.09 samples/s lr: 9.80e-04 [09/19 09:14:32] lb.utils.events INFO: eta: 2 days, 2:45:28 iteration: 34599/375342 consumed_samples: 35430400 total_loss: 0.5015 time: 0.5342 s/iter data_time: 0.0440 s/iter total_throughput: 1917.06 samples/s lr: 9.79e-04 [09/19 09:15:26] lb.utils.events INFO: eta: 2 days, 2:44:53 iteration: 34699/375342 consumed_samples: 35532800 total_loss: 0.4958 time: 0.5342 s/iter data_time: 0.0444 s/iter total_throughput: 1917.03 samples/s lr: 9.79e-04 [09/19 09:16:20] lb.utils.events INFO: eta: 2 days, 2:44:00 iteration: 34799/375342 consumed_samples: 35635200 total_loss: 0.4921 time: 0.5342 s/iter data_time: 0.0435 s/iter total_throughput: 1917.00 samples/s lr: 9.79e-04 [09/19 09:17:14] lb.utils.events INFO: eta: 2 days, 2:46:16 iteration: 34899/375342 consumed_samples: 35737600 total_loss: 0.492 time: 0.5342 s/iter data_time: 0.0458 s/iter total_throughput: 1916.97 samples/s lr: 9.79e-04 [09/19 09:18:07] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0034999 [09/19 09:18:08] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 09:18:08] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 09:18:13] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0468 s/iter. Inference: 0.2546 s/iter. Eval: 0.0023 s/iter. Total: 0.3037 s/iter. ETA=0:00:11 [09/19 09:18:18] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0756 s/iter. Inference: 0.2537 s/iter. Eval: 0.0023 s/iter. Total: 0.3319 s/iter. ETA=0:00:07 [09/19 09:18:23] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0736 s/iter. Inference: 0.2514 s/iter. Eval: 0.0022 s/iter. Total: 0.3275 s/iter. ETA=0:00:01 [09/19 09:18:25] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 09:18:25] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.128127 (0.000283 s / iter per device, on 8 devices) [09/19 09:18:25] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000220 s / iter per device, on 8 devices) [09/19 09:18:25] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 09:18:25] lb.evaluation.utils INFO: copypaste: Acc@1=63.483999999999995 [09/19 09:18:25] lb.evaluation.utils INFO: copypaste: Acc@5=85.47 [09/19 09:18:25] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 63.48400, better than last best score 61.08200 @ iteration 29999. [09/19 09:18:25] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 09:18:26] lb.utils.events INFO: eta: 2 days, 2:47:01 iteration: 34999/375342 consumed_samples: 35840000 total_loss: 0.4917 time: 0.5342 s/iter data_time: 0.0442 s/iter total_throughput: 1916.94 samples/s lr: 9.79e-04 [09/19 09:19:19] lb.utils.events INFO: eta: 2 days, 2:47:40 iteration: 35099/375342 consumed_samples: 35942400 total_loss: 0.4881 time: 0.5342 s/iter data_time: 0.0401 s/iter total_throughput: 1916.90 samples/s lr: 9.79e-04 [09/19 09:20:13] lb.utils.events INFO: eta: 2 days, 2:47:34 iteration: 35199/375342 consumed_samples: 36044800 total_loss: 0.4877 time: 0.5342 s/iter data_time: 0.0419 s/iter total_throughput: 1916.86 samples/s lr: 9.79e-04 [09/19 09:21:07] lb.utils.events INFO: eta: 2 days, 2:47:06 iteration: 35299/375342 consumed_samples: 36147200 total_loss: 0.4895 time: 0.5342 s/iter data_time: 0.0399 s/iter total_throughput: 1916.85 samples/s lr: 9.79e-04 [09/19 09:22:01] lb.utils.events INFO: eta: 2 days, 2:47:00 iteration: 35399/375342 consumed_samples: 36249600 total_loss: 0.4913 time: 0.5342 s/iter data_time: 0.0427 s/iter total_throughput: 1916.83 samples/s lr: 9.78e-04 [09/19 09:22:54] lb.utils.events INFO: eta: 2 days, 2:46:50 iteration: 35499/375342 consumed_samples: 36352000 total_loss: 0.4939 time: 0.5342 s/iter data_time: 0.0439 s/iter total_throughput: 1916.81 samples/s lr: 9.78e-04 [09/19 09:23:48] lb.utils.events INFO: eta: 2 days, 2:45:49 iteration: 35599/375342 consumed_samples: 36454400 total_loss: 0.4911 time: 0.5342 s/iter data_time: 0.0386 s/iter total_throughput: 1916.78 samples/s lr: 9.78e-04 [09/19 09:24:42] lb.utils.events INFO: eta: 2 days, 2:44:43 iteration: 35699/375342 consumed_samples: 36556800 total_loss: 0.4901 time: 0.5342 s/iter data_time: 0.0404 s/iter total_throughput: 1916.76 samples/s lr: 9.78e-04 [09/19 09:25:35] lb.utils.events INFO: eta: 2 days, 2:43:35 iteration: 35799/375342 consumed_samples: 36659200 total_loss: 0.485 time: 0.5342 s/iter data_time: 0.0380 s/iter total_throughput: 1916.73 samples/s lr: 9.78e-04 [09/19 09:26:29] lb.utils.events INFO: eta: 2 days, 2:42:23 iteration: 35899/375342 consumed_samples: 36761600 total_loss: 0.4856 time: 0.5343 s/iter data_time: 0.0389 s/iter total_throughput: 1916.70 samples/s lr: 9.78e-04 [09/19 09:27:23] lb.utils.events INFO: eta: 2 days, 2:41:21 iteration: 35999/375342 consumed_samples: 36864000 total_loss: 0.4948 time: 0.5343 s/iter data_time: 0.0435 s/iter total_throughput: 1916.67 samples/s lr: 9.78e-04 [09/19 09:28:17] lb.utils.events INFO: eta: 2 days, 2:41:07 iteration: 36099/375342 consumed_samples: 36966400 total_loss: 0.4978 time: 0.5343 s/iter data_time: 0.0394 s/iter total_throughput: 1916.62 samples/s lr: 9.78e-04 [09/19 09:29:11] lb.utils.events INFO: eta: 2 days, 2:40:39 iteration: 36199/375342 consumed_samples: 37068800 total_loss: 0.4941 time: 0.5343 s/iter data_time: 0.0425 s/iter total_throughput: 1916.58 samples/s lr: 9.77e-04 [09/19 09:30:04] lb.utils.events INFO: eta: 2 days, 2:39:59 iteration: 36299/375342 consumed_samples: 37171200 total_loss: 0.4857 time: 0.5343 s/iter data_time: 0.0424 s/iter total_throughput: 1916.54 samples/s lr: 9.77e-04 [09/19 09:30:58] lb.utils.events INFO: eta: 2 days, 2:38:52 iteration: 36399/375342 consumed_samples: 37273600 total_loss: 0.478 time: 0.5343 s/iter data_time: 0.0424 s/iter total_throughput: 1916.51 samples/s lr: 9.77e-04 [09/19 09:31:52] lb.utils.events INFO: eta: 2 days, 2:38:22 iteration: 36499/375342 consumed_samples: 37376000 total_loss: 0.4891 time: 0.5343 s/iter data_time: 0.0375 s/iter total_throughput: 1916.48 samples/s lr: 9.77e-04 [09/19 09:32:46] lb.utils.events INFO: eta: 2 days, 2:37:53 iteration: 36599/375342 consumed_samples: 37478400 total_loss: 0.4885 time: 0.5343 s/iter data_time: 0.0396 s/iter total_throughput: 1916.44 samples/s lr: 9.77e-04 [09/19 09:33:40] lb.utils.events INFO: eta: 2 days, 2:37:52 iteration: 36699/375342 consumed_samples: 37580800 total_loss: 0.4842 time: 0.5343 s/iter data_time: 0.0428 s/iter total_throughput: 1916.40 samples/s lr: 9.77e-04 [09/19 09:34:33] lb.utils.events INFO: eta: 2 days, 2:37:05 iteration: 36799/375342 consumed_samples: 37683200 total_loss: 0.4894 time: 0.5343 s/iter data_time: 0.0414 s/iter total_throughput: 1916.37 samples/s lr: 9.77e-04 [09/19 09:35:27] lb.utils.events INFO: eta: 2 days, 2:37:04 iteration: 36899/375342 consumed_samples: 37785600 total_loss: 0.486 time: 0.5344 s/iter data_time: 0.0423 s/iter total_throughput: 1916.33 samples/s lr: 9.77e-04 [09/19 09:36:21] lb.utils.events INFO: eta: 2 days, 2:35:27 iteration: 36999/375342 consumed_samples: 37888000 total_loss: 0.4825 time: 0.5344 s/iter data_time: 0.0426 s/iter total_throughput: 1916.30 samples/s lr: 9.76e-04 [09/19 09:37:15] lb.utils.events INFO: eta: 2 days, 2:32:54 iteration: 37099/375342 consumed_samples: 37990400 total_loss: 0.4865 time: 0.5344 s/iter data_time: 0.0438 s/iter total_throughput: 1916.27 samples/s lr: 9.76e-04 [09/19 09:38:08] lb.utils.events INFO: eta: 2 days, 2:29:57 iteration: 37199/375342 consumed_samples: 38092800 total_loss: 0.4889 time: 0.5344 s/iter data_time: 0.0421 s/iter total_throughput: 1916.25 samples/s lr: 9.76e-04 [09/19 09:39:02] lb.utils.events INFO: eta: 2 days, 2:28:51 iteration: 37299/375342 consumed_samples: 38195200 total_loss: 0.4858 time: 0.5344 s/iter data_time: 0.0436 s/iter total_throughput: 1916.23 samples/s lr: 9.76e-04 [09/19 09:39:56] lb.utils.events INFO: eta: 2 days, 2:27:26 iteration: 37399/375342 consumed_samples: 38297600 total_loss: 0.4836 time: 0.5344 s/iter data_time: 0.0427 s/iter total_throughput: 1916.22 samples/s lr: 9.76e-04 [09/19 09:40:49] lb.utils.events INFO: eta: 2 days, 2:25:17 iteration: 37499/375342 consumed_samples: 38400000 total_loss: 0.4846 time: 0.5344 s/iter data_time: 0.0430 s/iter total_throughput: 1916.20 samples/s lr: 9.76e-04 [09/19 09:41:43] lb.utils.events INFO: eta: 2 days, 2:23:03 iteration: 37599/375342 consumed_samples: 38502400 total_loss: 0.4839 time: 0.5344 s/iter data_time: 0.0414 s/iter total_throughput: 1916.19 samples/s lr: 9.76e-04 [09/19 09:42:36] lb.utils.events INFO: eta: 2 days, 2:19:59 iteration: 37699/375342 consumed_samples: 38604800 total_loss: 0.4848 time: 0.5344 s/iter data_time: 0.0406 s/iter total_throughput: 1916.18 samples/s lr: 9.76e-04 [09/19 09:43:30] lb.utils.events INFO: eta: 2 days, 2:17:48 iteration: 37799/375342 consumed_samples: 38707200 total_loss: 0.4786 time: 0.5344 s/iter data_time: 0.0408 s/iter total_throughput: 1916.17 samples/s lr: 9.75e-04 [09/19 09:44:24] lb.utils.events INFO: eta: 2 days, 2:15:21 iteration: 37899/375342 consumed_samples: 38809600 total_loss: 0.4798 time: 0.5344 s/iter data_time: 0.0428 s/iter total_throughput: 1916.16 samples/s lr: 9.75e-04 [09/19 09:45:18] lb.utils.events INFO: eta: 2 days, 2:13:46 iteration: 37999/375342 consumed_samples: 38912000 total_loss: 0.486 time: 0.5344 s/iter data_time: 0.0443 s/iter total_throughput: 1916.11 samples/s lr: 9.75e-04 [09/19 09:46:11] lb.utils.events INFO: eta: 2 days, 2:12:49 iteration: 38099/375342 consumed_samples: 39014400 total_loss: 0.4775 time: 0.5344 s/iter data_time: 0.0427 s/iter total_throughput: 1916.09 samples/s lr: 9.75e-04 [09/19 09:47:05] lb.utils.events INFO: eta: 2 days, 2:12:41 iteration: 38199/375342 consumed_samples: 39116800 total_loss: 0.4746 time: 0.5344 s/iter data_time: 0.0426 s/iter total_throughput: 1916.06 samples/s lr: 9.75e-04 [09/19 09:47:59] lb.utils.events INFO: eta: 2 days, 2:11:49 iteration: 38299/375342 consumed_samples: 39219200 total_loss: 0.4827 time: 0.5344 s/iter data_time: 0.0417 s/iter total_throughput: 1916.03 samples/s lr: 9.75e-04 [09/19 09:48:53] lb.utils.events INFO: eta: 2 days, 2:11:29 iteration: 38399/375342 consumed_samples: 39321600 total_loss: 0.4833 time: 0.5344 s/iter data_time: 0.0449 s/iter total_throughput: 1916.01 samples/s lr: 9.75e-04 [09/19 09:49:46] lb.utils.events INFO: eta: 2 days, 2:11:47 iteration: 38499/375342 consumed_samples: 39424000 total_loss: 0.4771 time: 0.5345 s/iter data_time: 0.0442 s/iter total_throughput: 1915.97 samples/s lr: 9.75e-04 [09/19 09:50:40] lb.utils.events INFO: eta: 2 days, 2:13:43 iteration: 38599/375342 consumed_samples: 39526400 total_loss: 0.4833 time: 0.5345 s/iter data_time: 0.0410 s/iter total_throughput: 1915.93 samples/s lr: 9.74e-04 [09/19 09:51:34] lb.utils.events INFO: eta: 2 days, 2:14:24 iteration: 38699/375342 consumed_samples: 39628800 total_loss: 0.483 time: 0.5345 s/iter data_time: 0.0422 s/iter total_throughput: 1915.90 samples/s lr: 9.74e-04 [09/19 09:52:28] lb.utils.events INFO: eta: 2 days, 2:14:42 iteration: 38799/375342 consumed_samples: 39731200 total_loss: 0.4784 time: 0.5345 s/iter data_time: 0.0425 s/iter total_throughput: 1915.88 samples/s lr: 9.74e-04 [09/19 09:53:21] lb.utils.events INFO: eta: 2 days, 2:15:06 iteration: 38899/375342 consumed_samples: 39833600 total_loss: 0.4914 time: 0.5345 s/iter data_time: 0.0417 s/iter total_throughput: 1915.87 samples/s lr: 9.74e-04 [09/19 09:54:15] lb.utils.events INFO: eta: 2 days, 2:15:22 iteration: 38999/375342 consumed_samples: 39936000 total_loss: 0.4932 time: 0.5345 s/iter data_time: 0.0444 s/iter total_throughput: 1915.85 samples/s lr: 9.74e-04 [09/19 09:55:09] lb.utils.events INFO: eta: 2 days, 2:14:09 iteration: 39099/375342 consumed_samples: 40038400 total_loss: 0.4862 time: 0.5345 s/iter data_time: 0.0428 s/iter total_throughput: 1915.84 samples/s lr: 9.74e-04 [09/19 09:56:02] lb.utils.events INFO: eta: 2 days, 2:11:22 iteration: 39199/375342 consumed_samples: 40140800 total_loss: 0.4841 time: 0.5345 s/iter data_time: 0.0384 s/iter total_throughput: 1915.82 samples/s lr: 9.74e-04 [09/19 09:56:56] lb.utils.events INFO: eta: 2 days, 2:11:31 iteration: 39299/375342 consumed_samples: 40243200 total_loss: 0.483 time: 0.5345 s/iter data_time: 0.0362 s/iter total_throughput: 1915.79 samples/s lr: 9.73e-04 [09/19 09:57:50] lb.utils.events INFO: eta: 2 days, 2:10:39 iteration: 39399/375342 consumed_samples: 40345600 total_loss: 0.4792 time: 0.5345 s/iter data_time: 0.0413 s/iter total_throughput: 1915.77 samples/s lr: 9.73e-04 [09/19 09:58:44] lb.utils.events INFO: eta: 2 days, 2:08:45 iteration: 39499/375342 consumed_samples: 40448000 total_loss: 0.4787 time: 0.5345 s/iter data_time: 0.0416 s/iter total_throughput: 1915.75 samples/s lr: 9.73e-04 [09/19 09:59:37] lb.utils.events INFO: eta: 2 days, 2:07:22 iteration: 39599/375342 consumed_samples: 40550400 total_loss: 0.4818 time: 0.5345 s/iter data_time: 0.0434 s/iter total_throughput: 1915.72 samples/s lr: 9.73e-04 [09/19 10:00:31] lb.utils.events INFO: eta: 2 days, 2:06:57 iteration: 39699/375342 consumed_samples: 40652800 total_loss: 0.483 time: 0.5345 s/iter data_time: 0.0423 s/iter total_throughput: 1915.68 samples/s lr: 9.73e-04 [09/19 10:01:25] lb.utils.events INFO: eta: 2 days, 2:06:11 iteration: 39799/375342 consumed_samples: 40755200 total_loss: 0.4797 time: 0.5345 s/iter data_time: 0.0434 s/iter total_throughput: 1915.66 samples/s lr: 9.73e-04 [09/19 10:02:19] lb.utils.events INFO: eta: 2 days, 2:06:30 iteration: 39899/375342 consumed_samples: 40857600 total_loss: 0.4756 time: 0.5346 s/iter data_time: 0.0436 s/iter total_throughput: 1915.63 samples/s lr: 9.73e-04 [09/19 10:03:13] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0039999 [09/19 10:03:13] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 10:03:13] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 10:03:18] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0728 s/iter. Inference: 0.2446 s/iter. Eval: 0.0022 s/iter. Total: 0.3196 s/iter. ETA=0:00:11 [09/19 10:03:23] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0715 s/iter. Inference: 0.2623 s/iter. Eval: 0.0023 s/iter. Total: 0.3364 s/iter. ETA=0:00:07 [09/19 10:03:28] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0718 s/iter. Inference: 0.2568 s/iter. Eval: 0.0023 s/iter. Total: 0.3313 s/iter. ETA=0:00:01 [09/19 10:03:30] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 10:03:30] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.247731 (0.000285 s / iter per device, on 8 devices) [09/19 10:03:30] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/19 10:03:30] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 10:03:30] lb.evaluation.utils INFO: copypaste: Acc@1=64.566 [09/19 10:03:30] lb.evaluation.utils INFO: copypaste: Acc@5=86.36399999999999 [09/19 10:03:30] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 64.56600, better than last best score 63.48400 @ iteration 34999. [09/19 10:03:30] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 10:03:31] lb.utils.events INFO: eta: 2 days, 2:05:49 iteration: 39999/375342 consumed_samples: 40960000 total_loss: 0.4753 time: 0.5346 s/iter data_time: 0.0414 s/iter total_throughput: 1915.60 samples/s lr: 9.73e-04 [09/19 10:04:25] lb.utils.events INFO: eta: 2 days, 2:05:54 iteration: 40099/375342 consumed_samples: 41062400 total_loss: 0.4684 time: 0.5346 s/iter data_time: 0.0427 s/iter total_throughput: 1915.56 samples/s lr: 9.72e-04 [09/19 10:05:19] lb.utils.events INFO: eta: 2 days, 2:05:45 iteration: 40199/375342 consumed_samples: 41164800 total_loss: 0.4746 time: 0.5346 s/iter data_time: 0.0427 s/iter total_throughput: 1915.53 samples/s lr: 9.72e-04 [09/19 10:06:12] lb.utils.events INFO: eta: 2 days, 2:05:23 iteration: 40299/375342 consumed_samples: 41267200 total_loss: 0.4789 time: 0.5346 s/iter data_time: 0.0427 s/iter total_throughput: 1915.50 samples/s lr: 9.72e-04 [09/19 10:07:06] lb.utils.events INFO: eta: 2 days, 2:05:33 iteration: 40399/375342 consumed_samples: 41369600 total_loss: 0.4735 time: 0.5346 s/iter data_time: 0.0424 s/iter total_throughput: 1915.46 samples/s lr: 9.72e-04 [09/19 10:08:00] lb.utils.events INFO: eta: 2 days, 2:05:51 iteration: 40499/375342 consumed_samples: 41472000 total_loss: 0.4813 time: 0.5346 s/iter data_time: 0.0448 s/iter total_throughput: 1915.42 samples/s lr: 9.72e-04 [09/19 10:08:54] lb.utils.events INFO: eta: 2 days, 2:04:58 iteration: 40599/375342 consumed_samples: 41574400 total_loss: 0.4922 time: 0.5346 s/iter data_time: 0.0433 s/iter total_throughput: 1915.38 samples/s lr: 9.72e-04 [09/19 10:09:48] lb.utils.events INFO: eta: 2 days, 2:04:04 iteration: 40699/375342 consumed_samples: 41676800 total_loss: 0.4831 time: 0.5346 s/iter data_time: 0.0447 s/iter total_throughput: 1915.36 samples/s lr: 9.72e-04 [09/19 10:10:42] lb.utils.events INFO: eta: 2 days, 2:03:40 iteration: 40799/375342 consumed_samples: 41779200 total_loss: 0.4739 time: 0.5346 s/iter data_time: 0.0453 s/iter total_throughput: 1915.32 samples/s lr: 9.71e-04 [09/19 10:11:36] lb.utils.events INFO: eta: 2 days, 2:02:21 iteration: 40899/375342 consumed_samples: 41881600 total_loss: 0.4813 time: 0.5346 s/iter data_time: 0.0427 s/iter total_throughput: 1915.29 samples/s lr: 9.71e-04 [09/19 10:12:29] lb.utils.events INFO: eta: 2 days, 2:00:18 iteration: 40999/375342 consumed_samples: 41984000 total_loss: 0.4783 time: 0.5346 s/iter data_time: 0.0421 s/iter total_throughput: 1915.27 samples/s lr: 9.71e-04 [09/19 10:13:23] lb.utils.events INFO: eta: 2 days, 1:58:05 iteration: 41099/375342 consumed_samples: 42086400 total_loss: 0.4763 time: 0.5347 s/iter data_time: 0.0422 s/iter total_throughput: 1915.25 samples/s lr: 9.71e-04 [09/19 10:14:17] lb.utils.events INFO: eta: 2 days, 1:56:14 iteration: 41199/375342 consumed_samples: 42188800 total_loss: 0.4787 time: 0.5347 s/iter data_time: 0.0419 s/iter total_throughput: 1915.23 samples/s lr: 9.71e-04 [09/19 10:15:10] lb.utils.events INFO: eta: 2 days, 1:54:16 iteration: 41299/375342 consumed_samples: 42291200 total_loss: 0.48 time: 0.5347 s/iter data_time: 0.0428 s/iter total_throughput: 1915.22 samples/s lr: 9.71e-04 [09/19 10:16:04] lb.utils.events INFO: eta: 2 days, 1:52:21 iteration: 41399/375342 consumed_samples: 42393600 total_loss: 0.4794 time: 0.5347 s/iter data_time: 0.0464 s/iter total_throughput: 1915.17 samples/s lr: 9.71e-04 [09/19 10:16:58] lb.utils.events INFO: eta: 2 days, 1:50:40 iteration: 41499/375342 consumed_samples: 42496000 total_loss: 0.4775 time: 0.5347 s/iter data_time: 0.0410 s/iter total_throughput: 1915.14 samples/s lr: 9.70e-04 [09/19 10:17:52] lb.utils.events INFO: eta: 2 days, 1:50:51 iteration: 41599/375342 consumed_samples: 42598400 total_loss: 0.4766 time: 0.5347 s/iter data_time: 0.0423 s/iter total_throughput: 1915.10 samples/s lr: 9.70e-04 [09/19 10:18:46] lb.utils.events INFO: eta: 2 days, 1:49:58 iteration: 41699/375342 consumed_samples: 42700800 total_loss: 0.4776 time: 0.5347 s/iter data_time: 0.0441 s/iter total_throughput: 1915.07 samples/s lr: 9.70e-04 [09/19 10:19:40] lb.utils.events INFO: eta: 2 days, 1:49:09 iteration: 41799/375342 consumed_samples: 42803200 total_loss: 0.4752 time: 0.5347 s/iter data_time: 0.0451 s/iter total_throughput: 1915.04 samples/s lr: 9.70e-04 [09/19 10:20:34] lb.utils.events INFO: eta: 2 days, 1:48:45 iteration: 41899/375342 consumed_samples: 42905600 total_loss: 0.4763 time: 0.5347 s/iter data_time: 0.0408 s/iter total_throughput: 1914.99 samples/s lr: 9.70e-04 [09/19 10:21:28] lb.utils.events INFO: eta: 2 days, 1:50:29 iteration: 41999/375342 consumed_samples: 43008000 total_loss: 0.4757 time: 0.5347 s/iter data_time: 0.0423 s/iter total_throughput: 1914.94 samples/s lr: 9.70e-04 [09/19 10:22:22] lb.utils.events INFO: eta: 2 days, 1:51:30 iteration: 42099/375342 consumed_samples: 43110400 total_loss: 0.4745 time: 0.5348 s/iter data_time: 0.0475 s/iter total_throughput: 1914.91 samples/s lr: 9.70e-04 [09/19 10:23:16] lb.utils.events INFO: eta: 2 days, 1:52:14 iteration: 42199/375342 consumed_samples: 43212800 total_loss: 0.4742 time: 0.5348 s/iter data_time: 0.0443 s/iter total_throughput: 1914.88 samples/s lr: 9.69e-04 [09/19 10:24:10] lb.utils.events INFO: eta: 2 days, 1:52:13 iteration: 42299/375342 consumed_samples: 43315200 total_loss: 0.4742 time: 0.5348 s/iter data_time: 0.0424 s/iter total_throughput: 1914.85 samples/s lr: 9.69e-04 [09/19 10:25:03] lb.utils.events INFO: eta: 2 days, 1:52:05 iteration: 42399/375342 consumed_samples: 43417600 total_loss: 0.4785 time: 0.5348 s/iter data_time: 0.0439 s/iter total_throughput: 1914.83 samples/s lr: 9.69e-04 [09/19 10:25:57] lb.utils.events INFO: eta: 2 days, 1:50:49 iteration: 42499/375342 consumed_samples: 43520000 total_loss: 0.4796 time: 0.5348 s/iter data_time: 0.0451 s/iter total_throughput: 1914.80 samples/s lr: 9.69e-04 [09/19 10:26:51] lb.utils.events INFO: eta: 2 days, 1:49:41 iteration: 42599/375342 consumed_samples: 43622400 total_loss: 0.4764 time: 0.5348 s/iter data_time: 0.0426 s/iter total_throughput: 1914.77 samples/s lr: 9.69e-04 [09/19 10:27:45] lb.utils.events INFO: eta: 2 days, 1:48:33 iteration: 42699/375342 consumed_samples: 43724800 total_loss: 0.4749 time: 0.5348 s/iter data_time: 0.0368 s/iter total_throughput: 1914.75 samples/s lr: 9.69e-04 [09/19 10:28:39] lb.utils.events INFO: eta: 2 days, 1:47:49 iteration: 42799/375342 consumed_samples: 43827200 total_loss: 0.4782 time: 0.5348 s/iter data_time: 0.0408 s/iter total_throughput: 1914.72 samples/s lr: 9.69e-04 [09/19 10:29:33] lb.utils.events INFO: eta: 2 days, 1:47:57 iteration: 42899/375342 consumed_samples: 43929600 total_loss: 0.4746 time: 0.5348 s/iter data_time: 0.0428 s/iter total_throughput: 1914.68 samples/s lr: 9.68e-04 [09/19 10:30:27] lb.utils.events INFO: eta: 2 days, 1:47:01 iteration: 42999/375342 consumed_samples: 44032000 total_loss: 0.4725 time: 0.5348 s/iter data_time: 0.0425 s/iter total_throughput: 1914.64 samples/s lr: 9.68e-04 [09/19 10:31:21] lb.utils.events INFO: eta: 2 days, 1:47:01 iteration: 43099/375342 consumed_samples: 44134400 total_loss: 0.4766 time: 0.5348 s/iter data_time: 0.0443 s/iter total_throughput: 1914.59 samples/s lr: 9.68e-04 [09/19 10:32:15] lb.utils.events INFO: eta: 2 days, 1:46:07 iteration: 43199/375342 consumed_samples: 44236800 total_loss: 0.4802 time: 0.5349 s/iter data_time: 0.0441 s/iter total_throughput: 1914.55 samples/s lr: 9.68e-04 [09/19 10:33:09] lb.utils.events INFO: eta: 2 days, 1:45:17 iteration: 43299/375342 consumed_samples: 44339200 total_loss: 0.4765 time: 0.5349 s/iter data_time: 0.0435 s/iter total_throughput: 1914.51 samples/s lr: 9.68e-04 [09/19 10:34:03] lb.utils.events INFO: eta: 2 days, 1:44:37 iteration: 43399/375342 consumed_samples: 44441600 total_loss: 0.4738 time: 0.5349 s/iter data_time: 0.0422 s/iter total_throughput: 1914.48 samples/s lr: 9.68e-04 [09/19 10:34:56] lb.utils.events INFO: eta: 2 days, 1:43:29 iteration: 43499/375342 consumed_samples: 44544000 total_loss: 0.4667 time: 0.5349 s/iter data_time: 0.0436 s/iter total_throughput: 1914.46 samples/s lr: 9.68e-04 [09/19 10:35:50] lb.utils.events INFO: eta: 2 days, 1:42:28 iteration: 43599/375342 consumed_samples: 44646400 total_loss: 0.4621 time: 0.5349 s/iter data_time: 0.0459 s/iter total_throughput: 1914.43 samples/s lr: 9.67e-04 [09/19 10:36:44] lb.utils.events INFO: eta: 2 days, 1:41:22 iteration: 43699/375342 consumed_samples: 44748800 total_loss: 0.4741 time: 0.5349 s/iter data_time: 0.0451 s/iter total_throughput: 1914.40 samples/s lr: 9.67e-04 [09/19 10:37:38] lb.utils.events INFO: eta: 2 days, 1:39:48 iteration: 43799/375342 consumed_samples: 44851200 total_loss: 0.4802 time: 0.5349 s/iter data_time: 0.0456 s/iter total_throughput: 1914.37 samples/s lr: 9.67e-04 [09/19 10:38:32] lb.utils.events INFO: eta: 2 days, 1:37:06 iteration: 43899/375342 consumed_samples: 44953600 total_loss: 0.475 time: 0.5349 s/iter data_time: 0.0461 s/iter total_throughput: 1914.34 samples/s lr: 9.67e-04 [09/19 10:39:26] lb.utils.events INFO: eta: 2 days, 1:34:59 iteration: 43999/375342 consumed_samples: 45056000 total_loss: 0.4696 time: 0.5349 s/iter data_time: 0.0449 s/iter total_throughput: 1914.32 samples/s lr: 9.67e-04 [09/19 10:40:19] lb.utils.events INFO: eta: 2 days, 1:32:59 iteration: 44099/375342 consumed_samples: 45158400 total_loss: 0.4677 time: 0.5349 s/iter data_time: 0.0461 s/iter total_throughput: 1914.30 samples/s lr: 9.67e-04 [09/19 10:41:13] lb.utils.events INFO: eta: 2 days, 1:30:36 iteration: 44199/375342 consumed_samples: 45260800 total_loss: 0.4654 time: 0.5349 s/iter data_time: 0.0448 s/iter total_throughput: 1914.29 samples/s lr: 9.67e-04 [09/19 10:42:07] lb.utils.events INFO: eta: 2 days, 1:28:29 iteration: 44299/375342 consumed_samples: 45363200 total_loss: 0.477 time: 0.5349 s/iter data_time: 0.0448 s/iter total_throughput: 1914.27 samples/s lr: 9.66e-04 [09/19 10:43:00] lb.utils.events INFO: eta: 2 days, 1:25:40 iteration: 44399/375342 consumed_samples: 45465600 total_loss: 0.4792 time: 0.5349 s/iter data_time: 0.0438 s/iter total_throughput: 1914.26 samples/s lr: 9.66e-04 [09/19 10:43:54] lb.utils.events INFO: eta: 2 days, 1:24:35 iteration: 44499/375342 consumed_samples: 45568000 total_loss: 0.4782 time: 0.5349 s/iter data_time: 0.0436 s/iter total_throughput: 1914.24 samples/s lr: 9.66e-04 [09/19 10:44:48] lb.utils.events INFO: eta: 2 days, 1:22:18 iteration: 44599/375342 consumed_samples: 45670400 total_loss: 0.4702 time: 0.5349 s/iter data_time: 0.0444 s/iter total_throughput: 1914.23 samples/s lr: 9.66e-04 [09/19 10:45:41] lb.utils.events INFO: eta: 2 days, 1:20:22 iteration: 44699/375342 consumed_samples: 45772800 total_loss: 0.4703 time: 0.5349 s/iter data_time: 0.0432 s/iter total_throughput: 1914.22 samples/s lr: 9.66e-04 [09/19 10:46:35] lb.utils.events INFO: eta: 2 days, 1:18:20 iteration: 44799/375342 consumed_samples: 45875200 total_loss: 0.4711 time: 0.5349 s/iter data_time: 0.0432 s/iter total_throughput: 1914.20 samples/s lr: 9.66e-04 [09/19 10:47:29] lb.utils.events INFO: eta: 2 days, 1:16:34 iteration: 44899/375342 consumed_samples: 45977600 total_loss: 0.4703 time: 0.5350 s/iter data_time: 0.0454 s/iter total_throughput: 1914.16 samples/s lr: 9.65e-04 [09/19 10:48:23] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0044999 [09/19 10:48:24] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 10:48:24] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 10:48:28] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0668 s/iter. Inference: 0.2533 s/iter. Eval: 0.0022 s/iter. Total: 0.3223 s/iter. ETA=0:00:11 [09/19 10:48:33] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0645 s/iter. Inference: 0.2686 s/iter. Eval: 0.0023 s/iter. Total: 0.3358 s/iter. ETA=0:00:07 [09/19 10:48:39] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0652 s/iter. Inference: 0.2614 s/iter. Eval: 0.0023 s/iter. Total: 0.3293 s/iter. ETA=0:00:01 [09/19 10:48:41] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 10:48:41] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.232101 (0.000285 s / iter per device, on 8 devices) [09/19 10:48:41] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000227 s / iter per device, on 8 devices) [09/19 10:48:41] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 10:48:41] lb.evaluation.utils INFO: copypaste: Acc@1=66.60199999999999 [09/19 10:48:41] lb.evaluation.utils INFO: copypaste: Acc@5=87.682 [09/19 10:48:41] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 66.60200, better than last best score 64.56600 @ iteration 39999. [09/19 10:48:41] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 10:48:41] lb.utils.events INFO: eta: 2 days, 1:15:47 iteration: 44999/375342 consumed_samples: 46080000 total_loss: 0.471 time: 0.5350 s/iter data_time: 0.0417 s/iter total_throughput: 1914.13 samples/s lr: 9.65e-04 [09/19 10:49:35] lb.utils.events INFO: eta: 2 days, 1:15:34 iteration: 45099/375342 consumed_samples: 46182400 total_loss: 0.4624 time: 0.5350 s/iter data_time: 0.0452 s/iter total_throughput: 1914.11 samples/s lr: 9.65e-04 [09/19 10:50:29] lb.utils.events INFO: eta: 2 days, 1:16:01 iteration: 45199/375342 consumed_samples: 46284800 total_loss: 0.4522 time: 0.5350 s/iter data_time: 0.0467 s/iter total_throughput: 1914.07 samples/s lr: 9.65e-04 [09/19 10:51:23] lb.utils.events INFO: eta: 2 days, 1:16:02 iteration: 45299/375342 consumed_samples: 46387200 total_loss: 0.4572 time: 0.5350 s/iter data_time: 0.0464 s/iter total_throughput: 1914.03 samples/s lr: 9.65e-04 [09/19 10:52:17] lb.utils.events INFO: eta: 2 days, 1:17:42 iteration: 45399/375342 consumed_samples: 46489600 total_loss: 0.4663 time: 0.5350 s/iter data_time: 0.0431 s/iter total_throughput: 1913.99 samples/s lr: 9.65e-04 [09/19 10:53:11] lb.utils.events INFO: eta: 2 days, 1:19:42 iteration: 45499/375342 consumed_samples: 46592000 total_loss: 0.4708 time: 0.5350 s/iter data_time: 0.0454 s/iter total_throughput: 1913.94 samples/s lr: 9.65e-04 [09/19 10:54:05] lb.utils.events INFO: eta: 2 days, 1:20:42 iteration: 45599/375342 consumed_samples: 46694400 total_loss: 0.4741 time: 0.5350 s/iter data_time: 0.0460 s/iter total_throughput: 1913.90 samples/s lr: 9.64e-04 [09/19 10:54:59] lb.utils.events INFO: eta: 2 days, 1:22:01 iteration: 45699/375342 consumed_samples: 46796800 total_loss: 0.4785 time: 0.5350 s/iter data_time: 0.0442 s/iter total_throughput: 1913.87 samples/s lr: 9.64e-04 [09/19 10:55:53] lb.utils.events INFO: eta: 2 days, 1:22:23 iteration: 45799/375342 consumed_samples: 46899200 total_loss: 0.4775 time: 0.5350 s/iter data_time: 0.0466 s/iter total_throughput: 1913.85 samples/s lr: 9.64e-04 [09/19 10:56:47] lb.utils.events INFO: eta: 2 days, 1:22:50 iteration: 45899/375342 consumed_samples: 47001600 total_loss: 0.4757 time: 0.5351 s/iter data_time: 0.0467 s/iter total_throughput: 1913.82 samples/s lr: 9.64e-04 [09/19 10:57:41] lb.utils.events INFO: eta: 2 days, 1:22:27 iteration: 45999/375342 consumed_samples: 47104000 total_loss: 0.4762 time: 0.5351 s/iter data_time: 0.0458 s/iter total_throughput: 1913.79 samples/s lr: 9.64e-04 [09/19 10:58:35] lb.utils.events INFO: eta: 2 days, 1:21:49 iteration: 46099/375342 consumed_samples: 47206400 total_loss: 0.4756 time: 0.5351 s/iter data_time: 0.0439 s/iter total_throughput: 1913.76 samples/s lr: 9.64e-04 [09/19 10:59:29] lb.utils.events INFO: eta: 2 days, 1:21:20 iteration: 46199/375342 consumed_samples: 47308800 total_loss: 0.4771 time: 0.5351 s/iter data_time: 0.0422 s/iter total_throughput: 1913.73 samples/s lr: 9.63e-04 [09/19 11:00:23] lb.utils.events INFO: eta: 2 days, 1:20:11 iteration: 46299/375342 consumed_samples: 47411200 total_loss: 0.4756 time: 0.5351 s/iter data_time: 0.0432 s/iter total_throughput: 1913.70 samples/s lr: 9.63e-04 [09/19 11:01:17] lb.utils.events INFO: eta: 2 days, 1:18:45 iteration: 46399/375342 consumed_samples: 47513600 total_loss: 0.4747 time: 0.5351 s/iter data_time: 0.0420 s/iter total_throughput: 1913.66 samples/s lr: 9.63e-04 [09/19 11:02:11] lb.utils.events INFO: eta: 2 days, 1:17:54 iteration: 46499/375342 consumed_samples: 47616000 total_loss: 0.4731 time: 0.5351 s/iter data_time: 0.0441 s/iter total_throughput: 1913.62 samples/s lr: 9.63e-04 [09/19 11:03:05] lb.utils.events INFO: eta: 2 days, 1:17:00 iteration: 46599/375342 consumed_samples: 47718400 total_loss: 0.4708 time: 0.5351 s/iter data_time: 0.0434 s/iter total_throughput: 1913.58 samples/s lr: 9.63e-04 [09/19 11:03:59] lb.utils.events INFO: eta: 2 days, 1:16:01 iteration: 46699/375342 consumed_samples: 47820800 total_loss: 0.4695 time: 0.5351 s/iter data_time: 0.0465 s/iter total_throughput: 1913.54 samples/s lr: 9.63e-04 [09/19 11:04:53] lb.utils.events INFO: eta: 2 days, 1:15:41 iteration: 46799/375342 consumed_samples: 47923200 total_loss: 0.4727 time: 0.5351 s/iter data_time: 0.0444 s/iter total_throughput: 1913.50 samples/s lr: 9.63e-04 [09/19 11:05:47] lb.utils.events INFO: eta: 2 days, 1:15:50 iteration: 46899/375342 consumed_samples: 48025600 total_loss: 0.4711 time: 0.5352 s/iter data_time: 0.0431 s/iter total_throughput: 1913.47 samples/s lr: 9.62e-04 [09/19 11:06:41] lb.utils.events INFO: eta: 2 days, 1:15:03 iteration: 46999/375342 consumed_samples: 48128000 total_loss: 0.4565 time: 0.5352 s/iter data_time: 0.0420 s/iter total_throughput: 1913.43 samples/s lr: 9.62e-04 [09/19 11:07:35] lb.utils.events INFO: eta: 2 days, 1:14:07 iteration: 47099/375342 consumed_samples: 48230400 total_loss: 0.4656 time: 0.5352 s/iter data_time: 0.0458 s/iter total_throughput: 1913.41 samples/s lr: 9.62e-04 [09/19 11:08:29] lb.utils.events INFO: eta: 2 days, 1:12:12 iteration: 47199/375342 consumed_samples: 48332800 total_loss: 0.4675 time: 0.5352 s/iter data_time: 0.0455 s/iter total_throughput: 1913.38 samples/s lr: 9.62e-04 [09/19 11:09:23] lb.utils.events INFO: eta: 2 days, 1:10:54 iteration: 47299/375342 consumed_samples: 48435200 total_loss: 0.464 time: 0.5352 s/iter data_time: 0.0463 s/iter total_throughput: 1913.36 samples/s lr: 9.62e-04 [09/19 11:10:16] lb.utils.events INFO: eta: 2 days, 1:08:57 iteration: 47399/375342 consumed_samples: 48537600 total_loss: 0.474 time: 0.5352 s/iter data_time: 0.0451 s/iter total_throughput: 1913.33 samples/s lr: 9.62e-04 [09/19 11:11:10] lb.utils.events INFO: eta: 2 days, 1:06:13 iteration: 47499/375342 consumed_samples: 48640000 total_loss: 0.4664 time: 0.5352 s/iter data_time: 0.0455 s/iter total_throughput: 1913.32 samples/s lr: 9.61e-04 [09/19 11:12:04] lb.utils.events INFO: eta: 2 days, 1:04:09 iteration: 47599/375342 consumed_samples: 48742400 total_loss: 0.4647 time: 0.5352 s/iter data_time: 0.0451 s/iter total_throughput: 1913.30 samples/s lr: 9.61e-04 [09/19 11:12:58] lb.utils.events INFO: eta: 2 days, 1:01:31 iteration: 47699/375342 consumed_samples: 48844800 total_loss: 0.4762 time: 0.5352 s/iter data_time: 0.0438 s/iter total_throughput: 1913.29 samples/s lr: 9.61e-04 [09/19 11:13:51] lb.utils.events INFO: eta: 2 days, 0:58:38 iteration: 47799/375342 consumed_samples: 48947200 total_loss: 0.4699 time: 0.5352 s/iter data_time: 0.0449 s/iter total_throughput: 1913.27 samples/s lr: 9.61e-04 [09/19 11:14:45] lb.utils.events INFO: eta: 2 days, 0:55:45 iteration: 47899/375342 consumed_samples: 49049600 total_loss: 0.4654 time: 0.5352 s/iter data_time: 0.0444 s/iter total_throughput: 1913.27 samples/s lr: 9.61e-04 [09/19 11:15:39] lb.utils.events INFO: eta: 2 days, 0:53:04 iteration: 47999/375342 consumed_samples: 49152000 total_loss: 0.4582 time: 0.5352 s/iter data_time: 0.0443 s/iter total_throughput: 1913.25 samples/s lr: 9.61e-04 [09/19 11:16:33] lb.utils.events INFO: eta: 2 days, 0:51:13 iteration: 48099/375342 consumed_samples: 49254400 total_loss: 0.4596 time: 0.5352 s/iter data_time: 0.0422 s/iter total_throughput: 1913.23 samples/s lr: 9.60e-04 [09/19 11:17:26] lb.utils.events INFO: eta: 2 days, 0:49:51 iteration: 48199/375342 consumed_samples: 49356800 total_loss: 0.4681 time: 0.5352 s/iter data_time: 0.0437 s/iter total_throughput: 1913.22 samples/s lr: 9.60e-04 [09/19 11:18:20] lb.utils.events INFO: eta: 2 days, 0:47:23 iteration: 48299/375342 consumed_samples: 49459200 total_loss: 0.4695 time: 0.5352 s/iter data_time: 0.0477 s/iter total_throughput: 1913.19 samples/s lr: 9.60e-04 [09/19 11:19:14] lb.utils.events INFO: eta: 2 days, 0:47:57 iteration: 48399/375342 consumed_samples: 49561600 total_loss: 0.4718 time: 0.5352 s/iter data_time: 0.0448 s/iter total_throughput: 1913.15 samples/s lr: 9.60e-04 [09/19 11:20:08] lb.utils.events INFO: eta: 2 days, 0:48:39 iteration: 48499/375342 consumed_samples: 49664000 total_loss: 0.4712 time: 0.5353 s/iter data_time: 0.0434 s/iter total_throughput: 1913.10 samples/s lr: 9.60e-04 [09/19 11:21:03] lb.utils.events INFO: eta: 2 days, 0:50:09 iteration: 48599/375342 consumed_samples: 49766400 total_loss: 0.4684 time: 0.5353 s/iter data_time: 0.0467 s/iter total_throughput: 1913.06 samples/s lr: 9.60e-04 [09/19 11:21:57] lb.utils.events INFO: eta: 2 days, 0:51:58 iteration: 48699/375342 consumed_samples: 49868800 total_loss: 0.4654 time: 0.5353 s/iter data_time: 0.0465 s/iter total_throughput: 1913.01 samples/s lr: 9.59e-04 [09/19 11:22:51] lb.utils.events INFO: eta: 2 days, 0:53:41 iteration: 48799/375342 consumed_samples: 49971200 total_loss: 0.4687 time: 0.5353 s/iter data_time: 0.0425 s/iter total_throughput: 1912.95 samples/s lr: 9.59e-04 [09/19 11:23:45] lb.utils.events INFO: eta: 2 days, 0:56:27 iteration: 48899/375342 consumed_samples: 50073600 total_loss: 0.4674 time: 0.5353 s/iter data_time: 0.0463 s/iter total_throughput: 1912.90 samples/s lr: 9.59e-04 [09/19 11:24:40] lb.utils.events INFO: eta: 2 days, 0:58:19 iteration: 48999/375342 consumed_samples: 50176000 total_loss: 0.4545 time: 0.5353 s/iter data_time: 0.0471 s/iter total_throughput: 1912.85 samples/s lr: 9.59e-04 [09/19 11:25:34] lb.utils.events INFO: eta: 2 days, 0:59:46 iteration: 49099/375342 consumed_samples: 50278400 total_loss: 0.4623 time: 0.5353 s/iter data_time: 0.0467 s/iter total_throughput: 1912.82 samples/s lr: 9.59e-04 [09/19 11:26:28] lb.utils.events INFO: eta: 2 days, 1:00:14 iteration: 49199/375342 consumed_samples: 50380800 total_loss: 0.4764 time: 0.5353 s/iter data_time: 0.0479 s/iter total_throughput: 1912.79 samples/s lr: 9.59e-04 [09/19 11:27:22] lb.utils.events INFO: eta: 2 days, 1:00:30 iteration: 49299/375342 consumed_samples: 50483200 total_loss: 0.466 time: 0.5354 s/iter data_time: 0.0488 s/iter total_throughput: 1912.76 samples/s lr: 9.58e-04 [09/19 11:28:15] lb.utils.events INFO: eta: 2 days, 0:59:25 iteration: 49399/375342 consumed_samples: 50585600 total_loss: 0.467 time: 0.5354 s/iter data_time: 0.0483 s/iter total_throughput: 1912.74 samples/s lr: 9.58e-04 [09/19 11:29:09] lb.utils.events INFO: eta: 2 days, 0:57:52 iteration: 49499/375342 consumed_samples: 50688000 total_loss: 0.4735 time: 0.5354 s/iter data_time: 0.0470 s/iter total_throughput: 1912.71 samples/s lr: 9.58e-04 [09/19 11:30:03] lb.utils.events INFO: eta: 2 days, 0:56:38 iteration: 49599/375342 consumed_samples: 50790400 total_loss: 0.4688 time: 0.5354 s/iter data_time: 0.0406 s/iter total_throughput: 1912.68 samples/s lr: 9.58e-04 [09/19 11:30:57] lb.utils.events INFO: eta: 2 days, 0:54:42 iteration: 49699/375342 consumed_samples: 50892800 total_loss: 0.4631 time: 0.5354 s/iter data_time: 0.0421 s/iter total_throughput: 1912.64 samples/s lr: 9.58e-04 [09/19 11:31:51] lb.utils.events INFO: eta: 2 days, 0:52:06 iteration: 49799/375342 consumed_samples: 50995200 total_loss: 0.4629 time: 0.5354 s/iter data_time: 0.0443 s/iter total_throughput: 1912.61 samples/s lr: 9.58e-04 [09/19 11:32:46] lb.utils.events INFO: eta: 2 days, 0:49:21 iteration: 49899/375342 consumed_samples: 51097600 total_loss: 0.4643 time: 0.5354 s/iter data_time: 0.0470 s/iter total_throughput: 1912.58 samples/s lr: 9.57e-04 [09/19 11:33:40] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0049999 [09/19 11:33:40] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 11:33:40] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 11:33:45] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0500 s/iter. Inference: 0.2476 s/iter. Eval: 0.0025 s/iter. Total: 0.3002 s/iter. ETA=0:00:11 [09/19 11:33:50] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0709 s/iter. Inference: 0.2552 s/iter. Eval: 0.0023 s/iter. Total: 0.3286 s/iter. ETA=0:00:07 [09/19 11:33:55] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0733 s/iter. Inference: 0.2497 s/iter. Eval: 0.0023 s/iter. Total: 0.3256 s/iter. ETA=0:00:01 [09/19 11:33:57] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 11:33:57] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.048317 (0.000281 s / iter per device, on 8 devices) [09/19 11:33:57] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000221 s / iter per device, on 8 devices) [09/19 11:33:57] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 11:33:57] lb.evaluation.utils INFO: copypaste: Acc@1=67.372 [09/19 11:33:57] lb.evaluation.utils INFO: copypaste: Acc@5=88.118 [09/19 11:33:57] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 67.37200, better than last best score 66.60200 @ iteration 44999. [09/19 11:33:57] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 11:33:58] lb.utils.events INFO: eta: 2 days, 0:46:41 iteration: 49999/375342 consumed_samples: 51200000 total_loss: 0.4602 time: 0.5354 s/iter data_time: 0.0461 s/iter total_throughput: 1912.54 samples/s lr: 9.57e-04 [09/19 11:34:52] lb.utils.events INFO: eta: 2 days, 0:45:29 iteration: 50099/375342 consumed_samples: 51302400 total_loss: 0.4647 time: 0.5354 s/iter data_time: 0.0458 s/iter total_throughput: 1912.51 samples/s lr: 9.57e-04 [09/19 11:35:46] lb.utils.events INFO: eta: 2 days, 0:44:12 iteration: 50199/375342 consumed_samples: 51404800 total_loss: 0.4631 time: 0.5354 s/iter data_time: 0.0448 s/iter total_throughput: 1912.48 samples/s lr: 9.57e-04 [09/19 11:36:40] lb.utils.events INFO: eta: 2 days, 0:43:06 iteration: 50299/375342 consumed_samples: 51507200 total_loss: 0.4517 time: 0.5354 s/iter data_time: 0.0471 s/iter total_throughput: 1912.45 samples/s lr: 9.57e-04 [09/19 11:37:34] lb.utils.events INFO: eta: 2 days, 0:42:11 iteration: 50399/375342 consumed_samples: 51609600 total_loss: 0.4563 time: 0.5354 s/iter data_time: 0.0472 s/iter total_throughput: 1912.42 samples/s lr: 9.57e-04 [09/19 11:38:28] lb.utils.events INFO: eta: 2 days, 0:41:17 iteration: 50499/375342 consumed_samples: 51712000 total_loss: 0.4588 time: 0.5355 s/iter data_time: 0.0453 s/iter total_throughput: 1912.39 samples/s lr: 9.56e-04 [09/19 11:39:22] lb.utils.events INFO: eta: 2 days, 0:39:05 iteration: 50599/375342 consumed_samples: 51814400 total_loss: 0.4592 time: 0.5355 s/iter data_time: 0.0471 s/iter total_throughput: 1912.37 samples/s lr: 9.56e-04 [09/19 11:40:15] lb.utils.events INFO: eta: 2 days, 0:37:53 iteration: 50699/375342 consumed_samples: 51916800 total_loss: 0.4626 time: 0.5355 s/iter data_time: 0.0455 s/iter total_throughput: 1912.34 samples/s lr: 9.56e-04 [09/19 11:41:09] lb.utils.events INFO: eta: 2 days, 0:36:19 iteration: 50799/375342 consumed_samples: 52019200 total_loss: 0.4611 time: 0.5355 s/iter data_time: 0.0458 s/iter total_throughput: 1912.32 samples/s lr: 9.56e-04 [09/19 11:42:03] lb.utils.events INFO: eta: 2 days, 0:33:46 iteration: 50899/375342 consumed_samples: 52121600 total_loss: 0.462 time: 0.5355 s/iter data_time: 0.0447 s/iter total_throughput: 1912.31 samples/s lr: 9.56e-04 [09/19 11:42:57] lb.utils.events INFO: eta: 2 days, 0:31:22 iteration: 50999/375342 consumed_samples: 52224000 total_loss: 0.4588 time: 0.5355 s/iter data_time: 0.0460 s/iter total_throughput: 1912.29 samples/s lr: 9.56e-04 [09/19 11:43:51] lb.utils.events INFO: eta: 2 days, 0:29:33 iteration: 51099/375342 consumed_samples: 52326400 total_loss: 0.4591 time: 0.5355 s/iter data_time: 0.0464 s/iter total_throughput: 1912.28 samples/s lr: 9.55e-04 [09/19 11:44:44] lb.utils.events INFO: eta: 2 days, 0:27:26 iteration: 51199/375342 consumed_samples: 52428800 total_loss: 0.4677 time: 0.5355 s/iter data_time: 0.0448 s/iter total_throughput: 1912.27 samples/s lr: 9.55e-04 [09/19 11:45:38] lb.utils.events INFO: eta: 2 days, 0:25:10 iteration: 51299/375342 consumed_samples: 52531200 total_loss: 0.471 time: 0.5355 s/iter data_time: 0.0450 s/iter total_throughput: 1912.26 samples/s lr: 9.55e-04 [09/19 11:46:32] lb.utils.events INFO: eta: 2 days, 0:22:11 iteration: 51399/375342 consumed_samples: 52633600 total_loss: 0.4635 time: 0.5355 s/iter data_time: 0.0423 s/iter total_throughput: 1912.25 samples/s lr: 9.55e-04 [09/19 11:47:26] lb.utils.events INFO: eta: 2 days, 0:20:32 iteration: 51499/375342 consumed_samples: 52736000 total_loss: 0.4565 time: 0.5355 s/iter data_time: 0.0449 s/iter total_throughput: 1912.23 samples/s lr: 9.55e-04 [09/19 11:48:19] lb.utils.events INFO: eta: 2 days, 0:19:10 iteration: 51599/375342 consumed_samples: 52838400 total_loss: 0.4579 time: 0.5355 s/iter data_time: 0.0415 s/iter total_throughput: 1912.23 samples/s lr: 9.55e-04 [09/19 11:49:13] lb.utils.events INFO: eta: 2 days, 0:16:27 iteration: 51699/375342 consumed_samples: 52940800 total_loss: 0.4671 time: 0.5355 s/iter data_time: 0.0447 s/iter total_throughput: 1912.22 samples/s lr: 9.54e-04 [09/19 11:50:07] lb.utils.events INFO: eta: 2 days, 0:15:07 iteration: 51799/375342 consumed_samples: 53043200 total_loss: 0.4678 time: 0.5355 s/iter data_time: 0.0465 s/iter total_throughput: 1912.17 samples/s lr: 9.54e-04 [09/19 11:51:01] lb.utils.events INFO: eta: 2 days, 0:16:06 iteration: 51899/375342 consumed_samples: 53145600 total_loss: 0.4637 time: 0.5355 s/iter data_time: 0.0449 s/iter total_throughput: 1912.14 samples/s lr: 9.54e-04 [09/19 11:51:55] lb.utils.events INFO: eta: 2 days, 0:17:11 iteration: 51999/375342 consumed_samples: 53248000 total_loss: 0.4632 time: 0.5355 s/iter data_time: 0.0446 s/iter total_throughput: 1912.11 samples/s lr: 9.54e-04 [09/19 11:52:49] lb.utils.events INFO: eta: 2 days, 0:17:17 iteration: 52099/375342 consumed_samples: 53350400 total_loss: 0.4594 time: 0.5355 s/iter data_time: 0.0452 s/iter total_throughput: 1912.08 samples/s lr: 9.54e-04 [09/19 11:53:43] lb.utils.events INFO: eta: 2 days, 0:18:04 iteration: 52199/375342 consumed_samples: 53452800 total_loss: 0.4589 time: 0.5356 s/iter data_time: 0.0453 s/iter total_throughput: 1912.05 samples/s lr: 9.54e-04 [09/19 11:54:37] lb.utils.events INFO: eta: 2 days, 0:19:28 iteration: 52299/375342 consumed_samples: 53555200 total_loss: 0.4557 time: 0.5356 s/iter data_time: 0.0476 s/iter total_throughput: 1912.01 samples/s lr: 9.53e-04 [09/19 11:55:32] lb.utils.events INFO: eta: 2 days, 0:21:58 iteration: 52399/375342 consumed_samples: 53657600 total_loss: 0.446 time: 0.5356 s/iter data_time: 0.0452 s/iter total_throughput: 1911.96 samples/s lr: 9.53e-04 [09/19 11:56:26] lb.utils.events INFO: eta: 2 days, 0:23:29 iteration: 52499/375342 consumed_samples: 53760000 total_loss: 0.4634 time: 0.5356 s/iter data_time: 0.0487 s/iter total_throughput: 1911.93 samples/s lr: 9.53e-04 [09/19 11:57:20] lb.utils.events INFO: eta: 2 days, 0:23:54 iteration: 52599/375342 consumed_samples: 53862400 total_loss: 0.466 time: 0.5356 s/iter data_time: 0.0455 s/iter total_throughput: 1911.90 samples/s lr: 9.53e-04 [09/19 11:58:14] lb.utils.events INFO: eta: 2 days, 0:23:45 iteration: 52699/375342 consumed_samples: 53964800 total_loss: 0.4633 time: 0.5356 s/iter data_time: 0.0451 s/iter total_throughput: 1911.89 samples/s lr: 9.53e-04 [09/19 11:59:07] lb.utils.events INFO: eta: 2 days, 0:23:38 iteration: 52799/375342 consumed_samples: 54067200 total_loss: 0.4638 time: 0.5356 s/iter data_time: 0.0468 s/iter total_throughput: 1911.87 samples/s lr: 9.52e-04 [09/19 12:00:01] lb.utils.events INFO: eta: 2 days, 0:21:51 iteration: 52899/375342 consumed_samples: 54169600 total_loss: 0.4607 time: 0.5356 s/iter data_time: 0.0474 s/iter total_throughput: 1911.85 samples/s lr: 9.52e-04 [09/19 12:00:55] lb.utils.events INFO: eta: 2 days, 0:20:43 iteration: 52999/375342 consumed_samples: 54272000 total_loss: 0.4584 time: 0.5356 s/iter data_time: 0.0467 s/iter total_throughput: 1911.83 samples/s lr: 9.52e-04 [09/19 12:01:49] lb.utils.events INFO: eta: 2 days, 0:19:49 iteration: 53099/375342 consumed_samples: 54374400 total_loss: 0.4612 time: 0.5356 s/iter data_time: 0.0433 s/iter total_throughput: 1911.80 samples/s lr: 9.52e-04 [09/19 12:02:43] lb.utils.events INFO: eta: 2 days, 0:18:28 iteration: 53199/375342 consumed_samples: 54476800 total_loss: 0.46 time: 0.5356 s/iter data_time: 0.0452 s/iter total_throughput: 1911.78 samples/s lr: 9.52e-04 [09/19 12:03:37] lb.utils.events INFO: eta: 2 days, 0:16:01 iteration: 53299/375342 consumed_samples: 54579200 total_loss: 0.4566 time: 0.5356 s/iter data_time: 0.0445 s/iter total_throughput: 1911.76 samples/s lr: 9.52e-04 [09/19 12:04:31] lb.utils.events INFO: eta: 2 days, 0:15:09 iteration: 53399/375342 consumed_samples: 54681600 total_loss: 0.4615 time: 0.5356 s/iter data_time: 0.0473 s/iter total_throughput: 1911.72 samples/s lr: 9.51e-04 [09/19 12:05:25] lb.utils.events INFO: eta: 2 days, 0:14:23 iteration: 53499/375342 consumed_samples: 54784000 total_loss: 0.466 time: 0.5357 s/iter data_time: 0.0476 s/iter total_throughput: 1911.69 samples/s lr: 9.51e-04 [09/19 12:06:19] lb.utils.events INFO: eta: 2 days, 0:13:18 iteration: 53599/375342 consumed_samples: 54886400 total_loss: 0.4598 time: 0.5357 s/iter data_time: 0.0464 s/iter total_throughput: 1911.66 samples/s lr: 9.51e-04 [09/19 12:07:13] lb.utils.events INFO: eta: 2 days, 0:12:55 iteration: 53699/375342 consumed_samples: 54988800 total_loss: 0.4581 time: 0.5357 s/iter data_time: 0.0440 s/iter total_throughput: 1911.62 samples/s lr: 9.51e-04 [09/19 12:08:07] lb.utils.events INFO: eta: 2 days, 0:12:54 iteration: 53799/375342 consumed_samples: 55091200 total_loss: 0.4611 time: 0.5357 s/iter data_time: 0.0473 s/iter total_throughput: 1911.59 samples/s lr: 9.51e-04 [09/19 12:09:02] lb.utils.events INFO: eta: 2 days, 0:13:06 iteration: 53899/375342 consumed_samples: 55193600 total_loss: 0.463 time: 0.5357 s/iter data_time: 0.0459 s/iter total_throughput: 1911.55 samples/s lr: 9.50e-04 [09/19 12:09:56] lb.utils.events INFO: eta: 2 days, 0:12:23 iteration: 53999/375342 consumed_samples: 55296000 total_loss: 0.457 time: 0.5357 s/iter data_time: 0.0465 s/iter total_throughput: 1911.52 samples/s lr: 9.50e-04 [09/19 12:10:50] lb.utils.events INFO: eta: 2 days, 0:12:35 iteration: 54099/375342 consumed_samples: 55398400 total_loss: 0.4598 time: 0.5357 s/iter data_time: 0.0465 s/iter total_throughput: 1911.49 samples/s lr: 9.50e-04 [09/19 12:11:44] lb.utils.events INFO: eta: 2 days, 0:11:47 iteration: 54199/375342 consumed_samples: 55500800 total_loss: 0.4608 time: 0.5357 s/iter data_time: 0.0475 s/iter total_throughput: 1911.46 samples/s lr: 9.50e-04 [09/19 12:12:38] lb.utils.events INFO: eta: 2 days, 0:11:22 iteration: 54299/375342 consumed_samples: 55603200 total_loss: 0.4584 time: 0.5357 s/iter data_time: 0.0470 s/iter total_throughput: 1911.43 samples/s lr: 9.50e-04 [09/19 12:13:32] lb.utils.events INFO: eta: 2 days, 0:08:40 iteration: 54399/375342 consumed_samples: 55705600 total_loss: 0.4619 time: 0.5357 s/iter data_time: 0.0479 s/iter total_throughput: 1911.42 samples/s lr: 9.50e-04 [09/19 12:14:25] lb.utils.events INFO: eta: 2 days, 0:06:37 iteration: 54499/375342 consumed_samples: 55808000 total_loss: 0.4543 time: 0.5357 s/iter data_time: 0.0466 s/iter total_throughput: 1911.40 samples/s lr: 9.49e-04 [09/19 12:15:19] lb.utils.events INFO: eta: 2 days, 0:04:49 iteration: 54599/375342 consumed_samples: 55910400 total_loss: 0.4525 time: 0.5357 s/iter data_time: 0.0454 s/iter total_throughput: 1911.38 samples/s lr: 9.49e-04 [09/19 12:16:13] lb.utils.events INFO: eta: 2 days, 0:03:01 iteration: 54699/375342 consumed_samples: 56012800 total_loss: 0.4547 time: 0.5357 s/iter data_time: 0.0464 s/iter total_throughput: 1911.37 samples/s lr: 9.49e-04 [09/19 12:17:07] lb.utils.events INFO: eta: 2 days, 0:00:02 iteration: 54799/375342 consumed_samples: 56115200 total_loss: 0.4607 time: 0.5357 s/iter data_time: 0.0468 s/iter total_throughput: 1911.35 samples/s lr: 9.49e-04 [09/19 12:18:01] lb.utils.events INFO: eta: 1 day, 23:58:12 iteration: 54899/375342 consumed_samples: 56217600 total_loss: 0.4658 time: 0.5358 s/iter data_time: 0.0447 s/iter total_throughput: 1911.32 samples/s lr: 9.49e-04 [09/19 12:18:55] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0054999 [09/19 12:18:55] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 12:18:55] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 12:19:00] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0497 s/iter. Inference: 0.2557 s/iter. Eval: 0.0026 s/iter. Total: 0.3080 s/iter. ETA=0:00:11 [09/19 12:19:05] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0744 s/iter. Inference: 0.2567 s/iter. Eval: 0.0026 s/iter. Total: 0.3339 s/iter. ETA=0:00:07 [09/19 12:19:10] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0752 s/iter. Inference: 0.2522 s/iter. Eval: 0.0025 s/iter. Total: 0.3302 s/iter. ETA=0:00:01 [09/19 12:19:12] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 12:19:12] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.216472 (0.000284 s / iter per device, on 8 devices) [09/19 12:19:12] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000220 s / iter per device, on 8 devices) [09/19 12:19:12] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 12:19:12] lb.evaluation.utils INFO: copypaste: Acc@1=68.026 [09/19 12:19:12] lb.evaluation.utils INFO: copypaste: Acc@5=88.416 [09/19 12:19:12] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 68.02600, better than last best score 67.37200 @ iteration 49999. [09/19 12:19:12] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 12:19:13] lb.utils.events INFO: eta: 1 day, 23:56:49 iteration: 54999/375342 consumed_samples: 56320000 total_loss: 0.4665 time: 0.5358 s/iter data_time: 0.0457 s/iter total_throughput: 1911.30 samples/s lr: 9.48e-04 [09/19 12:20:07] lb.utils.events INFO: eta: 1 day, 23:54:45 iteration: 55099/375342 consumed_samples: 56422400 total_loss: 0.4688 time: 0.5358 s/iter data_time: 0.0435 s/iter total_throughput: 1911.29 samples/s lr: 9.48e-04 [09/19 12:21:01] lb.utils.events INFO: eta: 1 day, 23:51:59 iteration: 55199/375342 consumed_samples: 56524800 total_loss: 0.4638 time: 0.5358 s/iter data_time: 0.0427 s/iter total_throughput: 1911.25 samples/s lr: 9.48e-04 [09/19 12:21:55] lb.utils.events INFO: eta: 1 day, 23:51:44 iteration: 55299/375342 consumed_samples: 56627200 total_loss: 0.4561 time: 0.5358 s/iter data_time: 0.0474 s/iter total_throughput: 1911.21 samples/s lr: 9.48e-04 [09/19 12:22:49] lb.utils.events INFO: eta: 1 day, 23:52:03 iteration: 55399/375342 consumed_samples: 56729600 total_loss: 0.4538 time: 0.5358 s/iter data_time: 0.0482 s/iter total_throughput: 1911.17 samples/s lr: 9.48e-04 [09/19 12:23:44] lb.utils.events INFO: eta: 1 day, 23:51:55 iteration: 55499/375342 consumed_samples: 56832000 total_loss: 0.4525 time: 0.5358 s/iter data_time: 0.0475 s/iter total_throughput: 1911.14 samples/s lr: 9.48e-04 [09/19 12:24:38] lb.utils.events INFO: eta: 1 day, 23:53:23 iteration: 55599/375342 consumed_samples: 56934400 total_loss: 0.4531 time: 0.5358 s/iter data_time: 0.0483 s/iter total_throughput: 1911.11 samples/s lr: 9.47e-04 [09/19 12:25:32] lb.utils.events INFO: eta: 1 day, 23:55:48 iteration: 55699/375342 consumed_samples: 57036800 total_loss: 0.4544 time: 0.5358 s/iter data_time: 0.0463 s/iter total_throughput: 1911.06 samples/s lr: 9.47e-04 [09/19 12:26:26] lb.utils.events INFO: eta: 1 day, 23:59:21 iteration: 55799/375342 consumed_samples: 57139200 total_loss: 0.461 time: 0.5358 s/iter data_time: 0.0486 s/iter total_throughput: 1911.01 samples/s lr: 9.47e-04 [09/19 12:27:21] lb.utils.events INFO: eta: 2 days, 0:00:08 iteration: 55899/375342 consumed_samples: 57241600 total_loss: 0.4636 time: 0.5359 s/iter data_time: 0.0485 s/iter total_throughput: 1910.97 samples/s lr: 9.47e-04 [09/19 12:28:15] lb.utils.events INFO: eta: 2 days, 0:01:32 iteration: 55999/375342 consumed_samples: 57344000 total_loss: 0.4605 time: 0.5359 s/iter data_time: 0.0487 s/iter total_throughput: 1910.93 samples/s lr: 9.47e-04 [09/19 12:29:09] lb.utils.events INFO: eta: 2 days, 0:01:06 iteration: 56099/375342 consumed_samples: 57446400 total_loss: 0.4603 time: 0.5359 s/iter data_time: 0.0479 s/iter total_throughput: 1910.91 samples/s lr: 9.46e-04 [09/19 12:30:03] lb.utils.events INFO: eta: 2 days, 0:01:30 iteration: 56199/375342 consumed_samples: 57548800 total_loss: 0.465 time: 0.5359 s/iter data_time: 0.0487 s/iter total_throughput: 1910.89 samples/s lr: 9.46e-04 [09/19 12:30:57] lb.utils.events INFO: eta: 1 day, 23:59:28 iteration: 56299/375342 consumed_samples: 57651200 total_loss: 0.4651 time: 0.5359 s/iter data_time: 0.0491 s/iter total_throughput: 1910.86 samples/s lr: 9.46e-04 [09/19 12:31:51] lb.utils.events INFO: eta: 1 day, 23:58:06 iteration: 56399/375342 consumed_samples: 57753600 total_loss: 0.4557 time: 0.5359 s/iter data_time: 0.0444 s/iter total_throughput: 1910.84 samples/s lr: 9.46e-04 [09/19 12:32:45] lb.utils.events INFO: eta: 1 day, 23:56:43 iteration: 56499/375342 consumed_samples: 57856000 total_loss: 0.4554 time: 0.5359 s/iter data_time: 0.0448 s/iter total_throughput: 1910.81 samples/s lr: 9.46e-04 [09/19 12:33:39] lb.utils.events INFO: eta: 1 day, 23:53:28 iteration: 56599/375342 consumed_samples: 57958400 total_loss: 0.4601 time: 0.5359 s/iter data_time: 0.0438 s/iter total_throughput: 1910.79 samples/s lr: 9.45e-04 [09/19 12:34:33] lb.utils.events INFO: eta: 1 day, 23:51:50 iteration: 56699/375342 consumed_samples: 58060800 total_loss: 0.4613 time: 0.5359 s/iter data_time: 0.0450 s/iter total_throughput: 1910.76 samples/s lr: 9.45e-04 [09/19 12:35:27] lb.utils.events INFO: eta: 1 day, 23:49:29 iteration: 56799/375342 consumed_samples: 58163200 total_loss: 0.4632 time: 0.5359 s/iter data_time: 0.0450 s/iter total_throughput: 1910.73 samples/s lr: 9.45e-04 [09/19 12:36:21] lb.utils.events INFO: eta: 1 day, 23:47:36 iteration: 56899/375342 consumed_samples: 58265600 total_loss: 0.4536 time: 0.5359 s/iter data_time: 0.0458 s/iter total_throughput: 1910.70 samples/s lr: 9.45e-04 [09/19 12:37:15] lb.utils.events INFO: eta: 1 day, 23:46:42 iteration: 56999/375342 consumed_samples: 58368000 total_loss: 0.455 time: 0.5359 s/iter data_time: 0.0459 s/iter total_throughput: 1910.66 samples/s lr: 9.45e-04 [09/19 12:38:09] lb.utils.events INFO: eta: 1 day, 23:45:51 iteration: 57099/375342 consumed_samples: 58470400 total_loss: 0.4627 time: 0.5359 s/iter data_time: 0.0474 s/iter total_throughput: 1910.64 samples/s lr: 9.45e-04 [09/19 12:39:03] lb.utils.events INFO: eta: 1 day, 23:44:35 iteration: 57199/375342 consumed_samples: 58572800 total_loss: 0.4653 time: 0.5360 s/iter data_time: 0.0435 s/iter total_throughput: 1910.62 samples/s lr: 9.44e-04 [09/19 12:39:57] lb.utils.events INFO: eta: 1 day, 23:43:56 iteration: 57299/375342 consumed_samples: 58675200 total_loss: 0.461 time: 0.5360 s/iter data_time: 0.0472 s/iter total_throughput: 1910.58 samples/s lr: 9.44e-04 [09/19 12:40:51] lb.utils.events INFO: eta: 1 day, 23:43:04 iteration: 57399/375342 consumed_samples: 58777600 total_loss: 0.4598 time: 0.5360 s/iter data_time: 0.0471 s/iter total_throughput: 1910.56 samples/s lr: 9.44e-04 [09/19 12:41:45] lb.utils.events INFO: eta: 1 day, 23:42:09 iteration: 57499/375342 consumed_samples: 58880000 total_loss: 0.4636 time: 0.5360 s/iter data_time: 0.0466 s/iter total_throughput: 1910.54 samples/s lr: 9.44e-04 [09/19 12:42:39] lb.utils.events INFO: eta: 1 day, 23:41:13 iteration: 57599/375342 consumed_samples: 58982400 total_loss: 0.4649 time: 0.5360 s/iter data_time: 0.0462 s/iter total_throughput: 1910.51 samples/s lr: 9.44e-04 [09/19 12:43:33] lb.utils.events INFO: eta: 1 day, 23:38:38 iteration: 57699/375342 consumed_samples: 59084800 total_loss: 0.4565 time: 0.5360 s/iter data_time: 0.0465 s/iter total_throughput: 1910.49 samples/s lr: 9.43e-04 [09/19 12:44:27] lb.utils.events INFO: eta: 1 day, 23:36:33 iteration: 57799/375342 consumed_samples: 59187200 total_loss: 0.4503 time: 0.5360 s/iter data_time: 0.0475 s/iter total_throughput: 1910.47 samples/s lr: 9.43e-04 [09/19 12:45:21] lb.utils.events INFO: eta: 1 day, 23:34:39 iteration: 57899/375342 consumed_samples: 59289600 total_loss: 0.4571 time: 0.5360 s/iter data_time: 0.0454 s/iter total_throughput: 1910.46 samples/s lr: 9.43e-04 [09/19 12:46:15] lb.utils.events INFO: eta: 1 day, 23:32:56 iteration: 57999/375342 consumed_samples: 59392000 total_loss: 0.4631 time: 0.5360 s/iter data_time: 0.0458 s/iter total_throughput: 1910.44 samples/s lr: 9.43e-04 [09/19 12:47:09] lb.utils.events INFO: eta: 1 day, 23:31:26 iteration: 58099/375342 consumed_samples: 59494400 total_loss: 0.4631 time: 0.5360 s/iter data_time: 0.0462 s/iter total_throughput: 1910.42 samples/s lr: 9.43e-04 [09/19 12:48:03] lb.utils.events INFO: eta: 1 day, 23:29:54 iteration: 58199/375342 consumed_samples: 59596800 total_loss: 0.4586 time: 0.5360 s/iter data_time: 0.0470 s/iter total_throughput: 1910.41 samples/s lr: 9.42e-04 [09/19 12:48:57] lb.utils.events INFO: eta: 1 day, 23:27:55 iteration: 58299/375342 consumed_samples: 59699200 total_loss: 0.458 time: 0.5360 s/iter data_time: 0.0434 s/iter total_throughput: 1910.40 samples/s lr: 9.42e-04 [09/19 12:49:51] lb.utils.events INFO: eta: 1 day, 23:26:47 iteration: 58399/375342 consumed_samples: 59801600 total_loss: 0.4539 time: 0.5360 s/iter data_time: 0.0435 s/iter total_throughput: 1910.38 samples/s lr: 9.42e-04 [09/19 12:50:45] lb.utils.events INFO: eta: 1 day, 23:25:29 iteration: 58499/375342 consumed_samples: 59904000 total_loss: 0.4513 time: 0.5360 s/iter data_time: 0.0468 s/iter total_throughput: 1910.36 samples/s lr: 9.42e-04 [09/19 12:51:38] lb.utils.events INFO: eta: 1 day, 23:24:50 iteration: 58599/375342 consumed_samples: 60006400 total_loss: 0.4528 time: 0.5360 s/iter data_time: 0.0459 s/iter total_throughput: 1910.34 samples/s lr: 9.42e-04 [09/19 12:52:33] lb.utils.events INFO: eta: 1 day, 23:23:38 iteration: 58699/375342 consumed_samples: 60108800 total_loss: 0.4466 time: 0.5360 s/iter data_time: 0.0473 s/iter total_throughput: 1910.29 samples/s lr: 9.41e-04 [09/19 12:53:27] lb.utils.events INFO: eta: 1 day, 23:23:44 iteration: 58799/375342 consumed_samples: 60211200 total_loss: 0.4571 time: 0.5361 s/iter data_time: 0.0483 s/iter total_throughput: 1910.27 samples/s lr: 9.41e-04 [09/19 12:54:21] lb.utils.events INFO: eta: 1 day, 23:22:58 iteration: 58899/375342 consumed_samples: 60313600 total_loss: 0.464 time: 0.5361 s/iter data_time: 0.0470 s/iter total_throughput: 1910.25 samples/s lr: 9.41e-04 [09/19 12:55:15] lb.utils.events INFO: eta: 1 day, 23:22:39 iteration: 58999/375342 consumed_samples: 60416000 total_loss: 0.4498 time: 0.5361 s/iter data_time: 0.0472 s/iter total_throughput: 1910.22 samples/s lr: 9.41e-04 [09/19 12:56:09] lb.utils.events INFO: eta: 1 day, 23:22:25 iteration: 59099/375342 consumed_samples: 60518400 total_loss: 0.4518 time: 0.5361 s/iter data_time: 0.0491 s/iter total_throughput: 1910.20 samples/s lr: 9.41e-04 [09/19 12:57:03] lb.utils.events INFO: eta: 1 day, 23:23:27 iteration: 59199/375342 consumed_samples: 60620800 total_loss: 0.4579 time: 0.5361 s/iter data_time: 0.0479 s/iter total_throughput: 1910.17 samples/s lr: 9.40e-04 [09/19 12:57:57] lb.utils.events INFO: eta: 1 day, 23:25:44 iteration: 59299/375342 consumed_samples: 60723200 total_loss: 0.4522 time: 0.5361 s/iter data_time: 0.0497 s/iter total_throughput: 1910.14 samples/s lr: 9.40e-04 [09/19 12:58:51] lb.utils.events INFO: eta: 1 day, 23:25:27 iteration: 59399/375342 consumed_samples: 60825600 total_loss: 0.4549 time: 0.5361 s/iter data_time: 0.0483 s/iter total_throughput: 1910.11 samples/s lr: 9.40e-04 [09/19 12:59:45] lb.utils.events INFO: eta: 1 day, 23:25:10 iteration: 59499/375342 consumed_samples: 60928000 total_loss: 0.4523 time: 0.5361 s/iter data_time: 0.0491 s/iter total_throughput: 1910.08 samples/s lr: 9.40e-04 [09/19 13:00:40] lb.utils.events INFO: eta: 1 day, 23:24:50 iteration: 59599/375342 consumed_samples: 61030400 total_loss: 0.4461 time: 0.5361 s/iter data_time: 0.0487 s/iter total_throughput: 1910.06 samples/s lr: 9.40e-04 [09/19 13:01:34] lb.utils.events INFO: eta: 1 day, 23:24:02 iteration: 59699/375342 consumed_samples: 61132800 total_loss: 0.4499 time: 0.5361 s/iter data_time: 0.0479 s/iter total_throughput: 1910.03 samples/s lr: 9.39e-04 [09/19 13:02:28] lb.utils.events INFO: eta: 1 day, 23:23:56 iteration: 59799/375342 consumed_samples: 61235200 total_loss: 0.4482 time: 0.5361 s/iter data_time: 0.0462 s/iter total_throughput: 1910.01 samples/s lr: 9.39e-04 [09/19 13:03:22] lb.utils.events INFO: eta: 1 day, 23:22:38 iteration: 59899/375342 consumed_samples: 61337600 total_loss: 0.4509 time: 0.5361 s/iter data_time: 0.0478 s/iter total_throughput: 1909.99 samples/s lr: 9.39e-04 [09/19 13:04:16] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0059999 [09/19 13:04:16] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 13:04:16] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 13:04:21] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0735 s/iter. Inference: 0.2465 s/iter. Eval: 0.0029 s/iter. Total: 0.3229 s/iter. ETA=0:00:11 [09/19 13:04:26] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0841 s/iter. Inference: 0.2509 s/iter. Eval: 0.0024 s/iter. Total: 0.3376 s/iter. ETA=0:00:07 [09/19 13:04:31] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0802 s/iter. Inference: 0.2502 s/iter. Eval: 0.0023 s/iter. Total: 0.3330 s/iter. ETA=0:00:01 [09/19 13:04:33] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 13:04:33] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.313122 (0.000286 s / iter per device, on 8 devices) [09/19 13:04:33] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000218 s / iter per device, on 8 devices) [09/19 13:04:33] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 13:04:33] lb.evaluation.utils INFO: copypaste: Acc@1=69.054 [09/19 13:04:33] lb.evaluation.utils INFO: copypaste: Acc@5=89.32 [09/19 13:04:33] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 69.05400, better than last best score 68.02600 @ iteration 54999. [09/19 13:04:33] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 13:04:34] lb.utils.events INFO: eta: 1 day, 23:22:27 iteration: 59999/375342 consumed_samples: 61440000 total_loss: 0.4621 time: 0.5361 s/iter data_time: 0.0473 s/iter total_throughput: 1909.96 samples/s lr: 9.39e-04 [09/19 13:05:28] lb.utils.events INFO: eta: 1 day, 23:21:33 iteration: 60099/375342 consumed_samples: 61542400 total_loss: 0.4554 time: 0.5361 s/iter data_time: 0.0449 s/iter total_throughput: 1909.94 samples/s lr: 9.39e-04 [09/19 13:06:22] lb.utils.events INFO: eta: 1 day, 23:20:07 iteration: 60199/375342 consumed_samples: 61644800 total_loss: 0.453 time: 0.5362 s/iter data_time: 0.0455 s/iter total_throughput: 1909.91 samples/s lr: 9.38e-04 [09/19 13:07:16] lb.utils.events INFO: eta: 1 day, 23:18:40 iteration: 60299/375342 consumed_samples: 61747200 total_loss: 0.4574 time: 0.5362 s/iter data_time: 0.0477 s/iter total_throughput: 1909.88 samples/s lr: 9.38e-04 [09/19 13:08:10] lb.utils.events INFO: eta: 1 day, 23:17:35 iteration: 60399/375342 consumed_samples: 61849600 total_loss: 0.4582 time: 0.5362 s/iter data_time: 0.0461 s/iter total_throughput: 1909.85 samples/s lr: 9.38e-04 [09/19 13:09:04] lb.utils.events INFO: eta: 1 day, 23:16:44 iteration: 60499/375342 consumed_samples: 61952000 total_loss: 0.4608 time: 0.5362 s/iter data_time: 0.0483 s/iter total_throughput: 1909.83 samples/s lr: 9.38e-04 [09/19 13:09:58] lb.utils.events INFO: eta: 1 day, 23:15:51 iteration: 60599/375342 consumed_samples: 62054400 total_loss: 0.4581 time: 0.5362 s/iter data_time: 0.0451 s/iter total_throughput: 1909.80 samples/s lr: 9.38e-04 [09/19 13:10:53] lb.utils.events INFO: eta: 1 day, 23:15:21 iteration: 60699/375342 consumed_samples: 62156800 total_loss: 0.4513 time: 0.5362 s/iter data_time: 0.0474 s/iter total_throughput: 1909.77 samples/s lr: 9.37e-04 [09/19 13:11:47] lb.utils.events INFO: eta: 1 day, 23:14:44 iteration: 60799/375342 consumed_samples: 62259200 total_loss: 0.455 time: 0.5362 s/iter data_time: 0.0464 s/iter total_throughput: 1909.75 samples/s lr: 9.37e-04 [09/19 13:12:41] lb.utils.events INFO: eta: 1 day, 23:14:12 iteration: 60899/375342 consumed_samples: 62361600 total_loss: 0.4562 time: 0.5362 s/iter data_time: 0.0487 s/iter total_throughput: 1909.72 samples/s lr: 9.37e-04 [09/19 13:13:35] lb.utils.events INFO: eta: 1 day, 23:13:10 iteration: 60999/375342 consumed_samples: 62464000 total_loss: 0.4561 time: 0.5362 s/iter data_time: 0.0460 s/iter total_throughput: 1909.70 samples/s lr: 9.37e-04 [09/19 13:14:29] lb.utils.events INFO: eta: 1 day, 23:11:46 iteration: 61099/375342 consumed_samples: 62566400 total_loss: 0.4561 time: 0.5362 s/iter data_time: 0.0480 s/iter total_throughput: 1909.67 samples/s lr: 9.37e-04 [09/19 13:15:23] lb.utils.events INFO: eta: 1 day, 23:10:52 iteration: 61199/375342 consumed_samples: 62668800 total_loss: 0.451 time: 0.5362 s/iter data_time: 0.0481 s/iter total_throughput: 1909.64 samples/s lr: 9.36e-04 [09/19 13:16:17] lb.utils.events INFO: eta: 1 day, 23:09:52 iteration: 61299/375342 consumed_samples: 62771200 total_loss: 0.4504 time: 0.5362 s/iter data_time: 0.0469 s/iter total_throughput: 1909.62 samples/s lr: 9.36e-04 [09/19 13:17:11] lb.utils.events INFO: eta: 1 day, 23:08:11 iteration: 61399/375342 consumed_samples: 62873600 total_loss: 0.4545 time: 0.5362 s/iter data_time: 0.0470 s/iter total_throughput: 1909.60 samples/s lr: 9.36e-04 [09/19 13:18:05] lb.utils.events INFO: eta: 1 day, 23:06:26 iteration: 61499/375342 consumed_samples: 62976000 total_loss: 0.4549 time: 0.5362 s/iter data_time: 0.0474 s/iter total_throughput: 1909.58 samples/s lr: 9.36e-04 [09/19 13:18:59] lb.utils.events INFO: eta: 1 day, 23:04:38 iteration: 61599/375342 consumed_samples: 63078400 total_loss: 0.4538 time: 0.5362 s/iter data_time: 0.0468 s/iter total_throughput: 1909.57 samples/s lr: 9.36e-04 [09/19 13:19:53] lb.utils.events INFO: eta: 1 day, 23:02:28 iteration: 61699/375342 consumed_samples: 63180800 total_loss: 0.4526 time: 0.5363 s/iter data_time: 0.0483 s/iter total_throughput: 1909.56 samples/s lr: 9.35e-04 [09/19 13:20:47] lb.utils.events INFO: eta: 1 day, 22:59:59 iteration: 61799/375342 consumed_samples: 63283200 total_loss: 0.45 time: 0.5363 s/iter data_time: 0.0439 s/iter total_throughput: 1909.54 samples/s lr: 9.35e-04 [09/19 13:21:41] lb.utils.events INFO: eta: 1 day, 22:58:07 iteration: 61899/375342 consumed_samples: 63385600 total_loss: 0.4493 time: 0.5363 s/iter data_time: 0.0431 s/iter total_throughput: 1909.52 samples/s lr: 9.35e-04 [09/19 13:22:35] lb.utils.events INFO: eta: 1 day, 22:56:41 iteration: 61999/375342 consumed_samples: 63488000 total_loss: 0.4511 time: 0.5363 s/iter data_time: 0.0448 s/iter total_throughput: 1909.51 samples/s lr: 9.35e-04 [09/19 13:23:29] lb.utils.events INFO: eta: 1 day, 22:55:40 iteration: 62099/375342 consumed_samples: 63590400 total_loss: 0.4491 time: 0.5363 s/iter data_time: 0.0489 s/iter total_throughput: 1909.47 samples/s lr: 9.35e-04 [09/19 13:24:23] lb.utils.events INFO: eta: 1 day, 22:54:32 iteration: 62199/375342 consumed_samples: 63692800 total_loss: 0.4529 time: 0.5363 s/iter data_time: 0.0488 s/iter total_throughput: 1909.45 samples/s lr: 9.34e-04 [09/19 13:25:17] lb.utils.events INFO: eta: 1 day, 22:53:33 iteration: 62299/375342 consumed_samples: 63795200 total_loss: 0.451 time: 0.5363 s/iter data_time: 0.0487 s/iter total_throughput: 1909.42 samples/s lr: 9.34e-04 [09/19 13:26:11] lb.utils.events INFO: eta: 1 day, 22:53:19 iteration: 62399/375342 consumed_samples: 63897600 total_loss: 0.4471 time: 0.5363 s/iter data_time: 0.0480 s/iter total_throughput: 1909.40 samples/s lr: 9.34e-04 [09/19 13:27:05] lb.utils.events INFO: eta: 1 day, 22:53:43 iteration: 62499/375342 consumed_samples: 64000000 total_loss: 0.4569 time: 0.5363 s/iter data_time: 0.0506 s/iter total_throughput: 1909.38 samples/s lr: 9.34e-04 [09/19 13:27:59] lb.utils.events INFO: eta: 1 day, 22:54:25 iteration: 62599/375342 consumed_samples: 64102400 total_loss: 0.4549 time: 0.5363 s/iter data_time: 0.0516 s/iter total_throughput: 1909.36 samples/s lr: 9.34e-04 [09/19 13:28:53] lb.utils.events INFO: eta: 1 day, 22:54:55 iteration: 62699/375342 consumed_samples: 64204800 total_loss: 0.4508 time: 0.5363 s/iter data_time: 0.0504 s/iter total_throughput: 1909.33 samples/s lr: 9.33e-04 [09/19 13:29:47] lb.utils.events INFO: eta: 1 day, 22:55:48 iteration: 62799/375342 consumed_samples: 64307200 total_loss: 0.4462 time: 0.5363 s/iter data_time: 0.0511 s/iter total_throughput: 1909.30 samples/s lr: 9.33e-04 [09/19 13:30:42] lb.utils.events INFO: eta: 1 day, 22:56:25 iteration: 62899/375342 consumed_samples: 64409600 total_loss: 0.4464 time: 0.5363 s/iter data_time: 0.0486 s/iter total_throughput: 1909.27 samples/s lr: 9.33e-04 [09/19 13:31:36] lb.utils.events INFO: eta: 1 day, 22:55:59 iteration: 62999/375342 consumed_samples: 64512000 total_loss: 0.4496 time: 0.5363 s/iter data_time: 0.0498 s/iter total_throughput: 1909.25 samples/s lr: 9.33e-04 [09/19 13:32:30] lb.utils.events INFO: eta: 1 day, 22:55:26 iteration: 63099/375342 consumed_samples: 64614400 total_loss: 0.4529 time: 0.5363 s/iter data_time: 0.0495 s/iter total_throughput: 1909.23 samples/s lr: 9.33e-04 [09/19 13:33:24] lb.utils.events INFO: eta: 1 day, 22:54:13 iteration: 63199/375342 consumed_samples: 64716800 total_loss: 0.448 time: 0.5363 s/iter data_time: 0.0473 s/iter total_throughput: 1909.21 samples/s lr: 9.32e-04 [09/19 13:34:18] lb.utils.events INFO: eta: 1 day, 22:52:48 iteration: 63299/375342 consumed_samples: 64819200 total_loss: 0.4482 time: 0.5364 s/iter data_time: 0.0495 s/iter total_throughput: 1909.19 samples/s lr: 9.32e-04 [09/19 13:35:12] lb.utils.events INFO: eta: 1 day, 22:52:43 iteration: 63399/375342 consumed_samples: 64921600 total_loss: 0.4515 time: 0.5364 s/iter data_time: 0.0450 s/iter total_throughput: 1909.16 samples/s lr: 9.32e-04 [09/19 13:36:06] lb.utils.events INFO: eta: 1 day, 22:52:06 iteration: 63499/375342 consumed_samples: 65024000 total_loss: 0.452 time: 0.5364 s/iter data_time: 0.0447 s/iter total_throughput: 1909.13 samples/s lr: 9.32e-04 [09/19 13:37:00] lb.utils.events INFO: eta: 1 day, 22:51:50 iteration: 63599/375342 consumed_samples: 65126400 total_loss: 0.4547 time: 0.5364 s/iter data_time: 0.0467 s/iter total_throughput: 1909.11 samples/s lr: 9.32e-04 [09/19 13:37:54] lb.utils.events INFO: eta: 1 day, 22:50:05 iteration: 63699/375342 consumed_samples: 65228800 total_loss: 0.4532 time: 0.5364 s/iter data_time: 0.0458 s/iter total_throughput: 1909.08 samples/s lr: 9.31e-04 [09/19 13:38:48] lb.utils.events INFO: eta: 1 day, 22:48:18 iteration: 63799/375342 consumed_samples: 65331200 total_loss: 0.4541 time: 0.5364 s/iter data_time: 0.0467 s/iter total_throughput: 1909.06 samples/s lr: 9.31e-04 [09/19 13:39:43] lb.utils.events INFO: eta: 1 day, 22:47:12 iteration: 63899/375342 consumed_samples: 65433600 total_loss: 0.4514 time: 0.5364 s/iter data_time: 0.0465 s/iter total_throughput: 1909.03 samples/s lr: 9.31e-04 [09/19 13:40:37] lb.utils.events INFO: eta: 1 day, 22:46:01 iteration: 63999/375342 consumed_samples: 65536000 total_loss: 0.4487 time: 0.5364 s/iter data_time: 0.0477 s/iter total_throughput: 1909.01 samples/s lr: 9.31e-04 [09/19 13:41:31] lb.utils.events INFO: eta: 1 day, 22:44:43 iteration: 64099/375342 consumed_samples: 65638400 total_loss: 0.452 time: 0.5364 s/iter data_time: 0.0455 s/iter total_throughput: 1908.98 samples/s lr: 9.30e-04 [09/19 13:42:25] lb.utils.events INFO: eta: 1 day, 22:44:19 iteration: 64199/375342 consumed_samples: 65740800 total_loss: 0.4534 time: 0.5364 s/iter data_time: 0.0472 s/iter total_throughput: 1908.96 samples/s lr: 9.30e-04 [09/19 13:43:19] lb.utils.events INFO: eta: 1 day, 22:43:35 iteration: 64299/375342 consumed_samples: 65843200 total_loss: 0.4546 time: 0.5364 s/iter data_time: 0.0467 s/iter total_throughput: 1908.94 samples/s lr: 9.30e-04 [09/19 13:44:13] lb.utils.events INFO: eta: 1 day, 22:41:45 iteration: 64399/375342 consumed_samples: 65945600 total_loss: 0.4467 time: 0.5364 s/iter data_time: 0.0479 s/iter total_throughput: 1908.92 samples/s lr: 9.30e-04 [09/19 13:45:07] lb.utils.events INFO: eta: 1 day, 22:40:36 iteration: 64499/375342 consumed_samples: 66048000 total_loss: 0.4437 time: 0.5364 s/iter data_time: 0.0484 s/iter total_throughput: 1908.90 samples/s lr: 9.30e-04 [09/19 13:46:01] lb.utils.events INFO: eta: 1 day, 22:38:19 iteration: 64599/375342 consumed_samples: 66150400 total_loss: 0.4477 time: 0.5364 s/iter data_time: 0.0473 s/iter total_throughput: 1908.88 samples/s lr: 9.29e-04 [09/19 13:46:55] lb.utils.events INFO: eta: 1 day, 22:36:23 iteration: 64699/375342 consumed_samples: 66252800 total_loss: 0.4571 time: 0.5364 s/iter data_time: 0.0479 s/iter total_throughput: 1908.87 samples/s lr: 9.29e-04 [09/19 13:47:49] lb.utils.events INFO: eta: 1 day, 22:34:58 iteration: 64799/375342 consumed_samples: 66355200 total_loss: 0.4618 time: 0.5364 s/iter data_time: 0.0458 s/iter total_throughput: 1908.86 samples/s lr: 9.29e-04 [09/19 13:48:43] lb.utils.events INFO: eta: 1 day, 22:33:26 iteration: 64899/375342 consumed_samples: 66457600 total_loss: 0.4493 time: 0.5365 s/iter data_time: 0.0483 s/iter total_throughput: 1908.84 samples/s lr: 9.29e-04 [09/19 13:49:36] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0064999 [09/19 13:49:37] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 13:49:37] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 13:49:42] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0569 s/iter. Inference: 0.2441 s/iter. Eval: 0.0025 s/iter. Total: 0.3034 s/iter. ETA=0:00:11 [09/19 13:49:47] lb.evaluation.evaluator INFO: Inference done 25600/50000. Dataloading: 0.0646 s/iter. Inference: 0.2738 s/iter. Eval: 0.0024 s/iter. Total: 0.3410 s/iter. ETA=0:00:07 [09/19 13:49:52] lb.evaluation.evaluator INFO: Inference done 41984/50000. Dataloading: 0.0673 s/iter. Inference: 0.2615 s/iter. Eval: 0.0023 s/iter. Total: 0.3315 s/iter. ETA=0:00:02 [09/19 13:49:54] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 13:49:54] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.248404 (0.000285 s / iter per device, on 8 devices) [09/19 13:49:54] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000228 s / iter per device, on 8 devices) [09/19 13:49:54] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 13:49:54] lb.evaluation.utils INFO: copypaste: Acc@1=69.152 [09/19 13:49:54] lb.evaluation.utils INFO: copypaste: Acc@5=89.568 [09/19 13:49:54] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 69.15200, better than last best score 69.05400 @ iteration 59999. [09/19 13:49:54] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 13:49:55] lb.utils.events INFO: eta: 1 day, 22:30:29 iteration: 64999/375342 consumed_samples: 66560000 total_loss: 0.4535 time: 0.5365 s/iter data_time: 0.0477 s/iter total_throughput: 1908.83 samples/s lr: 9.29e-04 [09/19 13:50:49] lb.utils.events INFO: eta: 1 day, 22:29:03 iteration: 65099/375342 consumed_samples: 66662400 total_loss: 0.4566 time: 0.5365 s/iter data_time: 0.0476 s/iter total_throughput: 1908.82 samples/s lr: 9.28e-04 [09/19 13:51:43] lb.utils.events INFO: eta: 1 day, 22:26:58 iteration: 65199/375342 consumed_samples: 66764800 total_loss: 0.4488 time: 0.5365 s/iter data_time: 0.0446 s/iter total_throughput: 1908.81 samples/s lr: 9.28e-04 [09/19 13:52:36] lb.utils.events INFO: eta: 1 day, 22:25:21 iteration: 65299/375342 consumed_samples: 66867200 total_loss: 0.4475 time: 0.5365 s/iter data_time: 0.0447 s/iter total_throughput: 1908.79 samples/s lr: 9.28e-04 [09/19 13:53:30] lb.utils.events INFO: eta: 1 day, 22:23:52 iteration: 65399/375342 consumed_samples: 66969600 total_loss: 0.4506 time: 0.5365 s/iter data_time: 0.0435 s/iter total_throughput: 1908.78 samples/s lr: 9.28e-04 [09/19 13:54:24] lb.utils.events INFO: eta: 1 day, 22:22:30 iteration: 65499/375342 consumed_samples: 67072000 total_loss: 0.4494 time: 0.5365 s/iter data_time: 0.0445 s/iter total_throughput: 1908.76 samples/s lr: 9.27e-04 [09/19 13:55:19] lb.utils.events INFO: eta: 1 day, 22:22:31 iteration: 65599/375342 consumed_samples: 67174400 total_loss: 0.4471 time: 0.5365 s/iter data_time: 0.0477 s/iter total_throughput: 1908.71 samples/s lr: 9.27e-04 [09/19 13:56:13] lb.utils.events INFO: eta: 1 day, 22:24:31 iteration: 65699/375342 consumed_samples: 67276800 total_loss: 0.4493 time: 0.5365 s/iter data_time: 0.0492 s/iter total_throughput: 1908.68 samples/s lr: 9.27e-04 [09/19 13:57:08] lb.utils.events INFO: eta: 1 day, 22:26:55 iteration: 65799/375342 consumed_samples: 67379200 total_loss: 0.4509 time: 0.5365 s/iter data_time: 0.0510 s/iter total_throughput: 1908.64 samples/s lr: 9.27e-04 [09/19 13:58:02] lb.utils.events INFO: eta: 1 day, 22:29:46 iteration: 65899/375342 consumed_samples: 67481600 total_loss: 0.4506 time: 0.5365 s/iter data_time: 0.0476 s/iter total_throughput: 1908.60 samples/s lr: 9.27e-04 [09/19 13:58:57] lb.utils.events INFO: eta: 1 day, 22:32:15 iteration: 65999/375342 consumed_samples: 67584000 total_loss: 0.4511 time: 0.5365 s/iter data_time: 0.0496 s/iter total_throughput: 1908.55 samples/s lr: 9.26e-04 [09/19 13:59:51] lb.utils.events INFO: eta: 1 day, 22:35:49 iteration: 66099/375342 consumed_samples: 67686400 total_loss: 0.4507 time: 0.5365 s/iter data_time: 0.0478 s/iter total_throughput: 1908.49 samples/s lr: 9.26e-04 [09/19 14:00:46] lb.utils.events INFO: eta: 1 day, 22:38:46 iteration: 66199/375342 consumed_samples: 67788800 total_loss: 0.4494 time: 0.5366 s/iter data_time: 0.0522 s/iter total_throughput: 1908.44 samples/s lr: 9.26e-04 [09/19 14:01:40] lb.utils.events INFO: eta: 1 day, 22:39:18 iteration: 66299/375342 consumed_samples: 67891200 total_loss: 0.4518 time: 0.5366 s/iter data_time: 0.0526 s/iter total_throughput: 1908.41 samples/s lr: 9.26e-04 [09/19 14:02:34] lb.utils.events INFO: eta: 1 day, 22:39:11 iteration: 66399/375342 consumed_samples: 67993600 total_loss: 0.4517 time: 0.5366 s/iter data_time: 0.0514 s/iter total_throughput: 1908.38 samples/s lr: 9.26e-04 [09/19 14:03:28] lb.utils.events INFO: eta: 1 day, 22:38:22 iteration: 66499/375342 consumed_samples: 68096000 total_loss: 0.4525 time: 0.5366 s/iter data_time: 0.0499 s/iter total_throughput: 1908.37 samples/s lr: 9.25e-04 [09/19 14:04:23] lb.utils.events INFO: eta: 1 day, 22:36:25 iteration: 66599/375342 consumed_samples: 68198400 total_loss: 0.4538 time: 0.5366 s/iter data_time: 0.0499 s/iter total_throughput: 1908.34 samples/s lr: 9.25e-04 [09/19 14:05:17] lb.utils.events INFO: eta: 1 day, 22:35:06 iteration: 66699/375342 consumed_samples: 68300800 total_loss: 0.4595 time: 0.5366 s/iter data_time: 0.0515 s/iter total_throughput: 1908.32 samples/s lr: 9.25e-04 [09/19 14:06:11] lb.utils.events INFO: eta: 1 day, 22:32:55 iteration: 66799/375342 consumed_samples: 68403200 total_loss: 0.4569 time: 0.5366 s/iter data_time: 0.0512 s/iter total_throughput: 1908.29 samples/s lr: 9.25e-04 [09/19 14:07:05] lb.utils.events INFO: eta: 1 day, 22:31:14 iteration: 66899/375342 consumed_samples: 68505600 total_loss: 0.4572 time: 0.5366 s/iter data_time: 0.0476 s/iter total_throughput: 1908.26 samples/s lr: 9.24e-04 [09/19 14:07:59] lb.utils.events INFO: eta: 1 day, 22:28:03 iteration: 66999/375342 consumed_samples: 68608000 total_loss: 0.456 time: 0.5366 s/iter data_time: 0.0480 s/iter total_throughput: 1908.23 samples/s lr: 9.24e-04 [09/19 14:08:54] lb.utils.events INFO: eta: 1 day, 22:25:27 iteration: 67099/375342 consumed_samples: 68710400 total_loss: 0.4469 time: 0.5366 s/iter data_time: 0.0476 s/iter total_throughput: 1908.20 samples/s lr: 9.24e-04 [09/19 14:09:48] lb.utils.events INFO: eta: 1 day, 22:23:10 iteration: 67199/375342 consumed_samples: 68812800 total_loss: 0.442 time: 0.5366 s/iter data_time: 0.0501 s/iter total_throughput: 1908.17 samples/s lr: 9.24e-04 [09/19 14:10:42] lb.utils.events INFO: eta: 1 day, 22:22:15 iteration: 67299/375342 consumed_samples: 68915200 total_loss: 0.4391 time: 0.5367 s/iter data_time: 0.0491 s/iter total_throughput: 1908.13 samples/s lr: 9.24e-04 [09/19 14:11:37] lb.utils.events INFO: eta: 1 day, 22:22:20 iteration: 67399/375342 consumed_samples: 69017600 total_loss: 0.4421 time: 0.5367 s/iter data_time: 0.0502 s/iter total_throughput: 1908.10 samples/s lr: 9.23e-04 [09/19 14:12:31] lb.utils.events INFO: eta: 1 day, 22:22:46 iteration: 67499/375342 consumed_samples: 69120000 total_loss: 0.4428 time: 0.5367 s/iter data_time: 0.0494 s/iter total_throughput: 1908.06 samples/s lr: 9.23e-04 [09/19 14:13:25] lb.utils.events INFO: eta: 1 day, 22:23:22 iteration: 67599/375342 consumed_samples: 69222400 total_loss: 0.452 time: 0.5367 s/iter data_time: 0.0488 s/iter total_throughput: 1908.03 samples/s lr: 9.23e-04 [09/19 14:14:20] lb.utils.events INFO: eta: 1 day, 22:23:06 iteration: 67699/375342 consumed_samples: 69324800 total_loss: 0.4521 time: 0.5367 s/iter data_time: 0.0493 s/iter total_throughput: 1908.00 samples/s lr: 9.23e-04 [09/19 14:15:14] lb.utils.events INFO: eta: 1 day, 22:22:26 iteration: 67799/375342 consumed_samples: 69427200 total_loss: 0.4493 time: 0.5367 s/iter data_time: 0.0495 s/iter total_throughput: 1907.97 samples/s lr: 9.22e-04 [09/19 14:16:08] lb.utils.events INFO: eta: 1 day, 22:20:46 iteration: 67899/375342 consumed_samples: 69529600 total_loss: 0.4495 time: 0.5367 s/iter data_time: 0.0493 s/iter total_throughput: 1907.94 samples/s lr: 9.22e-04 [09/19 14:17:02] lb.utils.events INFO: eta: 1 day, 22:20:38 iteration: 67999/375342 consumed_samples: 69632000 total_loss: 0.4488 time: 0.5367 s/iter data_time: 0.0496 s/iter total_throughput: 1907.91 samples/s lr: 9.22e-04 [09/19 14:17:57] lb.utils.events INFO: eta: 1 day, 22:18:10 iteration: 68099/375342 consumed_samples: 69734400 total_loss: 0.4492 time: 0.5367 s/iter data_time: 0.0502 s/iter total_throughput: 1907.88 samples/s lr: 9.22e-04 [09/19 14:18:51] lb.utils.events INFO: eta: 1 day, 22:16:06 iteration: 68199/375342 consumed_samples: 69836800 total_loss: 0.451 time: 0.5367 s/iter data_time: 0.0487 s/iter total_throughput: 1907.86 samples/s lr: 9.22e-04 [09/19 14:19:45] lb.utils.events INFO: eta: 1 day, 22:14:06 iteration: 68299/375342 consumed_samples: 69939200 total_loss: 0.4497 time: 0.5367 s/iter data_time: 0.0486 s/iter total_throughput: 1907.84 samples/s lr: 9.21e-04 [09/19 14:20:39] lb.utils.events INFO: eta: 1 day, 22:11:11 iteration: 68399/375342 consumed_samples: 70041600 total_loss: 0.448 time: 0.5367 s/iter data_time: 0.0480 s/iter total_throughput: 1907.82 samples/s lr: 9.21e-04 [09/19 14:21:33] lb.utils.events INFO: eta: 1 day, 22:09:09 iteration: 68499/375342 consumed_samples: 70144000 total_loss: 0.4492 time: 0.5367 s/iter data_time: 0.0504 s/iter total_throughput: 1907.80 samples/s lr: 9.21e-04 [09/19 14:22:27] lb.utils.events INFO: eta: 1 day, 22:07:26 iteration: 68599/375342 consumed_samples: 70246400 total_loss: 0.4544 time: 0.5368 s/iter data_time: 0.0483 s/iter total_throughput: 1907.78 samples/s lr: 9.21e-04 [09/19 14:23:21] lb.utils.events INFO: eta: 1 day, 22:05:56 iteration: 68699/375342 consumed_samples: 70348800 total_loss: 0.4512 time: 0.5368 s/iter data_time: 0.0456 s/iter total_throughput: 1907.75 samples/s lr: 9.20e-04 [09/19 14:24:16] lb.utils.events INFO: eta: 1 day, 22:04:18 iteration: 68799/375342 consumed_samples: 70451200 total_loss: 0.439 time: 0.5368 s/iter data_time: 0.0462 s/iter total_throughput: 1907.73 samples/s lr: 9.20e-04 [09/19 14:25:10] lb.utils.events INFO: eta: 1 day, 22:03:05 iteration: 68899/375342 consumed_samples: 70553600 total_loss: 0.442 time: 0.5368 s/iter data_time: 0.0469 s/iter total_throughput: 1907.71 samples/s lr: 9.20e-04 [09/19 14:26:04] lb.utils.events INFO: eta: 1 day, 22:01:17 iteration: 68999/375342 consumed_samples: 70656000 total_loss: 0.4419 time: 0.5368 s/iter data_time: 0.0503 s/iter total_throughput: 1907.66 samples/s lr: 9.20e-04 [09/19 14:26:58] lb.utils.events INFO: eta: 1 day, 22:00:31 iteration: 69099/375342 consumed_samples: 70758400 total_loss: 0.4479 time: 0.5368 s/iter data_time: 0.0480 s/iter total_throughput: 1907.64 samples/s lr: 9.19e-04 [09/19 14:27:53] lb.utils.events INFO: eta: 1 day, 22:01:04 iteration: 69199/375342 consumed_samples: 70860800 total_loss: 0.4541 time: 0.5368 s/iter data_time: 0.0505 s/iter total_throughput: 1907.61 samples/s lr: 9.19e-04 [09/19 14:28:47] lb.utils.events INFO: eta: 1 day, 22:01:19 iteration: 69299/375342 consumed_samples: 70963200 total_loss: 0.4517 time: 0.5368 s/iter data_time: 0.0498 s/iter total_throughput: 1907.58 samples/s lr: 9.19e-04 [09/19 14:29:41] lb.utils.events INFO: eta: 1 day, 22:01:58 iteration: 69399/375342 consumed_samples: 71065600 total_loss: 0.4498 time: 0.5368 s/iter data_time: 0.0509 s/iter total_throughput: 1907.55 samples/s lr: 9.19e-04 [09/19 14:30:35] lb.utils.events INFO: eta: 1 day, 22:01:51 iteration: 69499/375342 consumed_samples: 71168000 total_loss: 0.4488 time: 0.5368 s/iter data_time: 0.0522 s/iter total_throughput: 1907.52 samples/s lr: 9.19e-04 [09/19 14:31:30] lb.utils.events INFO: eta: 1 day, 22:02:34 iteration: 69599/375342 consumed_samples: 71270400 total_loss: 0.449 time: 0.5368 s/iter data_time: 0.0527 s/iter total_throughput: 1907.49 samples/s lr: 9.18e-04 [09/19 14:32:24] lb.utils.events INFO: eta: 1 day, 22:02:10 iteration: 69699/375342 consumed_samples: 71372800 total_loss: 0.4489 time: 0.5368 s/iter data_time: 0.0531 s/iter total_throughput: 1907.45 samples/s lr: 9.18e-04 [09/19 14:33:19] lb.utils.events INFO: eta: 1 day, 22:02:22 iteration: 69799/375342 consumed_samples: 71475200 total_loss: 0.4502 time: 0.5368 s/iter data_time: 0.0501 s/iter total_throughput: 1907.42 samples/s lr: 9.18e-04 [09/19 14:34:13] lb.utils.events INFO: eta: 1 day, 22:02:02 iteration: 69899/375342 consumed_samples: 71577600 total_loss: 0.4546 time: 0.5369 s/iter data_time: 0.0518 s/iter total_throughput: 1907.39 samples/s lr: 9.18e-04 [09/19 14:35:07] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0069999 [09/19 14:35:08] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 14:35:08] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 14:35:12] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0475 s/iter. Inference: 0.2425 s/iter. Eval: 0.0023 s/iter. Total: 0.2923 s/iter. ETA=0:00:10 [09/19 14:35:17] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0578 s/iter. Inference: 0.2686 s/iter. Eval: 0.0024 s/iter. Total: 0.3292 s/iter. ETA=0:00:07 [09/19 14:35:23] lb.evaluation.evaluator INFO: Inference done 41984/50000. Dataloading: 0.0399 s/iter. Inference: 0.2948 s/iter. Eval: 0.0026 s/iter. Total: 0.3376 s/iter. ETA=0:00:02 [09/19 14:35:25] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 14:35:25] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.423184 (0.000288 s / iter per device, on 8 devices) [09/19 14:35:25] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:12 (0.000257 s / iter per device, on 8 devices) [09/19 14:35:25] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 14:35:25] lb.evaluation.utils INFO: copypaste: Acc@1=69.854 [09/19 14:35:25] lb.evaluation.utils INFO: copypaste: Acc@5=89.706 [09/19 14:35:25] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 69.85400, better than last best score 69.15200 @ iteration 64999. [09/19 14:35:25] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 14:35:26] lb.utils.events INFO: eta: 1 day, 22:01:38 iteration: 69999/375342 consumed_samples: 71680000 total_loss: 0.4513 time: 0.5369 s/iter data_time: 0.0505 s/iter total_throughput: 1907.36 samples/s lr: 9.17e-04 [09/19 14:36:20] lb.utils.events INFO: eta: 1 day, 22:00:23 iteration: 70099/375342 consumed_samples: 71782400 total_loss: 0.4459 time: 0.5369 s/iter data_time: 0.0523 s/iter total_throughput: 1907.34 samples/s lr: 9.17e-04 [09/19 14:37:14] lb.utils.events INFO: eta: 1 day, 21:59:34 iteration: 70199/375342 consumed_samples: 71884800 total_loss: 0.4486 time: 0.5369 s/iter data_time: 0.0502 s/iter total_throughput: 1907.31 samples/s lr: 9.17e-04 [09/19 14:38:08] lb.utils.events INFO: eta: 1 day, 21:58:31 iteration: 70299/375342 consumed_samples: 71987200 total_loss: 0.4481 time: 0.5369 s/iter data_time: 0.0483 s/iter total_throughput: 1907.29 samples/s lr: 9.17e-04 [09/19 14:39:03] lb.utils.events INFO: eta: 1 day, 21:57:38 iteration: 70399/375342 consumed_samples: 72089600 total_loss: 0.4486 time: 0.5369 s/iter data_time: 0.0482 s/iter total_throughput: 1907.26 samples/s lr: 9.17e-04 [09/19 14:39:57] lb.utils.events INFO: eta: 1 day, 21:56:43 iteration: 70499/375342 consumed_samples: 72192000 total_loss: 0.455 time: 0.5369 s/iter data_time: 0.0479 s/iter total_throughput: 1907.23 samples/s lr: 9.16e-04 [09/19 14:40:51] lb.utils.events INFO: eta: 1 day, 21:55:35 iteration: 70599/375342 consumed_samples: 72294400 total_loss: 0.457 time: 0.5369 s/iter data_time: 0.0477 s/iter total_throughput: 1907.20 samples/s lr: 9.16e-04 [09/19 14:41:45] lb.utils.events INFO: eta: 1 day, 21:54:49 iteration: 70699/375342 consumed_samples: 72396800 total_loss: 0.4505 time: 0.5369 s/iter data_time: 0.0489 s/iter total_throughput: 1907.17 samples/s lr: 9.16e-04 [09/19 14:42:40] lb.utils.events INFO: eta: 1 day, 21:53:26 iteration: 70799/375342 consumed_samples: 72499200 total_loss: 0.4512 time: 0.5369 s/iter data_time: 0.0485 s/iter total_throughput: 1907.15 samples/s lr: 9.16e-04 [09/19 14:43:34] lb.utils.events INFO: eta: 1 day, 21:51:39 iteration: 70899/375342 consumed_samples: 72601600 total_loss: 0.4456 time: 0.5369 s/iter data_time: 0.0475 s/iter total_throughput: 1907.12 samples/s lr: 9.15e-04 [09/19 14:44:28] lb.utils.events INFO: eta: 1 day, 21:50:17 iteration: 70999/375342 consumed_samples: 72704000 total_loss: 0.4436 time: 0.5369 s/iter data_time: 0.0481 s/iter total_throughput: 1907.10 samples/s lr: 9.15e-04 [09/19 14:45:22] lb.utils.events INFO: eta: 1 day, 21:49:41 iteration: 71099/375342 consumed_samples: 72806400 total_loss: 0.4517 time: 0.5369 s/iter data_time: 0.0481 s/iter total_throughput: 1907.07 samples/s lr: 9.15e-04 [09/19 14:46:17] lb.utils.events INFO: eta: 1 day, 21:48:56 iteration: 71199/375342 consumed_samples: 72908800 total_loss: 0.4515 time: 0.5370 s/iter data_time: 0.0499 s/iter total_throughput: 1907.04 samples/s lr: 9.15e-04 [09/19 14:47:11] lb.utils.events INFO: eta: 1 day, 21:47:42 iteration: 71299/375342 consumed_samples: 73011200 total_loss: 0.439 time: 0.5370 s/iter data_time: 0.0482 s/iter total_throughput: 1907.02 samples/s lr: 9.14e-04 [09/19 14:48:05] lb.utils.events INFO: eta: 1 day, 21:46:09 iteration: 71399/375342 consumed_samples: 73113600 total_loss: 0.4436 time: 0.5370 s/iter data_time: 0.0502 s/iter total_throughput: 1907.00 samples/s lr: 9.14e-04 [09/19 14:48:59] lb.utils.events INFO: eta: 1 day, 21:45:05 iteration: 71499/375342 consumed_samples: 73216000 total_loss: 0.4511 time: 0.5370 s/iter data_time: 0.0489 s/iter total_throughput: 1906.97 samples/s lr: 9.14e-04 [09/19 14:49:53] lb.utils.events INFO: eta: 1 day, 21:43:32 iteration: 71599/375342 consumed_samples: 73318400 total_loss: 0.4515 time: 0.5370 s/iter data_time: 0.0490 s/iter total_throughput: 1906.95 samples/s lr: 9.14e-04 [09/19 14:50:47] lb.utils.events INFO: eta: 1 day, 21:41:00 iteration: 71699/375342 consumed_samples: 73420800 total_loss: 0.4443 time: 0.5370 s/iter data_time: 0.0492 s/iter total_throughput: 1906.93 samples/s lr: 9.14e-04 [09/19 14:51:42] lb.utils.events INFO: eta: 1 day, 21:40:06 iteration: 71799/375342 consumed_samples: 73523200 total_loss: 0.4438 time: 0.5370 s/iter data_time: 0.0494 s/iter total_throughput: 1906.91 samples/s lr: 9.13e-04 [09/19 14:52:36] lb.utils.events INFO: eta: 1 day, 21:39:07 iteration: 71899/375342 consumed_samples: 73625600 total_loss: 0.4442 time: 0.5370 s/iter data_time: 0.0504 s/iter total_throughput: 1906.89 samples/s lr: 9.13e-04 [09/19 14:53:30] lb.utils.events INFO: eta: 1 day, 21:37:28 iteration: 71999/375342 consumed_samples: 73728000 total_loss: 0.447 time: 0.5370 s/iter data_time: 0.0492 s/iter total_throughput: 1906.87 samples/s lr: 9.13e-04 [09/19 14:54:24] lb.utils.events INFO: eta: 1 day, 21:36:29 iteration: 72099/375342 consumed_samples: 73830400 total_loss: 0.4528 time: 0.5370 s/iter data_time: 0.0472 s/iter total_throughput: 1906.85 samples/s lr: 9.13e-04 [09/19 14:55:18] lb.utils.events INFO: eta: 1 day, 21:35:09 iteration: 72199/375342 consumed_samples: 73932800 total_loss: 0.4487 time: 0.5370 s/iter data_time: 0.0471 s/iter total_throughput: 1906.83 samples/s lr: 9.12e-04 [09/19 14:56:12] lb.utils.events INFO: eta: 1 day, 21:34:03 iteration: 72299/375342 consumed_samples: 74035200 total_loss: 0.4461 time: 0.5370 s/iter data_time: 0.0478 s/iter total_throughput: 1906.81 samples/s lr: 9.12e-04 [09/19 14:57:06] lb.utils.events INFO: eta: 1 day, 21:32:47 iteration: 72399/375342 consumed_samples: 74137600 total_loss: 0.4443 time: 0.5370 s/iter data_time: 0.0482 s/iter total_throughput: 1906.79 samples/s lr: 9.12e-04 [09/19 14:58:01] lb.utils.events INFO: eta: 1 day, 21:31:44 iteration: 72499/375342 consumed_samples: 74240000 total_loss: 0.4411 time: 0.5370 s/iter data_time: 0.0508 s/iter total_throughput: 1906.74 samples/s lr: 9.12e-04 [09/19 14:58:55] lb.utils.events INFO: eta: 1 day, 21:31:07 iteration: 72599/375342 consumed_samples: 74342400 total_loss: 0.446 time: 0.5370 s/iter data_time: 0.0539 s/iter total_throughput: 1906.72 samples/s lr: 9.11e-04 [09/19 14:59:49] lb.utils.events INFO: eta: 1 day, 21:31:07 iteration: 72699/375342 consumed_samples: 74444800 total_loss: 0.4479 time: 0.5371 s/iter data_time: 0.0494 s/iter total_throughput: 1906.69 samples/s lr: 9.11e-04 [09/19 15:00:44] lb.utils.events INFO: eta: 1 day, 21:30:23 iteration: 72799/375342 consumed_samples: 74547200 total_loss: 0.4434 time: 0.5371 s/iter data_time: 0.0529 s/iter total_throughput: 1906.67 samples/s lr: 9.11e-04 [09/19 15:01:38] lb.utils.events INFO: eta: 1 day, 21:30:33 iteration: 72899/375342 consumed_samples: 74649600 total_loss: 0.4428 time: 0.5371 s/iter data_time: 0.0517 s/iter total_throughput: 1906.64 samples/s lr: 9.11e-04 [09/19 15:02:32] lb.utils.events INFO: eta: 1 day, 21:31:04 iteration: 72999/375342 consumed_samples: 74752000 total_loss: 0.4481 time: 0.5371 s/iter data_time: 0.0532 s/iter total_throughput: 1906.61 samples/s lr: 9.10e-04 [09/19 15:03:27] lb.utils.events INFO: eta: 1 day, 21:31:34 iteration: 73099/375342 consumed_samples: 74854400 total_loss: 0.4498 time: 0.5371 s/iter data_time: 0.0537 s/iter total_throughput: 1906.58 samples/s lr: 9.10e-04 [09/19 15:04:21] lb.utils.events INFO: eta: 1 day, 21:31:41 iteration: 73199/375342 consumed_samples: 74956800 total_loss: 0.4467 time: 0.5371 s/iter data_time: 0.0523 s/iter total_throughput: 1906.55 samples/s lr: 9.10e-04 [09/19 15:05:15] lb.utils.events INFO: eta: 1 day, 21:32:19 iteration: 73299/375342 consumed_samples: 75059200 total_loss: 0.4395 time: 0.5371 s/iter data_time: 0.0549 s/iter total_throughput: 1906.52 samples/s lr: 9.10e-04 [09/19 15:06:10] lb.utils.events INFO: eta: 1 day, 21:32:07 iteration: 73399/375342 consumed_samples: 75161600 total_loss: 0.4427 time: 0.5371 s/iter data_time: 0.0522 s/iter total_throughput: 1906.49 samples/s lr: 9.09e-04 [09/19 15:07:04] lb.utils.events INFO: eta: 1 day, 21:31:59 iteration: 73499/375342 consumed_samples: 75264000 total_loss: 0.4448 time: 0.5371 s/iter data_time: 0.0504 s/iter total_throughput: 1906.46 samples/s lr: 9.09e-04 [09/19 15:07:58] lb.utils.events INFO: eta: 1 day, 21:31:52 iteration: 73599/375342 consumed_samples: 75366400 total_loss: 0.4509 time: 0.5371 s/iter data_time: 0.0522 s/iter total_throughput: 1906.43 samples/s lr: 9.09e-04 [09/19 15:08:53] lb.utils.events INFO: eta: 1 day, 21:30:58 iteration: 73699/375342 consumed_samples: 75468800 total_loss: 0.4442 time: 0.5371 s/iter data_time: 0.0506 s/iter total_throughput: 1906.41 samples/s lr: 9.09e-04 [09/19 15:09:47] lb.utils.events INFO: eta: 1 day, 21:30:43 iteration: 73799/375342 consumed_samples: 75571200 total_loss: 0.4431 time: 0.5371 s/iter data_time: 0.0485 s/iter total_throughput: 1906.38 samples/s lr: 9.09e-04 [09/19 15:10:41] lb.utils.events INFO: eta: 1 day, 21:30:06 iteration: 73899/375342 consumed_samples: 75673600 total_loss: 0.4451 time: 0.5372 s/iter data_time: 0.0479 s/iter total_throughput: 1906.35 samples/s lr: 9.08e-04 [09/19 15:11:36] lb.utils.events INFO: eta: 1 day, 21:28:53 iteration: 73999/375342 consumed_samples: 75776000 total_loss: 0.4468 time: 0.5372 s/iter data_time: 0.0480 s/iter total_throughput: 1906.32 samples/s lr: 9.08e-04 [09/19 15:12:30] lb.utils.events INFO: eta: 1 day, 21:27:54 iteration: 74099/375342 consumed_samples: 75878400 total_loss: 0.4514 time: 0.5372 s/iter data_time: 0.0486 s/iter total_throughput: 1906.29 samples/s lr: 9.08e-04 [09/19 15:13:24] lb.utils.events INFO: eta: 1 day, 21:26:26 iteration: 74199/375342 consumed_samples: 75980800 total_loss: 0.451 time: 0.5372 s/iter data_time: 0.0480 s/iter total_throughput: 1906.27 samples/s lr: 9.08e-04 [09/19 15:14:19] lb.utils.events INFO: eta: 1 day, 21:24:50 iteration: 74299/375342 consumed_samples: 76083200 total_loss: 0.4445 time: 0.5372 s/iter data_time: 0.0502 s/iter total_throughput: 1906.24 samples/s lr: 9.07e-04 [09/19 15:15:13] lb.utils.events INFO: eta: 1 day, 21:22:35 iteration: 74399/375342 consumed_samples: 76185600 total_loss: 0.4453 time: 0.5372 s/iter data_time: 0.0477 s/iter total_throughput: 1906.22 samples/s lr: 9.07e-04 [09/19 15:16:07] lb.utils.events INFO: eta: 1 day, 21:21:12 iteration: 74499/375342 consumed_samples: 76288000 total_loss: 0.4355 time: 0.5372 s/iter data_time: 0.0499 s/iter total_throughput: 1906.19 samples/s lr: 9.07e-04 [09/19 15:17:01] lb.utils.events INFO: eta: 1 day, 21:19:30 iteration: 74599/375342 consumed_samples: 76390400 total_loss: 0.4458 time: 0.5372 s/iter data_time: 0.0489 s/iter total_throughput: 1906.17 samples/s lr: 9.07e-04 [09/19 15:17:56] lb.utils.events INFO: eta: 1 day, 21:18:27 iteration: 74699/375342 consumed_samples: 76492800 total_loss: 0.4553 time: 0.5372 s/iter data_time: 0.0483 s/iter total_throughput: 1906.14 samples/s lr: 9.06e-04 [09/19 15:18:50] lb.utils.events INFO: eta: 1 day, 21:17:00 iteration: 74799/375342 consumed_samples: 76595200 total_loss: 0.4554 time: 0.5372 s/iter data_time: 0.0501 s/iter total_throughput: 1906.12 samples/s lr: 9.06e-04 [09/19 15:19:44] lb.utils.events INFO: eta: 1 day, 21:15:44 iteration: 74899/375342 consumed_samples: 76697600 total_loss: 0.4513 time: 0.5372 s/iter data_time: 0.0503 s/iter total_throughput: 1906.09 samples/s lr: 9.06e-04 [09/19 15:20:38] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0074999 [09/19 15:20:39] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 15:20:39] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 15:20:44] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0606 s/iter. Inference: 0.2461 s/iter. Eval: 0.0024 s/iter. Total: 0.3092 s/iter. ETA=0:00:11 [09/19 15:20:49] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0746 s/iter. Inference: 0.2572 s/iter. Eval: 0.0024 s/iter. Total: 0.3343 s/iter. ETA=0:00:07 [09/19 15:20:54] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0720 s/iter. Inference: 0.2546 s/iter. Eval: 0.0026 s/iter. Total: 0.3295 s/iter. ETA=0:00:01 [09/19 15:20:56] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 15:20:56] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.151809 (0.000283 s / iter per device, on 8 devices) [09/19 15:20:56] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000223 s / iter per device, on 8 devices) [09/19 15:20:56] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 15:20:56] lb.evaluation.utils INFO: copypaste: Acc@1=70.05 [09/19 15:20:56] lb.evaluation.utils INFO: copypaste: Acc@5=90.008 [09/19 15:20:56] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 70.05000, better than last best score 69.85400 @ iteration 69999. [09/19 15:20:56] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 15:20:57] lb.utils.events INFO: eta: 1 day, 21:14:16 iteration: 74999/375342 consumed_samples: 76800000 total_loss: 0.443 time: 0.5372 s/iter data_time: 0.0490 s/iter total_throughput: 1906.07 samples/s lr: 9.06e-04 [09/19 15:21:51] lb.utils.events INFO: eta: 1 day, 21:12:14 iteration: 75099/375342 consumed_samples: 76902400 total_loss: 0.4393 time: 0.5372 s/iter data_time: 0.0502 s/iter total_throughput: 1906.05 samples/s lr: 9.05e-04 [09/19 15:22:45] lb.utils.events INFO: eta: 1 day, 21:10:40 iteration: 75199/375342 consumed_samples: 77004800 total_loss: 0.4445 time: 0.5372 s/iter data_time: 0.0506 s/iter total_throughput: 1906.03 samples/s lr: 9.05e-04 [09/19 15:23:39] lb.utils.events INFO: eta: 1 day, 21:10:05 iteration: 75299/375342 consumed_samples: 77107200 total_loss: 0.4404 time: 0.5373 s/iter data_time: 0.0481 s/iter total_throughput: 1905.99 samples/s lr: 9.05e-04 [09/19 15:24:34] lb.utils.events INFO: eta: 1 day, 21:10:16 iteration: 75399/375342 consumed_samples: 77209600 total_loss: 0.4354 time: 0.5373 s/iter data_time: 0.0485 s/iter total_throughput: 1905.96 samples/s lr: 9.05e-04 [09/19 15:25:28] lb.utils.events INFO: eta: 1 day, 21:09:38 iteration: 75499/375342 consumed_samples: 77312000 total_loss: 0.4399 time: 0.5373 s/iter data_time: 0.0498 s/iter total_throughput: 1905.93 samples/s lr: 9.04e-04 [09/19 15:26:23] lb.utils.events INFO: eta: 1 day, 21:08:54 iteration: 75599/375342 consumed_samples: 77414400 total_loss: 0.4428 time: 0.5373 s/iter data_time: 0.0506 s/iter total_throughput: 1905.90 samples/s lr: 9.04e-04 [09/19 15:27:17] lb.utils.events INFO: eta: 1 day, 21:08:56 iteration: 75699/375342 consumed_samples: 77516800 total_loss: 0.4469 time: 0.5373 s/iter data_time: 0.0521 s/iter total_throughput: 1905.88 samples/s lr: 9.04e-04 [09/19 15:28:11] lb.utils.events INFO: eta: 1 day, 21:09:41 iteration: 75799/375342 consumed_samples: 77619200 total_loss: 0.4504 time: 0.5373 s/iter data_time: 0.0511 s/iter total_throughput: 1905.84 samples/s lr: 9.04e-04 [09/19 15:29:06] lb.utils.events INFO: eta: 1 day, 21:09:34 iteration: 75899/375342 consumed_samples: 77721600 total_loss: 0.45 time: 0.5373 s/iter data_time: 0.0540 s/iter total_throughput: 1905.80 samples/s lr: 9.03e-04 [09/19 15:30:00] lb.utils.events INFO: eta: 1 day, 21:09:29 iteration: 75999/375342 consumed_samples: 77824000 total_loss: 0.44 time: 0.5373 s/iter data_time: 0.0524 s/iter total_throughput: 1905.77 samples/s lr: 9.03e-04 [09/19 15:30:55] lb.utils.events INFO: eta: 1 day, 21:09:01 iteration: 76099/375342 consumed_samples: 77926400 total_loss: 0.4417 time: 0.5373 s/iter data_time: 0.0522 s/iter total_throughput: 1905.74 samples/s lr: 9.03e-04 [09/19 15:31:49] lb.utils.events INFO: eta: 1 day, 21:08:24 iteration: 76199/375342 consumed_samples: 78028800 total_loss: 0.4484 time: 0.5373 s/iter data_time: 0.0512 s/iter total_throughput: 1905.72 samples/s lr: 9.03e-04 [09/19 15:32:43] lb.utils.events INFO: eta: 1 day, 21:07:00 iteration: 76299/375342 consumed_samples: 78131200 total_loss: 0.4347 time: 0.5373 s/iter data_time: 0.0510 s/iter total_throughput: 1905.70 samples/s lr: 9.02e-04 [09/19 15:33:37] lb.utils.events INFO: eta: 1 day, 21:05:25 iteration: 76399/375342 consumed_samples: 78233600 total_loss: 0.4268 time: 0.5373 s/iter data_time: 0.0499 s/iter total_throughput: 1905.68 samples/s lr: 9.02e-04 [09/19 15:34:32] lb.utils.events INFO: eta: 1 day, 21:04:25 iteration: 76499/375342 consumed_samples: 78336000 total_loss: 0.4387 time: 0.5373 s/iter data_time: 0.0516 s/iter total_throughput: 1905.66 samples/s lr: 9.02e-04 [09/19 15:35:26] lb.utils.events INFO: eta: 1 day, 21:03:18 iteration: 76599/375342 consumed_samples: 78438400 total_loss: 0.4447 time: 0.5374 s/iter data_time: 0.0519 s/iter total_throughput: 1905.63 samples/s lr: 9.02e-04 [09/19 15:36:20] lb.utils.events INFO: eta: 1 day, 21:02:24 iteration: 76699/375342 consumed_samples: 78540800 total_loss: 0.4409 time: 0.5374 s/iter data_time: 0.0521 s/iter total_throughput: 1905.61 samples/s lr: 9.01e-04 [09/19 15:37:15] lb.utils.events INFO: eta: 1 day, 21:00:46 iteration: 76799/375342 consumed_samples: 78643200 total_loss: 0.4444 time: 0.5374 s/iter data_time: 0.0515 s/iter total_throughput: 1905.58 samples/s lr: 9.01e-04 [09/19 15:38:09] lb.utils.events INFO: eta: 1 day, 20:58:54 iteration: 76899/375342 consumed_samples: 78745600 total_loss: 0.4455 time: 0.5374 s/iter data_time: 0.0531 s/iter total_throughput: 1905.56 samples/s lr: 9.01e-04 [09/19 15:39:03] lb.utils.events INFO: eta: 1 day, 20:57:48 iteration: 76999/375342 consumed_samples: 78848000 total_loss: 0.4444 time: 0.5374 s/iter data_time: 0.0514 s/iter total_throughput: 1905.53 samples/s lr: 9.01e-04 [09/19 15:39:57] lb.utils.events INFO: eta: 1 day, 20:56:04 iteration: 77099/375342 consumed_samples: 78950400 total_loss: 0.4446 time: 0.5374 s/iter data_time: 0.0504 s/iter total_throughput: 1905.51 samples/s lr: 9.00e-04 [09/19 15:40:52] lb.utils.events INFO: eta: 1 day, 20:55:21 iteration: 77199/375342 consumed_samples: 79052800 total_loss: 0.4439 time: 0.5374 s/iter data_time: 0.0504 s/iter total_throughput: 1905.49 samples/s lr: 9.00e-04 [09/19 15:41:46] lb.utils.events INFO: eta: 1 day, 20:54:36 iteration: 77299/375342 consumed_samples: 79155200 total_loss: 0.4436 time: 0.5374 s/iter data_time: 0.0527 s/iter total_throughput: 1905.47 samples/s lr: 9.00e-04 [09/19 15:42:40] lb.utils.events INFO: eta: 1 day, 20:54:12 iteration: 77399/375342 consumed_samples: 79257600 total_loss: 0.4464 time: 0.5374 s/iter data_time: 0.0509 s/iter total_throughput: 1905.44 samples/s lr: 9.00e-04 [09/19 15:43:35] lb.utils.events INFO: eta: 1 day, 20:53:54 iteration: 77499/375342 consumed_samples: 79360000 total_loss: 0.4539 time: 0.5374 s/iter data_time: 0.0530 s/iter total_throughput: 1905.42 samples/s lr: 8.99e-04 [09/19 15:44:29] lb.utils.events INFO: eta: 1 day, 20:52:30 iteration: 77599/375342 consumed_samples: 79462400 total_loss: 0.4412 time: 0.5374 s/iter data_time: 0.0512 s/iter total_throughput: 1905.39 samples/s lr: 8.99e-04 [09/19 15:45:23] lb.utils.events INFO: eta: 1 day, 20:50:55 iteration: 77699/375342 consumed_samples: 79564800 total_loss: 0.4405 time: 0.5374 s/iter data_time: 0.0522 s/iter total_throughput: 1905.37 samples/s lr: 8.99e-04 [09/19 15:46:17] lb.utils.events INFO: eta: 1 day, 20:49:42 iteration: 77799/375342 consumed_samples: 79667200 total_loss: 0.4441 time: 0.5374 s/iter data_time: 0.0519 s/iter total_throughput: 1905.35 samples/s lr: 8.99e-04 [09/19 15:47:12] lb.utils.events INFO: eta: 1 day, 20:48:48 iteration: 77899/375342 consumed_samples: 79769600 total_loss: 0.4437 time: 0.5374 s/iter data_time: 0.0486 s/iter total_throughput: 1905.33 samples/s lr: 8.98e-04 [09/19 15:48:06] lb.utils.events INFO: eta: 1 day, 20:47:50 iteration: 77999/375342 consumed_samples: 79872000 total_loss: 0.4484 time: 0.5374 s/iter data_time: 0.0508 s/iter total_throughput: 1905.30 samples/s lr: 8.98e-04 [09/19 15:49:00] lb.utils.events INFO: eta: 1 day, 20:48:01 iteration: 78099/375342 consumed_samples: 79974400 total_loss: 0.4488 time: 0.5375 s/iter data_time: 0.0497 s/iter total_throughput: 1905.27 samples/s lr: 8.98e-04 [09/19 15:49:55] lb.utils.events INFO: eta: 1 day, 20:47:49 iteration: 78199/375342 consumed_samples: 80076800 total_loss: 0.4488 time: 0.5375 s/iter data_time: 0.0503 s/iter total_throughput: 1905.25 samples/s lr: 8.98e-04 [09/19 15:50:49] lb.utils.events INFO: eta: 1 day, 20:47:44 iteration: 78299/375342 consumed_samples: 80179200 total_loss: 0.4396 time: 0.5375 s/iter data_time: 0.0493 s/iter total_throughput: 1905.22 samples/s lr: 8.97e-04 [09/19 15:51:44] lb.utils.events INFO: eta: 1 day, 20:47:51 iteration: 78399/375342 consumed_samples: 80281600 total_loss: 0.4407 time: 0.5375 s/iter data_time: 0.0472 s/iter total_throughput: 1905.19 samples/s lr: 8.97e-04 [09/19 15:52:38] lb.utils.events INFO: eta: 1 day, 20:47:12 iteration: 78499/375342 consumed_samples: 80384000 total_loss: 0.4526 time: 0.5375 s/iter data_time: 0.0490 s/iter total_throughput: 1905.16 samples/s lr: 8.97e-04 [09/19 15:53:32] lb.utils.events INFO: eta: 1 day, 20:46:45 iteration: 78599/375342 consumed_samples: 80486400 total_loss: 0.4472 time: 0.5375 s/iter data_time: 0.0492 s/iter total_throughput: 1905.13 samples/s lr: 8.97e-04 [09/19 15:54:27] lb.utils.events INFO: eta: 1 day, 20:46:45 iteration: 78699/375342 consumed_samples: 80588800 total_loss: 0.4465 time: 0.5375 s/iter data_time: 0.0497 s/iter total_throughput: 1905.10 samples/s lr: 8.96e-04 [09/19 15:55:21] lb.utils.events INFO: eta: 1 day, 20:46:19 iteration: 78799/375342 consumed_samples: 80691200 total_loss: 0.4515 time: 0.5375 s/iter data_time: 0.0479 s/iter total_throughput: 1905.07 samples/s lr: 8.96e-04 [09/19 15:56:16] lb.utils.events INFO: eta: 1 day, 20:46:20 iteration: 78899/375342 consumed_samples: 80793600 total_loss: 0.4481 time: 0.5375 s/iter data_time: 0.0500 s/iter total_throughput: 1905.05 samples/s lr: 8.96e-04 [09/19 15:57:10] lb.utils.events INFO: eta: 1 day, 20:46:19 iteration: 78999/375342 consumed_samples: 80896000 total_loss: 0.4436 time: 0.5375 s/iter data_time: 0.0491 s/iter total_throughput: 1905.02 samples/s lr: 8.96e-04 [09/19 15:58:04] lb.utils.events INFO: eta: 1 day, 20:45:23 iteration: 79099/375342 consumed_samples: 80998400 total_loss: 0.4421 time: 0.5375 s/iter data_time: 0.0479 s/iter total_throughput: 1904.99 samples/s lr: 8.95e-04 [09/19 15:58:59] lb.utils.events INFO: eta: 1 day, 20:44:21 iteration: 79199/375342 consumed_samples: 81100800 total_loss: 0.4422 time: 0.5375 s/iter data_time: 0.0492 s/iter total_throughput: 1904.97 samples/s lr: 8.95e-04 [09/19 15:59:53] lb.utils.events INFO: eta: 1 day, 20:43:11 iteration: 79299/375342 consumed_samples: 81203200 total_loss: 0.4379 time: 0.5375 s/iter data_time: 0.0486 s/iter total_throughput: 1904.94 samples/s lr: 8.95e-04 [09/19 16:00:48] lb.utils.events INFO: eta: 1 day, 20:41:06 iteration: 79399/375342 consumed_samples: 81305600 total_loss: 0.4375 time: 0.5376 s/iter data_time: 0.0518 s/iter total_throughput: 1904.90 samples/s lr: 8.95e-04 [09/19 16:01:42] lb.utils.events INFO: eta: 1 day, 20:39:34 iteration: 79499/375342 consumed_samples: 81408000 total_loss: 0.4428 time: 0.5376 s/iter data_time: 0.0515 s/iter total_throughput: 1904.88 samples/s lr: 8.94e-04 [09/19 16:02:36] lb.utils.events INFO: eta: 1 day, 20:37:37 iteration: 79599/375342 consumed_samples: 81510400 total_loss: 0.4374 time: 0.5376 s/iter data_time: 0.0526 s/iter total_throughput: 1904.86 samples/s lr: 8.94e-04 [09/19 16:03:30] lb.utils.events INFO: eta: 1 day, 20:35:43 iteration: 79699/375342 consumed_samples: 81612800 total_loss: 0.4383 time: 0.5376 s/iter data_time: 0.0531 s/iter total_throughput: 1904.84 samples/s lr: 8.94e-04 [09/19 16:04:25] lb.utils.events INFO: eta: 1 day, 20:33:58 iteration: 79799/375342 consumed_samples: 81715200 total_loss: 0.4402 time: 0.5376 s/iter data_time: 0.0526 s/iter total_throughput: 1904.83 samples/s lr: 8.94e-04 [09/19 16:05:19] lb.utils.events INFO: eta: 1 day, 20:32:45 iteration: 79899/375342 consumed_samples: 81817600 total_loss: 0.4395 time: 0.5376 s/iter data_time: 0.0526 s/iter total_throughput: 1904.81 samples/s lr: 8.93e-04 [09/19 16:06:13] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0079999 [09/19 16:06:14] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 16:06:14] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 16:06:18] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0594 s/iter. Inference: 0.2495 s/iter. Eval: 0.0022 s/iter. Total: 0.3111 s/iter. ETA=0:00:11 [09/19 16:06:23] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0715 s/iter. Inference: 0.2596 s/iter. Eval: 0.0026 s/iter. Total: 0.3341 s/iter. ETA=0:00:07 [09/19 16:06:29] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0721 s/iter. Inference: 0.2551 s/iter. Eval: 0.0024 s/iter. Total: 0.3301 s/iter. ETA=0:00:01 [09/19 16:06:31] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 16:06:31] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.192819 (0.000284 s / iter per device, on 8 devices) [09/19 16:06:31] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000223 s / iter per device, on 8 devices) [09/19 16:06:31] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 16:06:31] lb.evaluation.utils INFO: copypaste: Acc@1=70.512 [09/19 16:06:31] lb.evaluation.utils INFO: copypaste: Acc@5=90.16999999999999 [09/19 16:06:31] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 70.51200, better than last best score 70.05000 @ iteration 74999. [09/19 16:06:31] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 16:06:31] lb.utils.events INFO: eta: 1 day, 20:31:08 iteration: 79999/375342 consumed_samples: 81920000 total_loss: 0.4428 time: 0.5376 s/iter data_time: 0.0525 s/iter total_throughput: 1904.78 samples/s lr: 8.93e-04 [09/19 16:07:26] lb.utils.events INFO: eta: 1 day, 20:30:02 iteration: 80099/375342 consumed_samples: 82022400 total_loss: 0.4441 time: 0.5376 s/iter data_time: 0.0519 s/iter total_throughput: 1904.76 samples/s lr: 8.93e-04 [09/19 16:08:20] lb.utils.events INFO: eta: 1 day, 20:30:11 iteration: 80199/375342 consumed_samples: 82124800 total_loss: 0.4436 time: 0.5376 s/iter data_time: 0.0526 s/iter total_throughput: 1904.73 samples/s lr: 8.93e-04 [09/19 16:09:15] lb.utils.events INFO: eta: 1 day, 20:30:01 iteration: 80299/375342 consumed_samples: 82227200 total_loss: 0.4394 time: 0.5376 s/iter data_time: 0.0524 s/iter total_throughput: 1904.70 samples/s lr: 8.92e-04 [09/19 16:10:09] lb.utils.events INFO: eta: 1 day, 20:29:45 iteration: 80399/375342 consumed_samples: 82329600 total_loss: 0.4405 time: 0.5376 s/iter data_time: 0.0535 s/iter total_throughput: 1904.67 samples/s lr: 8.92e-04 [09/19 16:11:03] lb.utils.events INFO: eta: 1 day, 20:29:34 iteration: 80499/375342 consumed_samples: 82432000 total_loss: 0.4341 time: 0.5376 s/iter data_time: 0.0541 s/iter total_throughput: 1904.64 samples/s lr: 8.92e-04 [09/19 16:11:58] lb.utils.events INFO: eta: 1 day, 20:30:00 iteration: 80599/375342 consumed_samples: 82534400 total_loss: 0.4375 time: 0.5376 s/iter data_time: 0.0539 s/iter total_throughput: 1904.61 samples/s lr: 8.92e-04 [09/19 16:12:52] lb.utils.events INFO: eta: 1 day, 20:30:32 iteration: 80699/375342 consumed_samples: 82636800 total_loss: 0.4512 time: 0.5377 s/iter data_time: 0.0550 s/iter total_throughput: 1904.58 samples/s lr: 8.91e-04 [09/19 16:13:47] lb.utils.events INFO: eta: 1 day, 20:31:15 iteration: 80799/375342 consumed_samples: 82739200 total_loss: 0.4477 time: 0.5377 s/iter data_time: 0.0544 s/iter total_throughput: 1904.55 samples/s lr: 8.91e-04 [09/19 16:14:41] lb.utils.events INFO: eta: 1 day, 20:30:36 iteration: 80899/375342 consumed_samples: 82841600 total_loss: 0.4401 time: 0.5377 s/iter data_time: 0.0538 s/iter total_throughput: 1904.53 samples/s lr: 8.91e-04 [09/19 16:15:36] lb.utils.events INFO: eta: 1 day, 20:30:03 iteration: 80999/375342 consumed_samples: 82944000 total_loss: 0.4471 time: 0.5377 s/iter data_time: 0.0530 s/iter total_throughput: 1904.50 samples/s lr: 8.91e-04 [09/19 16:16:30] lb.utils.events INFO: eta: 1 day, 20:29:44 iteration: 81099/375342 consumed_samples: 83046400 total_loss: 0.4422 time: 0.5377 s/iter data_time: 0.0490 s/iter total_throughput: 1904.46 samples/s lr: 8.90e-04 [09/19 16:17:25] lb.utils.events INFO: eta: 1 day, 20:29:28 iteration: 81199/375342 consumed_samples: 83148800 total_loss: 0.4434 time: 0.5377 s/iter data_time: 0.0495 s/iter total_throughput: 1904.43 samples/s lr: 8.90e-04 [09/19 16:18:20] lb.utils.events INFO: eta: 1 day, 20:29:03 iteration: 81299/375342 consumed_samples: 83251200 total_loss: 0.4432 time: 0.5377 s/iter data_time: 0.0511 s/iter total_throughput: 1904.39 samples/s lr: 8.90e-04 [09/19 16:19:14] lb.utils.events INFO: eta: 1 day, 20:29:02 iteration: 81399/375342 consumed_samples: 83353600 total_loss: 0.4386 time: 0.5377 s/iter data_time: 0.0528 s/iter total_throughput: 1904.36 samples/s lr: 8.89e-04 [09/19 16:20:09] lb.utils.events INFO: eta: 1 day, 20:28:46 iteration: 81499/375342 consumed_samples: 83456000 total_loss: 0.4393 time: 0.5377 s/iter data_time: 0.0512 s/iter total_throughput: 1904.32 samples/s lr: 8.89e-04 [09/19 16:21:03] lb.utils.events INFO: eta: 1 day, 20:28:38 iteration: 81599/375342 consumed_samples: 83558400 total_loss: 0.4393 time: 0.5377 s/iter data_time: 0.0488 s/iter total_throughput: 1904.29 samples/s lr: 8.89e-04 [09/19 16:21:58] lb.utils.events INFO: eta: 1 day, 20:28:14 iteration: 81699/375342 consumed_samples: 83660800 total_loss: 0.4401 time: 0.5377 s/iter data_time: 0.0499 s/iter total_throughput: 1904.26 samples/s lr: 8.89e-04 [09/19 16:22:53] lb.utils.events INFO: eta: 1 day, 20:28:42 iteration: 81799/375342 consumed_samples: 83763200 total_loss: 0.4435 time: 0.5378 s/iter data_time: 0.0516 s/iter total_throughput: 1904.22 samples/s lr: 8.88e-04 [09/19 16:23:47] lb.utils.events INFO: eta: 1 day, 20:27:59 iteration: 81899/375342 consumed_samples: 83865600 total_loss: 0.4428 time: 0.5378 s/iter data_time: 0.0512 s/iter total_throughput: 1904.18 samples/s lr: 8.88e-04 [09/19 16:24:42] lb.utils.events INFO: eta: 1 day, 20:27:21 iteration: 81999/375342 consumed_samples: 83968000 total_loss: 0.4339 time: 0.5378 s/iter data_time: 0.0508 s/iter total_throughput: 1904.15 samples/s lr: 8.88e-04 [09/19 16:25:36] lb.utils.events INFO: eta: 1 day, 20:26:17 iteration: 82099/375342 consumed_samples: 84070400 total_loss: 0.4339 time: 0.5378 s/iter data_time: 0.0499 s/iter total_throughput: 1904.12 samples/s lr: 8.88e-04 [09/19 16:26:31] lb.utils.events INFO: eta: 1 day, 20:25:43 iteration: 82199/375342 consumed_samples: 84172800 total_loss: 0.4429 time: 0.5378 s/iter data_time: 0.0517 s/iter total_throughput: 1904.08 samples/s lr: 8.87e-04 [09/19 16:27:25] lb.utils.events INFO: eta: 1 day, 20:24:28 iteration: 82299/375342 consumed_samples: 84275200 total_loss: 0.4486 time: 0.5378 s/iter data_time: 0.0519 s/iter total_throughput: 1904.05 samples/s lr: 8.87e-04 [09/19 16:28:20] lb.utils.events INFO: eta: 1 day, 20:23:32 iteration: 82399/375342 consumed_samples: 84377600 total_loss: 0.4452 time: 0.5378 s/iter data_time: 0.0491 s/iter total_throughput: 1904.02 samples/s lr: 8.87e-04 [09/19 16:29:14] lb.utils.events INFO: eta: 1 day, 20:21:46 iteration: 82499/375342 consumed_samples: 84480000 total_loss: 0.4389 time: 0.5378 s/iter data_time: 0.0489 s/iter total_throughput: 1904.00 samples/s lr: 8.87e-04 [09/19 16:30:09] lb.utils.events INFO: eta: 1 day, 20:20:17 iteration: 82599/375342 consumed_samples: 84582400 total_loss: 0.4419 time: 0.5378 s/iter data_time: 0.0495 s/iter total_throughput: 1903.97 samples/s lr: 8.86e-04 [09/19 16:31:03] lb.utils.events INFO: eta: 1 day, 20:18:28 iteration: 82699/375342 consumed_samples: 84684800 total_loss: 0.4444 time: 0.5378 s/iter data_time: 0.0489 s/iter total_throughput: 1903.94 samples/s lr: 8.86e-04 [09/19 16:31:58] lb.utils.events INFO: eta: 1 day, 20:15:43 iteration: 82799/375342 consumed_samples: 84787200 total_loss: 0.4394 time: 0.5378 s/iter data_time: 0.0530 s/iter total_throughput: 1903.90 samples/s lr: 8.86e-04 [09/19 16:32:52] lb.utils.events INFO: eta: 1 day, 20:14:08 iteration: 82899/375342 consumed_samples: 84889600 total_loss: 0.4352 time: 0.5378 s/iter data_time: 0.0531 s/iter total_throughput: 1903.88 samples/s lr: 8.86e-04 [09/19 16:33:47] lb.utils.events INFO: eta: 1 day, 20:12:05 iteration: 82999/375342 consumed_samples: 84992000 total_loss: 0.4387 time: 0.5379 s/iter data_time: 0.0514 s/iter total_throughput: 1903.85 samples/s lr: 8.85e-04 [09/19 16:34:41] lb.utils.events INFO: eta: 1 day, 20:10:54 iteration: 83099/375342 consumed_samples: 85094400 total_loss: 0.4478 time: 0.5379 s/iter data_time: 0.0529 s/iter total_throughput: 1903.83 samples/s lr: 8.85e-04 [09/19 16:35:35] lb.utils.events INFO: eta: 1 day, 20:08:19 iteration: 83199/375342 consumed_samples: 85196800 total_loss: 0.4444 time: 0.5379 s/iter data_time: 0.0536 s/iter total_throughput: 1903.80 samples/s lr: 8.85e-04 [09/19 16:36:30] lb.utils.events INFO: eta: 1 day, 20:06:51 iteration: 83299/375342 consumed_samples: 85299200 total_loss: 0.4403 time: 0.5379 s/iter data_time: 0.0511 s/iter total_throughput: 1903.78 samples/s lr: 8.84e-04 [09/19 16:37:24] lb.utils.events INFO: eta: 1 day, 20:05:11 iteration: 83399/375342 consumed_samples: 85401600 total_loss: 0.4503 time: 0.5379 s/iter data_time: 0.0526 s/iter total_throughput: 1903.76 samples/s lr: 8.84e-04 [09/19 16:38:19] lb.utils.events INFO: eta: 1 day, 20:03:54 iteration: 83499/375342 consumed_samples: 85504000 total_loss: 0.4413 time: 0.5379 s/iter data_time: 0.0511 s/iter total_throughput: 1903.73 samples/s lr: 8.84e-04 [09/19 16:39:13] lb.utils.events INFO: eta: 1 day, 20:03:44 iteration: 83599/375342 consumed_samples: 85606400 total_loss: 0.4371 time: 0.5379 s/iter data_time: 0.0534 s/iter total_throughput: 1903.70 samples/s lr: 8.84e-04 [09/19 16:40:08] lb.utils.events INFO: eta: 1 day, 20:03:36 iteration: 83699/375342 consumed_samples: 85708800 total_loss: 0.4382 time: 0.5379 s/iter data_time: 0.0514 s/iter total_throughput: 1903.67 samples/s lr: 8.83e-04 [09/19 16:41:02] lb.utils.events INFO: eta: 1 day, 20:03:18 iteration: 83799/375342 consumed_samples: 85811200 total_loss: 0.4412 time: 0.5379 s/iter data_time: 0.0550 s/iter total_throughput: 1903.65 samples/s lr: 8.83e-04 [09/19 16:41:57] lb.utils.events INFO: eta: 1 day, 20:02:25 iteration: 83899/375342 consumed_samples: 85913600 total_loss: 0.4414 time: 0.5379 s/iter data_time: 0.0534 s/iter total_throughput: 1903.62 samples/s lr: 8.83e-04 [09/19 16:42:51] lb.utils.events INFO: eta: 1 day, 20:01:45 iteration: 83999/375342 consumed_samples: 86016000 total_loss: 0.4358 time: 0.5379 s/iter data_time: 0.0552 s/iter total_throughput: 1903.59 samples/s lr: 8.83e-04 [09/19 16:43:46] lb.utils.events INFO: eta: 1 day, 20:01:32 iteration: 84099/375342 consumed_samples: 86118400 total_loss: 0.4423 time: 0.5379 s/iter data_time: 0.0554 s/iter total_throughput: 1903.56 samples/s lr: 8.82e-04 [09/19 16:44:40] lb.utils.events INFO: eta: 1 day, 20:02:01 iteration: 84199/375342 consumed_samples: 86220800 total_loss: 0.4475 time: 0.5379 s/iter data_time: 0.0527 s/iter total_throughput: 1903.53 samples/s lr: 8.82e-04 [09/19 16:45:35] lb.utils.events INFO: eta: 1 day, 20:01:22 iteration: 84299/375342 consumed_samples: 86323200 total_loss: 0.4442 time: 0.5380 s/iter data_time: 0.0532 s/iter total_throughput: 1903.51 samples/s lr: 8.82e-04 [09/19 16:46:29] lb.utils.events INFO: eta: 1 day, 20:01:39 iteration: 84399/375342 consumed_samples: 86425600 total_loss: 0.4396 time: 0.5380 s/iter data_time: 0.0497 s/iter total_throughput: 1903.48 samples/s lr: 8.82e-04 [09/19 16:47:24] lb.utils.events INFO: eta: 1 day, 20:01:06 iteration: 84499/375342 consumed_samples: 86528000 total_loss: 0.4473 time: 0.5380 s/iter data_time: 0.0496 s/iter total_throughput: 1903.45 samples/s lr: 8.81e-04 [09/19 16:48:18] lb.utils.events INFO: eta: 1 day, 20:00:02 iteration: 84599/375342 consumed_samples: 86630400 total_loss: 0.4453 time: 0.5380 s/iter data_time: 0.0489 s/iter total_throughput: 1903.42 samples/s lr: 8.81e-04 [09/19 16:49:12] lb.utils.events INFO: eta: 1 day, 19:58:23 iteration: 84699/375342 consumed_samples: 86732800 total_loss: 0.4431 time: 0.5380 s/iter data_time: 0.0518 s/iter total_throughput: 1903.40 samples/s lr: 8.81e-04 [09/19 16:50:07] lb.utils.events INFO: eta: 1 day, 19:57:25 iteration: 84799/375342 consumed_samples: 86835200 total_loss: 0.4355 time: 0.5380 s/iter data_time: 0.0494 s/iter total_throughput: 1903.37 samples/s lr: 8.80e-04 [09/19 16:51:01] lb.utils.events INFO: eta: 1 day, 19:56:12 iteration: 84899/375342 consumed_samples: 86937600 total_loss: 0.4416 time: 0.5380 s/iter data_time: 0.0484 s/iter total_throughput: 1903.34 samples/s lr: 8.80e-04 [09/19 16:51:56] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0084999 [09/19 16:51:56] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 16:51:56] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 16:52:01] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0572 s/iter. Inference: 0.2471 s/iter. Eval: 0.0022 s/iter. Total: 0.3066 s/iter. ETA=0:00:11 [09/19 16:52:06] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0632 s/iter. Inference: 0.2634 s/iter. Eval: 0.0027 s/iter. Total: 0.3295 s/iter. ETA=0:00:07 [09/19 16:52:11] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0663 s/iter. Inference: 0.2577 s/iter. Eval: 0.0025 s/iter. Total: 0.3268 s/iter. ETA=0:00:01 [09/19 16:52:13] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 16:52:13] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.195177 (0.000284 s / iter per device, on 8 devices) [09/19 16:52:13] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000225 s / iter per device, on 8 devices) [09/19 16:52:13] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 16:52:13] lb.evaluation.utils INFO: copypaste: Acc@1=70.878 [09/19 16:52:13] lb.evaluation.utils INFO: copypaste: Acc@5=90.288 [09/19 16:52:13] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 70.87800, better than last best score 70.51200 @ iteration 79999. [09/19 16:52:13] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 16:52:14] lb.utils.events INFO: eta: 1 day, 19:54:42 iteration: 84999/375342 consumed_samples: 87040000 total_loss: 0.438 time: 0.5380 s/iter data_time: 0.0486 s/iter total_throughput: 1903.32 samples/s lr: 8.80e-04 [09/19 16:53:09] lb.utils.events INFO: eta: 1 day, 19:53:37 iteration: 85099/375342 consumed_samples: 87142400 total_loss: 0.4328 time: 0.5380 s/iter data_time: 0.0478 s/iter total_throughput: 1903.29 samples/s lr: 8.80e-04 [09/19 16:54:03] lb.utils.events INFO: eta: 1 day, 19:51:59 iteration: 85199/375342 consumed_samples: 87244800 total_loss: 0.4327 time: 0.5380 s/iter data_time: 0.0496 s/iter total_throughput: 1903.26 samples/s lr: 8.79e-04 [09/19 16:54:57] lb.utils.events INFO: eta: 1 day, 19:50:53 iteration: 85299/375342 consumed_samples: 87347200 total_loss: 0.4385 time: 0.5380 s/iter data_time: 0.0489 s/iter total_throughput: 1903.24 samples/s lr: 8.79e-04 [09/19 16:55:52] lb.utils.events INFO: eta: 1 day, 19:49:58 iteration: 85399/375342 consumed_samples: 87449600 total_loss: 0.4409 time: 0.5380 s/iter data_time: 0.0505 s/iter total_throughput: 1903.21 samples/s lr: 8.79e-04 [09/19 16:56:46] lb.utils.events INFO: eta: 1 day, 19:49:04 iteration: 85499/375342 consumed_samples: 87552000 total_loss: 0.4372 time: 0.5380 s/iter data_time: 0.0499 s/iter total_throughput: 1903.19 samples/s lr: 8.79e-04 [09/19 16:57:41] lb.utils.events INFO: eta: 1 day, 19:48:29 iteration: 85599/375342 consumed_samples: 87654400 total_loss: 0.4349 time: 0.5381 s/iter data_time: 0.0513 s/iter total_throughput: 1903.16 samples/s lr: 8.78e-04 [09/19 16:58:35] lb.utils.events INFO: eta: 1 day, 19:47:06 iteration: 85699/375342 consumed_samples: 87756800 total_loss: 0.4334 time: 0.5381 s/iter data_time: 0.0500 s/iter total_throughput: 1903.14 samples/s lr: 8.78e-04 [09/19 16:59:29] lb.utils.events INFO: eta: 1 day, 19:45:19 iteration: 85799/375342 consumed_samples: 87859200 total_loss: 0.4373 time: 0.5381 s/iter data_time: 0.0496 s/iter total_throughput: 1903.12 samples/s lr: 8.78e-04 [09/19 17:00:24] lb.utils.events INFO: eta: 1 day, 19:44:11 iteration: 85899/375342 consumed_samples: 87961600 total_loss: 0.4372 time: 0.5381 s/iter data_time: 0.0510 s/iter total_throughput: 1903.10 samples/s lr: 8.77e-04 [09/19 17:01:18] lb.utils.events INFO: eta: 1 day, 19:42:42 iteration: 85999/375342 consumed_samples: 88064000 total_loss: 0.4338 time: 0.5381 s/iter data_time: 0.0505 s/iter total_throughput: 1903.08 samples/s lr: 8.77e-04 [09/19 17:02:12] lb.utils.events INFO: eta: 1 day, 19:40:08 iteration: 86099/375342 consumed_samples: 88166400 total_loss: 0.4334 time: 0.5381 s/iter data_time: 0.0505 s/iter total_throughput: 1903.07 samples/s lr: 8.77e-04 [09/19 17:03:06] lb.utils.events INFO: eta: 1 day, 19:37:32 iteration: 86199/375342 consumed_samples: 88268800 total_loss: 0.4367 time: 0.5381 s/iter data_time: 0.0487 s/iter total_throughput: 1903.06 samples/s lr: 8.77e-04 [09/19 17:04:01] lb.utils.events INFO: eta: 1 day, 19:36:15 iteration: 86299/375342 consumed_samples: 88371200 total_loss: 0.4358 time: 0.5381 s/iter data_time: 0.0526 s/iter total_throughput: 1903.02 samples/s lr: 8.76e-04 [09/19 17:04:55] lb.utils.events INFO: eta: 1 day, 19:34:46 iteration: 86399/375342 consumed_samples: 88473600 total_loss: 0.4296 time: 0.5381 s/iter data_time: 0.0528 s/iter total_throughput: 1902.99 samples/s lr: 8.76e-04 [09/19 17:05:50] lb.utils.events INFO: eta: 1 day, 19:33:11 iteration: 86499/375342 consumed_samples: 88576000 total_loss: 0.4369 time: 0.5381 s/iter data_time: 0.0530 s/iter total_throughput: 1902.97 samples/s lr: 8.76e-04 [09/19 17:06:44] lb.utils.events INFO: eta: 1 day, 19:30:41 iteration: 86599/375342 consumed_samples: 88678400 total_loss: 0.4403 time: 0.5381 s/iter data_time: 0.0536 s/iter total_throughput: 1902.95 samples/s lr: 8.76e-04 [09/19 17:07:39] lb.utils.events INFO: eta: 1 day, 19:29:53 iteration: 86699/375342 consumed_samples: 88780800 total_loss: 0.4365 time: 0.5381 s/iter data_time: 0.0533 s/iter total_throughput: 1902.93 samples/s lr: 8.75e-04 [09/19 17:08:33] lb.utils.events INFO: eta: 1 day, 19:29:14 iteration: 86799/375342 consumed_samples: 88883200 total_loss: 0.4374 time: 0.5381 s/iter data_time: 0.0502 s/iter total_throughput: 1902.91 samples/s lr: 8.75e-04 [09/19 17:09:27] lb.utils.events INFO: eta: 1 day, 19:28:40 iteration: 86899/375342 consumed_samples: 88985600 total_loss: 0.4347 time: 0.5381 s/iter data_time: 0.0534 s/iter total_throughput: 1902.89 samples/s lr: 8.75e-04 [09/19 17:10:22] lb.utils.events INFO: eta: 1 day, 19:30:00 iteration: 86999/375342 consumed_samples: 89088000 total_loss: 0.4349 time: 0.5381 s/iter data_time: 0.0535 s/iter total_throughput: 1902.86 samples/s lr: 8.74e-04 [09/19 17:11:16] lb.utils.events INFO: eta: 1 day, 19:30:25 iteration: 87099/375342 consumed_samples: 89190400 total_loss: 0.4365 time: 0.5381 s/iter data_time: 0.0501 s/iter total_throughput: 1902.84 samples/s lr: 8.74e-04 [09/19 17:12:11] lb.utils.events INFO: eta: 1 day, 19:31:26 iteration: 87199/375342 consumed_samples: 89292800 total_loss: 0.4426 time: 0.5382 s/iter data_time: 0.0504 s/iter total_throughput: 1902.81 samples/s lr: 8.74e-04 [09/19 17:13:05] lb.utils.events INFO: eta: 1 day, 19:31:43 iteration: 87299/375342 consumed_samples: 89395200 total_loss: 0.4366 time: 0.5382 s/iter data_time: 0.0536 s/iter total_throughput: 1902.78 samples/s lr: 8.74e-04 [09/19 17:14:00] lb.utils.events INFO: eta: 1 day, 19:31:16 iteration: 87399/375342 consumed_samples: 89497600 total_loss: 0.4292 time: 0.5382 s/iter data_time: 0.0539 s/iter total_throughput: 1902.76 samples/s lr: 8.73e-04 [09/19 17:14:54] lb.utils.events INFO: eta: 1 day, 19:30:25 iteration: 87499/375342 consumed_samples: 89600000 total_loss: 0.4294 time: 0.5382 s/iter data_time: 0.0534 s/iter total_throughput: 1902.74 samples/s lr: 8.73e-04 [09/19 17:15:49] lb.utils.events INFO: eta: 1 day, 19:29:29 iteration: 87599/375342 consumed_samples: 89702400 total_loss: 0.4298 time: 0.5382 s/iter data_time: 0.0528 s/iter total_throughput: 1902.71 samples/s lr: 8.73e-04 [09/19 17:16:43] lb.utils.events INFO: eta: 1 day, 19:28:54 iteration: 87699/375342 consumed_samples: 89804800 total_loss: 0.4371 time: 0.5382 s/iter data_time: 0.0519 s/iter total_throughput: 1902.69 samples/s lr: 8.73e-04 [09/19 17:17:37] lb.utils.events INFO: eta: 1 day, 19:28:37 iteration: 87799/375342 consumed_samples: 89907200 total_loss: 0.4392 time: 0.5382 s/iter data_time: 0.0500 s/iter total_throughput: 1902.66 samples/s lr: 8.72e-04 [09/19 17:18:32] lb.utils.events INFO: eta: 1 day, 19:27:56 iteration: 87899/375342 consumed_samples: 90009600 total_loss: 0.4434 time: 0.5382 s/iter data_time: 0.0492 s/iter total_throughput: 1902.64 samples/s lr: 8.72e-04 [09/19 17:19:26] lb.utils.events INFO: eta: 1 day, 19:26:48 iteration: 87999/375342 consumed_samples: 90112000 total_loss: 0.4445 time: 0.5382 s/iter data_time: 0.0484 s/iter total_throughput: 1902.61 samples/s lr: 8.72e-04 [09/19 17:20:21] lb.utils.events INFO: eta: 1 day, 19:26:31 iteration: 88099/375342 consumed_samples: 90214400 total_loss: 0.4445 time: 0.5382 s/iter data_time: 0.0495 s/iter total_throughput: 1902.59 samples/s lr: 8.71e-04 [09/19 17:21:15] lb.utils.events INFO: eta: 1 day, 19:25:07 iteration: 88199/375342 consumed_samples: 90316800 total_loss: 0.4447 time: 0.5382 s/iter data_time: 0.0478 s/iter total_throughput: 1902.56 samples/s lr: 8.71e-04 [09/19 17:22:10] lb.utils.events INFO: eta: 1 day, 19:24:19 iteration: 88299/375342 consumed_samples: 90419200 total_loss: 0.4419 time: 0.5382 s/iter data_time: 0.0499 s/iter total_throughput: 1902.53 samples/s lr: 8.71e-04 [09/19 17:23:04] lb.utils.events INFO: eta: 1 day, 19:23:40 iteration: 88399/375342 consumed_samples: 90521600 total_loss: 0.4422 time: 0.5382 s/iter data_time: 0.0482 s/iter total_throughput: 1902.51 samples/s lr: 8.71e-04 [09/19 17:23:59] lb.utils.events INFO: eta: 1 day, 19:22:46 iteration: 88499/375342 consumed_samples: 90624000 total_loss: 0.4384 time: 0.5382 s/iter data_time: 0.0503 s/iter total_throughput: 1902.49 samples/s lr: 8.70e-04 [09/19 17:24:53] lb.utils.events INFO: eta: 1 day, 19:21:42 iteration: 88599/375342 consumed_samples: 90726400 total_loss: 0.4347 time: 0.5382 s/iter data_time: 0.0483 s/iter total_throughput: 1902.47 samples/s lr: 8.70e-04 [09/19 17:25:47] lb.utils.events INFO: eta: 1 day, 19:20:35 iteration: 88699/375342 consumed_samples: 90828800 total_loss: 0.4399 time: 0.5383 s/iter data_time: 0.0497 s/iter total_throughput: 1902.45 samples/s lr: 8.70e-04 [09/19 17:26:42] lb.utils.events INFO: eta: 1 day, 19:19:07 iteration: 88799/375342 consumed_samples: 90931200 total_loss: 0.4442 time: 0.5383 s/iter data_time: 0.0502 s/iter total_throughput: 1902.43 samples/s lr: 8.69e-04 [09/19 17:27:36] lb.utils.events INFO: eta: 1 day, 19:16:37 iteration: 88899/375342 consumed_samples: 91033600 total_loss: 0.4442 time: 0.5383 s/iter data_time: 0.0488 s/iter total_throughput: 1902.41 samples/s lr: 8.69e-04 [09/19 17:28:30] lb.utils.events INFO: eta: 1 day, 19:15:18 iteration: 88999/375342 consumed_samples: 91136000 total_loss: 0.4392 time: 0.5383 s/iter data_time: 0.0496 s/iter total_throughput: 1902.39 samples/s lr: 8.69e-04 [09/19 17:29:25] lb.utils.events INFO: eta: 1 day, 19:12:54 iteration: 89099/375342 consumed_samples: 91238400 total_loss: 0.4384 time: 0.5383 s/iter data_time: 0.0486 s/iter total_throughput: 1902.38 samples/s lr: 8.69e-04 [09/19 17:30:19] lb.utils.events INFO: eta: 1 day, 19:11:00 iteration: 89199/375342 consumed_samples: 91340800 total_loss: 0.4337 time: 0.5383 s/iter data_time: 0.0486 s/iter total_throughput: 1902.36 samples/s lr: 8.68e-04 [09/19 17:31:13] lb.utils.events INFO: eta: 1 day, 19:07:41 iteration: 89299/375342 consumed_samples: 91443200 total_loss: 0.4294 time: 0.5383 s/iter data_time: 0.0507 s/iter total_throughput: 1902.35 samples/s lr: 8.68e-04 [09/19 17:32:07] lb.utils.events INFO: eta: 1 day, 19:05:41 iteration: 89399/375342 consumed_samples: 91545600 total_loss: 0.4286 time: 0.5383 s/iter data_time: 0.0481 s/iter total_throughput: 1902.33 samples/s lr: 8.68e-04 [09/19 17:33:01] lb.utils.events INFO: eta: 1 day, 19:03:41 iteration: 89499/375342 consumed_samples: 91648000 total_loss: 0.4341 time: 0.5383 s/iter data_time: 0.0495 s/iter total_throughput: 1902.32 samples/s lr: 8.67e-04 [09/19 17:33:56] lb.utils.events INFO: eta: 1 day, 19:02:56 iteration: 89599/375342 consumed_samples: 91750400 total_loss: 0.4408 time: 0.5383 s/iter data_time: 0.0478 s/iter total_throughput: 1902.30 samples/s lr: 8.67e-04 [09/19 17:34:50] lb.utils.events INFO: eta: 1 day, 19:01:57 iteration: 89699/375342 consumed_samples: 91852800 total_loss: 0.4391 time: 0.5383 s/iter data_time: 0.0553 s/iter total_throughput: 1902.27 samples/s lr: 8.67e-04 [09/19 17:35:45] lb.utils.events INFO: eta: 1 day, 19:00:28 iteration: 89799/375342 consumed_samples: 91955200 total_loss: 0.4437 time: 0.5383 s/iter data_time: 0.0519 s/iter total_throughput: 1902.25 samples/s lr: 8.67e-04 [09/19 17:36:39] lb.utils.events INFO: eta: 1 day, 18:59:38 iteration: 89899/375342 consumed_samples: 92057600 total_loss: 0.4418 time: 0.5383 s/iter data_time: 0.0532 s/iter total_throughput: 1902.23 samples/s lr: 8.66e-04 [09/19 17:37:33] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0089999 [09/19 17:37:34] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 17:37:34] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 17:37:38] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0568 s/iter. Inference: 0.2429 s/iter. Eval: 0.0021 s/iter. Total: 0.3019 s/iter. ETA=0:00:11 [09/19 17:37:44] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0653 s/iter. Inference: 0.2630 s/iter. Eval: 0.0023 s/iter. Total: 0.3309 s/iter. ETA=0:00:07 [09/19 17:37:49] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0674 s/iter. Inference: 0.2572 s/iter. Eval: 0.0022 s/iter. Total: 0.3272 s/iter. ETA=0:00:01 [09/19 17:37:51] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 17:37:51] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.099256 (0.000282 s / iter per device, on 8 devices) [09/19 17:37:51] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000226 s / iter per device, on 8 devices) [09/19 17:37:51] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 17:37:51] lb.evaluation.utils INFO: copypaste: Acc@1=71.352 [09/19 17:37:51] lb.evaluation.utils INFO: copypaste: Acc@5=90.62 [09/19 17:37:51] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 71.35200, better than last best score 70.87800 @ iteration 84999. [09/19 17:37:51] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 17:37:51] lb.utils.events INFO: eta: 1 day, 18:58:21 iteration: 89999/375342 consumed_samples: 92160000 total_loss: 0.4411 time: 0.5383 s/iter data_time: 0.0530 s/iter total_throughput: 1902.22 samples/s lr: 8.66e-04 [09/19 17:38:46] lb.utils.events INFO: eta: 1 day, 18:58:31 iteration: 90099/375342 consumed_samples: 92262400 total_loss: 0.4411 time: 0.5383 s/iter data_time: 0.0538 s/iter total_throughput: 1902.20 samples/s lr: 8.66e-04 [09/19 17:39:40] lb.utils.events INFO: eta: 1 day, 18:58:06 iteration: 90199/375342 consumed_samples: 92364800 total_loss: 0.4374 time: 0.5383 s/iter data_time: 0.0529 s/iter total_throughput: 1902.18 samples/s lr: 8.66e-04 [09/19 17:40:34] lb.utils.events INFO: eta: 1 day, 18:58:49 iteration: 90299/375342 consumed_samples: 92467200 total_loss: 0.4407 time: 0.5383 s/iter data_time: 0.0527 s/iter total_throughput: 1902.16 samples/s lr: 8.65e-04 [09/19 17:41:29] lb.utils.events INFO: eta: 1 day, 18:58:55 iteration: 90399/375342 consumed_samples: 92569600 total_loss: 0.4438 time: 0.5383 s/iter data_time: 0.0546 s/iter total_throughput: 1902.14 samples/s lr: 8.65e-04 [09/19 17:42:23] lb.utils.events INFO: eta: 1 day, 18:59:25 iteration: 90499/375342 consumed_samples: 92672000 total_loss: 0.4463 time: 0.5383 s/iter data_time: 0.0508 s/iter total_throughput: 1902.11 samples/s lr: 8.65e-04 [09/19 17:43:18] lb.utils.events INFO: eta: 1 day, 18:59:13 iteration: 90599/375342 consumed_samples: 92774400 total_loss: 0.448 time: 0.5384 s/iter data_time: 0.0538 s/iter total_throughput: 1902.09 samples/s lr: 8.64e-04 [09/19 17:44:12] lb.utils.events INFO: eta: 1 day, 18:59:49 iteration: 90699/375342 consumed_samples: 92876800 total_loss: 0.4365 time: 0.5384 s/iter data_time: 0.0517 s/iter total_throughput: 1902.06 samples/s lr: 8.64e-04 [09/19 17:45:07] lb.utils.events INFO: eta: 1 day, 18:59:26 iteration: 90799/375342 consumed_samples: 92979200 total_loss: 0.4365 time: 0.5384 s/iter data_time: 0.0526 s/iter total_throughput: 1902.04 samples/s lr: 8.64e-04 [09/19 17:46:01] lb.utils.events INFO: eta: 1 day, 18:59:24 iteration: 90899/375342 consumed_samples: 93081600 total_loss: 0.4382 time: 0.5384 s/iter data_time: 0.0538 s/iter total_throughput: 1902.01 samples/s lr: 8.64e-04 [09/19 17:46:56] lb.utils.events INFO: eta: 1 day, 18:59:24 iteration: 90999/375342 consumed_samples: 93184000 total_loss: 0.4386 time: 0.5384 s/iter data_time: 0.0507 s/iter total_throughput: 1901.99 samples/s lr: 8.63e-04 [09/19 17:47:50] lb.utils.events INFO: eta: 1 day, 18:58:31 iteration: 91099/375342 consumed_samples: 93286400 total_loss: 0.4429 time: 0.5384 s/iter data_time: 0.0507 s/iter total_throughput: 1901.97 samples/s lr: 8.63e-04 [09/19 17:48:45] lb.utils.events INFO: eta: 1 day, 18:58:00 iteration: 91199/375342 consumed_samples: 93388800 total_loss: 0.4344 time: 0.5384 s/iter data_time: 0.0489 s/iter total_throughput: 1901.95 samples/s lr: 8.63e-04 [09/19 17:49:39] lb.utils.events INFO: eta: 1 day, 18:56:54 iteration: 91299/375342 consumed_samples: 93491200 total_loss: 0.4285 time: 0.5384 s/iter data_time: 0.0483 s/iter total_throughput: 1901.93 samples/s lr: 8.62e-04 [09/19 17:50:33] lb.utils.events INFO: eta: 1 day, 18:55:56 iteration: 91399/375342 consumed_samples: 93593600 total_loss: 0.4341 time: 0.5384 s/iter data_time: 0.0480 s/iter total_throughput: 1901.90 samples/s lr: 8.62e-04 [09/19 17:51:28] lb.utils.events INFO: eta: 1 day, 18:54:55 iteration: 91499/375342 consumed_samples: 93696000 total_loss: 0.4435 time: 0.5384 s/iter data_time: 0.0476 s/iter total_throughput: 1901.88 samples/s lr: 8.62e-04 [09/19 17:52:22] lb.utils.events INFO: eta: 1 day, 18:53:45 iteration: 91599/375342 consumed_samples: 93798400 total_loss: 0.446 time: 0.5384 s/iter data_time: 0.0467 s/iter total_throughput: 1901.86 samples/s lr: 8.62e-04 [09/19 17:53:17] lb.utils.events INFO: eta: 1 day, 18:52:13 iteration: 91699/375342 consumed_samples: 93900800 total_loss: 0.434 time: 0.5384 s/iter data_time: 0.0502 s/iter total_throughput: 1901.84 samples/s lr: 8.61e-04 [09/19 17:54:11] lb.utils.events INFO: eta: 1 day, 18:51:13 iteration: 91799/375342 consumed_samples: 94003200 total_loss: 0.4367 time: 0.5384 s/iter data_time: 0.0488 s/iter total_throughput: 1901.82 samples/s lr: 8.61e-04 [09/19 17:55:06] lb.utils.events INFO: eta: 1 day, 18:49:50 iteration: 91899/375342 consumed_samples: 94105600 total_loss: 0.4353 time: 0.5384 s/iter data_time: 0.0499 s/iter total_throughput: 1901.80 samples/s lr: 8.61e-04 [09/19 17:56:00] lb.utils.events INFO: eta: 1 day, 18:49:00 iteration: 91999/375342 consumed_samples: 94208000 total_loss: 0.4286 time: 0.5384 s/iter data_time: 0.0494 s/iter total_throughput: 1901.78 samples/s lr: 8.60e-04 [09/19 17:56:54] lb.utils.events INFO: eta: 1 day, 18:47:55 iteration: 92099/375342 consumed_samples: 94310400 total_loss: 0.4296 time: 0.5384 s/iter data_time: 0.0486 s/iter total_throughput: 1901.76 samples/s lr: 8.60e-04 [09/19 17:57:49] lb.utils.events INFO: eta: 1 day, 18:46:28 iteration: 92199/375342 consumed_samples: 94412800 total_loss: 0.4369 time: 0.5385 s/iter data_time: 0.0507 s/iter total_throughput: 1901.74 samples/s lr: 8.60e-04 [09/19 17:58:43] lb.utils.events INFO: eta: 1 day, 18:44:06 iteration: 92299/375342 consumed_samples: 94515200 total_loss: 0.4441 time: 0.5385 s/iter data_time: 0.0505 s/iter total_throughput: 1901.73 samples/s lr: 8.59e-04 [09/19 17:59:37] lb.utils.events INFO: eta: 1 day, 18:42:36 iteration: 92399/375342 consumed_samples: 94617600 total_loss: 0.4363 time: 0.5385 s/iter data_time: 0.0493 s/iter total_throughput: 1901.71 samples/s lr: 8.59e-04 [09/19 18:00:31] lb.utils.events INFO: eta: 1 day, 18:40:09 iteration: 92499/375342 consumed_samples: 94720000 total_loss: 0.4384 time: 0.5385 s/iter data_time: 0.0494 s/iter total_throughput: 1901.70 samples/s lr: 8.59e-04 [09/19 18:01:26] lb.utils.events INFO: eta: 1 day, 18:37:52 iteration: 92599/375342 consumed_samples: 94822400 total_loss: 0.4383 time: 0.5385 s/iter data_time: 0.0504 s/iter total_throughput: 1901.69 samples/s lr: 8.59e-04 [09/19 18:02:20] lb.utils.events INFO: eta: 1 day, 18:36:08 iteration: 92699/375342 consumed_samples: 94924800 total_loss: 0.4427 time: 0.5385 s/iter data_time: 0.0502 s/iter total_throughput: 1901.67 samples/s lr: 8.58e-04 [09/19 18:03:14] lb.utils.events INFO: eta: 1 day, 18:34:27 iteration: 92799/375342 consumed_samples: 95027200 total_loss: 0.4469 time: 0.5385 s/iter data_time: 0.0471 s/iter total_throughput: 1901.66 samples/s lr: 8.58e-04 [09/19 18:04:08] lb.utils.events INFO: eta: 1 day, 18:32:52 iteration: 92899/375342 consumed_samples: 95129600 total_loss: 0.4327 time: 0.5385 s/iter data_time: 0.0469 s/iter total_throughput: 1901.65 samples/s lr: 8.58e-04 [09/19 18:05:03] lb.utils.events INFO: eta: 1 day, 18:31:22 iteration: 92999/375342 consumed_samples: 95232000 total_loss: 0.4269 time: 0.5385 s/iter data_time: 0.0482 s/iter total_throughput: 1901.63 samples/s lr: 8.57e-04 [09/19 18:05:57] lb.utils.events INFO: eta: 1 day, 18:30:01 iteration: 93099/375342 consumed_samples: 95334400 total_loss: 0.4324 time: 0.5385 s/iter data_time: 0.0478 s/iter total_throughput: 1901.62 samples/s lr: 8.57e-04 [09/19 18:06:51] lb.utils.events INFO: eta: 1 day, 18:29:17 iteration: 93199/375342 consumed_samples: 95436800 total_loss: 0.4412 time: 0.5385 s/iter data_time: 0.0531 s/iter total_throughput: 1901.58 samples/s lr: 8.57e-04 [09/19 18:07:46] lb.utils.events INFO: eta: 1 day, 18:29:25 iteration: 93299/375342 consumed_samples: 95539200 total_loss: 0.4412 time: 0.5385 s/iter data_time: 0.0535 s/iter total_throughput: 1901.56 samples/s lr: 8.57e-04 [09/19 18:08:40] lb.utils.events INFO: eta: 1 day, 18:29:44 iteration: 93399/375342 consumed_samples: 95641600 total_loss: 0.4334 time: 0.5385 s/iter data_time: 0.0532 s/iter total_throughput: 1901.54 samples/s lr: 8.56e-04 [09/19 18:09:35] lb.utils.events INFO: eta: 1 day, 18:29:59 iteration: 93499/375342 consumed_samples: 95744000 total_loss: 0.4378 time: 0.5385 s/iter data_time: 0.0554 s/iter total_throughput: 1901.52 samples/s lr: 8.56e-04 [09/19 18:10:29] lb.utils.events INFO: eta: 1 day, 18:30:54 iteration: 93599/375342 consumed_samples: 95846400 total_loss: 0.4361 time: 0.5385 s/iter data_time: 0.0549 s/iter total_throughput: 1901.49 samples/s lr: 8.56e-04 [09/19 18:11:24] lb.utils.events INFO: eta: 1 day, 18:31:50 iteration: 93699/375342 consumed_samples: 95948800 total_loss: 0.4336 time: 0.5385 s/iter data_time: 0.0558 s/iter total_throughput: 1901.46 samples/s lr: 8.55e-04 [09/19 18:12:19] lb.utils.events INFO: eta: 1 day, 18:33:41 iteration: 93799/375342 consumed_samples: 96051200 total_loss: 0.4367 time: 0.5385 s/iter data_time: 0.0568 s/iter total_throughput: 1901.43 samples/s lr: 8.55e-04 [09/19 18:13:13] lb.utils.events INFO: eta: 1 day, 18:35:03 iteration: 93899/375342 consumed_samples: 96153600 total_loss: 0.4357 time: 0.5385 s/iter data_time: 0.0531 s/iter total_throughput: 1901.41 samples/s lr: 8.55e-04 [09/19 18:14:08] lb.utils.events INFO: eta: 1 day, 18:36:05 iteration: 93999/375342 consumed_samples: 96256000 total_loss: 0.4351 time: 0.5386 s/iter data_time: 0.0553 s/iter total_throughput: 1901.38 samples/s lr: 8.55e-04 [09/19 18:15:03] lb.utils.events INFO: eta: 1 day, 18:37:32 iteration: 94099/375342 consumed_samples: 96358400 total_loss: 0.4406 time: 0.5386 s/iter data_time: 0.0558 s/iter total_throughput: 1901.35 samples/s lr: 8.54e-04 [09/19 18:15:57] lb.utils.events INFO: eta: 1 day, 18:37:37 iteration: 94199/375342 consumed_samples: 96460800 total_loss: 0.436 time: 0.5386 s/iter data_time: 0.0549 s/iter total_throughput: 1901.32 samples/s lr: 8.54e-04 [09/19 18:16:52] lb.utils.events INFO: eta: 1 day, 18:36:50 iteration: 94299/375342 consumed_samples: 96563200 total_loss: 0.438 time: 0.5386 s/iter data_time: 0.0535 s/iter total_throughput: 1901.30 samples/s lr: 8.54e-04 [09/19 18:17:46] lb.utils.events INFO: eta: 1 day, 18:35:24 iteration: 94399/375342 consumed_samples: 96665600 total_loss: 0.4437 time: 0.5386 s/iter data_time: 0.0547 s/iter total_throughput: 1901.27 samples/s lr: 8.53e-04 [09/19 18:18:41] lb.utils.events INFO: eta: 1 day, 18:34:40 iteration: 94499/375342 consumed_samples: 96768000 total_loss: 0.4459 time: 0.5386 s/iter data_time: 0.0522 s/iter total_throughput: 1901.25 samples/s lr: 8.53e-04 [09/19 18:19:35] lb.utils.events INFO: eta: 1 day, 18:33:37 iteration: 94599/375342 consumed_samples: 96870400 total_loss: 0.4407 time: 0.5386 s/iter data_time: 0.0504 s/iter total_throughput: 1901.23 samples/s lr: 8.53e-04 [09/19 18:20:30] lb.utils.events INFO: eta: 1 day, 18:32:39 iteration: 94699/375342 consumed_samples: 96972800 total_loss: 0.4342 time: 0.5386 s/iter data_time: 0.0510 s/iter total_throughput: 1901.21 samples/s lr: 8.52e-04 [09/19 18:21:24] lb.utils.events INFO: eta: 1 day, 18:30:19 iteration: 94799/375342 consumed_samples: 97075200 total_loss: 0.4356 time: 0.5386 s/iter data_time: 0.0496 s/iter total_throughput: 1901.18 samples/s lr: 8.52e-04 [09/19 18:22:19] lb.utils.events INFO: eta: 1 day, 18:28:47 iteration: 94899/375342 consumed_samples: 97177600 total_loss: 0.4377 time: 0.5386 s/iter data_time: 0.0501 s/iter total_throughput: 1901.16 samples/s lr: 8.52e-04 [09/19 18:23:13] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0094999 [09/19 18:23:14] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 18:23:14] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 18:23:19] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0655 s/iter. Inference: 0.2457 s/iter. Eval: 0.0023 s/iter. Total: 0.3135 s/iter. ETA=0:00:11 [09/19 18:23:24] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0823 s/iter. Inference: 0.2509 s/iter. Eval: 0.0022 s/iter. Total: 0.3357 s/iter. ETA=0:00:07 [09/19 18:23:29] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0764 s/iter. Inference: 0.2510 s/iter. Eval: 0.0023 s/iter. Total: 0.3300 s/iter. ETA=0:00:01 [09/19 18:23:31] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 18:23:31] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.152430 (0.000283 s / iter per device, on 8 devices) [09/19 18:23:31] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000220 s / iter per device, on 8 devices) [09/19 18:23:31] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 18:23:31] lb.evaluation.utils INFO: copypaste: Acc@1=70.66 [09/19 18:23:31] lb.evaluation.utils INFO: copypaste: Acc@5=90.338 [09/19 18:23:31] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 70.66000, not better than best score 71.35200 @ iteration 89999. [09/19 18:23:31] lb.utils.events INFO: eta: 1 day, 18:27:23 iteration: 94999/375342 consumed_samples: 97280000 total_loss: 0.4333 time: 0.5386 s/iter data_time: 0.0506 s/iter total_throughput: 1901.13 samples/s lr: 8.52e-04 [09/19 18:24:25] lb.utils.events INFO: eta: 1 day, 18:26:02 iteration: 95099/375342 consumed_samples: 97382400 total_loss: 0.4411 time: 0.5386 s/iter data_time: 0.0515 s/iter total_throughput: 1901.11 samples/s lr: 8.51e-04 [09/19 18:25:20] lb.utils.events INFO: eta: 1 day, 18:22:56 iteration: 95199/375342 consumed_samples: 97484800 total_loss: 0.4384 time: 0.5386 s/iter data_time: 0.0494 s/iter total_throughput: 1901.09 samples/s lr: 8.51e-04 [09/19 18:26:14] lb.utils.events INFO: eta: 1 day, 18:21:15 iteration: 95299/375342 consumed_samples: 97587200 total_loss: 0.43 time: 0.5386 s/iter data_time: 0.0517 s/iter total_throughput: 1901.07 samples/s lr: 8.51e-04 [09/19 18:27:09] lb.utils.events INFO: eta: 1 day, 18:21:07 iteration: 95399/375342 consumed_samples: 97689600 total_loss: 0.4363 time: 0.5386 s/iter data_time: 0.0503 s/iter total_throughput: 1901.05 samples/s lr: 8.50e-04 [09/19 18:28:03] lb.utils.events INFO: eta: 1 day, 18:19:55 iteration: 95499/375342 consumed_samples: 97792000 total_loss: 0.4516 time: 0.5387 s/iter data_time: 0.0507 s/iter total_throughput: 1901.03 samples/s lr: 8.50e-04 [09/19 18:28:58] lb.utils.events INFO: eta: 1 day, 18:18:38 iteration: 95599/375342 consumed_samples: 97894400 total_loss: 0.4424 time: 0.5387 s/iter data_time: 0.0500 s/iter total_throughput: 1901.01 samples/s lr: 8.50e-04 [09/19 18:29:52] lb.utils.events INFO: eta: 1 day, 18:16:37 iteration: 95699/375342 consumed_samples: 97996800 total_loss: 0.4331 time: 0.5387 s/iter data_time: 0.0497 s/iter total_throughput: 1900.99 samples/s lr: 8.50e-04 [09/19 18:30:46] lb.utils.events INFO: eta: 1 day, 18:15:17 iteration: 95799/375342 consumed_samples: 98099200 total_loss: 0.4409 time: 0.5387 s/iter data_time: 0.0522 s/iter total_throughput: 1900.97 samples/s lr: 8.49e-04 [09/19 18:31:41] lb.utils.events INFO: eta: 1 day, 18:14:05 iteration: 95899/375342 consumed_samples: 98201600 total_loss: 0.4434 time: 0.5387 s/iter data_time: 0.0510 s/iter total_throughput: 1900.96 samples/s lr: 8.49e-04 [09/19 18:32:35] lb.utils.events INFO: eta: 1 day, 18:11:21 iteration: 95999/375342 consumed_samples: 98304000 total_loss: 0.4377 time: 0.5387 s/iter data_time: 0.0523 s/iter total_throughput: 1900.94 samples/s lr: 8.49e-04 [09/19 18:33:29] lb.utils.events INFO: eta: 1 day, 18:09:34 iteration: 96099/375342 consumed_samples: 98406400 total_loss: 0.4338 time: 0.5387 s/iter data_time: 0.0513 s/iter total_throughput: 1900.93 samples/s lr: 8.48e-04 [09/19 18:34:23] lb.utils.events INFO: eta: 1 day, 18:08:25 iteration: 96199/375342 consumed_samples: 98508800 total_loss: 0.4343 time: 0.5387 s/iter data_time: 0.0515 s/iter total_throughput: 1900.92 samples/s lr: 8.48e-04 [09/19 18:35:18] lb.utils.events INFO: eta: 1 day, 18:07:21 iteration: 96299/375342 consumed_samples: 98611200 total_loss: 0.4353 time: 0.5387 s/iter data_time: 0.0479 s/iter total_throughput: 1900.90 samples/s lr: 8.48e-04 [09/19 18:36:12] lb.utils.events INFO: eta: 1 day, 18:05:39 iteration: 96399/375342 consumed_samples: 98713600 total_loss: 0.4365 time: 0.5387 s/iter data_time: 0.0472 s/iter total_throughput: 1900.88 samples/s lr: 8.47e-04 [09/19 18:37:07] lb.utils.events INFO: eta: 1 day, 18:04:05 iteration: 96499/375342 consumed_samples: 98816000 total_loss: 0.4322 time: 0.5387 s/iter data_time: 0.0475 s/iter total_throughput: 1900.86 samples/s lr: 8.47e-04 [09/19 18:38:01] lb.utils.events INFO: eta: 1 day, 18:03:22 iteration: 96599/375342 consumed_samples: 98918400 total_loss: 0.4383 time: 0.5387 s/iter data_time: 0.0568 s/iter total_throughput: 1900.83 samples/s lr: 8.47e-04 [09/19 18:38:56] lb.utils.events INFO: eta: 1 day, 18:02:35 iteration: 96699/375342 consumed_samples: 99020800 total_loss: 0.4404 time: 0.5387 s/iter data_time: 0.0528 s/iter total_throughput: 1900.81 samples/s lr: 8.47e-04 [09/19 18:39:50] lb.utils.events INFO: eta: 1 day, 18:01:40 iteration: 96799/375342 consumed_samples: 99123200 total_loss: 0.4341 time: 0.5387 s/iter data_time: 0.0548 s/iter total_throughput: 1900.79 samples/s lr: 8.46e-04 [09/19 18:40:45] lb.utils.events INFO: eta: 1 day, 18:00:46 iteration: 96899/375342 consumed_samples: 99225600 total_loss: 0.4281 time: 0.5387 s/iter data_time: 0.0531 s/iter total_throughput: 1900.77 samples/s lr: 8.46e-04 [09/19 18:41:39] lb.utils.events INFO: eta: 1 day, 18:00:20 iteration: 96999/375342 consumed_samples: 99328000 total_loss: 0.4352 time: 0.5387 s/iter data_time: 0.0525 s/iter total_throughput: 1900.75 samples/s lr: 8.46e-04 [09/19 18:42:34] lb.utils.events INFO: eta: 1 day, 18:01:02 iteration: 97099/375342 consumed_samples: 99430400 total_loss: 0.4438 time: 0.5387 s/iter data_time: 0.0551 s/iter total_throughput: 1900.73 samples/s lr: 8.45e-04 [09/19 18:43:28] lb.utils.events INFO: eta: 1 day, 18:02:01 iteration: 97199/375342 consumed_samples: 99532800 total_loss: 0.4446 time: 0.5387 s/iter data_time: 0.0533 s/iter total_throughput: 1900.70 samples/s lr: 8.45e-04 [09/19 18:44:23] lb.utils.events INFO: eta: 1 day, 18:02:19 iteration: 97299/375342 consumed_samples: 99635200 total_loss: 0.4455 time: 0.5388 s/iter data_time: 0.0553 s/iter total_throughput: 1900.68 samples/s lr: 8.45e-04 [09/19 18:45:18] lb.utils.events INFO: eta: 1 day, 18:02:47 iteration: 97399/375342 consumed_samples: 99737600 total_loss: 0.4421 time: 0.5388 s/iter data_time: 0.0542 s/iter total_throughput: 1900.65 samples/s lr: 8.44e-04 [09/19 18:46:12] lb.utils.events INFO: eta: 1 day, 18:02:42 iteration: 97499/375342 consumed_samples: 99840000 total_loss: 0.4408 time: 0.5388 s/iter data_time: 0.0548 s/iter total_throughput: 1900.63 samples/s lr: 8.44e-04 [09/19 18:47:07] lb.utils.events INFO: eta: 1 day, 18:02:46 iteration: 97599/375342 consumed_samples: 99942400 total_loss: 0.4404 time: 0.5388 s/iter data_time: 0.0545 s/iter total_throughput: 1900.60 samples/s lr: 8.44e-04 [09/19 18:48:01] lb.utils.events INFO: eta: 1 day, 18:02:24 iteration: 97699/375342 consumed_samples: 100044800 total_loss: 0.4347 time: 0.5388 s/iter data_time: 0.0544 s/iter total_throughput: 1900.58 samples/s lr: 8.44e-04 [09/19 18:48:56] lb.utils.events INFO: eta: 1 day, 18:01:43 iteration: 97799/375342 consumed_samples: 100147200 total_loss: 0.4288 time: 0.5388 s/iter data_time: 0.0556 s/iter total_throughput: 1900.56 samples/s lr: 8.43e-04 [09/19 18:49:50] lb.utils.events INFO: eta: 1 day, 18:00:59 iteration: 97899/375342 consumed_samples: 100249600 total_loss: 0.4362 time: 0.5388 s/iter data_time: 0.0519 s/iter total_throughput: 1900.54 samples/s lr: 8.43e-04 [09/19 18:50:45] lb.utils.events INFO: eta: 1 day, 18:00:27 iteration: 97999/375342 consumed_samples: 100352000 total_loss: 0.436 time: 0.5388 s/iter data_time: 0.0501 s/iter total_throughput: 1900.52 samples/s lr: 8.43e-04 [09/19 18:51:39] lb.utils.events INFO: eta: 1 day, 17:59:38 iteration: 98099/375342 consumed_samples: 100454400 total_loss: 0.4355 time: 0.5388 s/iter data_time: 0.0499 s/iter total_throughput: 1900.49 samples/s lr: 8.42e-04 [09/19 18:52:34] lb.utils.events INFO: eta: 1 day, 17:57:55 iteration: 98199/375342 consumed_samples: 100556800 total_loss: 0.4304 time: 0.5388 s/iter data_time: 0.0487 s/iter total_throughput: 1900.47 samples/s lr: 8.42e-04 [09/19 18:53:28] lb.utils.events INFO: eta: 1 day, 17:56:39 iteration: 98299/375342 consumed_samples: 100659200 total_loss: 0.426 time: 0.5388 s/iter data_time: 0.0503 s/iter total_throughput: 1900.45 samples/s lr: 8.42e-04 [09/19 18:54:23] lb.utils.events INFO: eta: 1 day, 17:55:09 iteration: 98399/375342 consumed_samples: 100761600 total_loss: 0.4304 time: 0.5388 s/iter data_time: 0.0500 s/iter total_throughput: 1900.43 samples/s lr: 8.41e-04 [09/19 18:55:17] lb.utils.events INFO: eta: 1 day, 17:53:44 iteration: 98499/375342 consumed_samples: 100864000 total_loss: 0.4398 time: 0.5388 s/iter data_time: 0.0492 s/iter total_throughput: 1900.40 samples/s lr: 8.41e-04 [09/19 18:56:12] lb.utils.events INFO: eta: 1 day, 17:52:27 iteration: 98599/375342 consumed_samples: 100966400 total_loss: 0.4422 time: 0.5388 s/iter data_time: 0.0499 s/iter total_throughput: 1900.38 samples/s lr: 8.41e-04 [09/19 18:57:06] lb.utils.events INFO: eta: 1 day, 17:50:52 iteration: 98699/375342 consumed_samples: 101068800 total_loss: 0.4401 time: 0.5388 s/iter data_time: 0.0507 s/iter total_throughput: 1900.36 samples/s lr: 8.40e-04 [09/19 18:58:01] lb.utils.events INFO: eta: 1 day, 17:49:42 iteration: 98799/375342 consumed_samples: 101171200 total_loss: 0.437 time: 0.5388 s/iter data_time: 0.0506 s/iter total_throughput: 1900.34 samples/s lr: 8.40e-04 [09/19 18:58:55] lb.utils.events INFO: eta: 1 day, 17:49:03 iteration: 98899/375342 consumed_samples: 101273600 total_loss: 0.4367 time: 0.5389 s/iter data_time: 0.0530 s/iter total_throughput: 1900.32 samples/s lr: 8.40e-04 [09/19 18:59:50] lb.utils.events INFO: eta: 1 day, 17:48:07 iteration: 98999/375342 consumed_samples: 101376000 total_loss: 0.4352 time: 0.5389 s/iter data_time: 0.0505 s/iter total_throughput: 1900.30 samples/s lr: 8.40e-04 [09/19 19:00:44] lb.utils.events INFO: eta: 1 day, 17:46:36 iteration: 99099/375342 consumed_samples: 101478400 total_loss: 0.4369 time: 0.5389 s/iter data_time: 0.0502 s/iter total_throughput: 1900.29 samples/s lr: 8.39e-04 [09/19 19:01:38] lb.utils.events INFO: eta: 1 day, 17:45:00 iteration: 99199/375342 consumed_samples: 101580800 total_loss: 0.4404 time: 0.5389 s/iter data_time: 0.0511 s/iter total_throughput: 1900.27 samples/s lr: 8.39e-04 [09/19 19:02:33] lb.utils.events INFO: eta: 1 day, 17:43:52 iteration: 99299/375342 consumed_samples: 101683200 total_loss: 0.4368 time: 0.5389 s/iter data_time: 0.0509 s/iter total_throughput: 1900.25 samples/s lr: 8.39e-04 [09/19 19:03:27] lb.utils.events INFO: eta: 1 day, 17:42:20 iteration: 99399/375342 consumed_samples: 101785600 total_loss: 0.4327 time: 0.5389 s/iter data_time: 0.0523 s/iter total_throughput: 1900.24 samples/s lr: 8.38e-04 [09/19 19:04:22] lb.utils.events INFO: eta: 1 day, 17:40:24 iteration: 99499/375342 consumed_samples: 101888000 total_loss: 0.4347 time: 0.5389 s/iter data_time: 0.0513 s/iter total_throughput: 1900.22 samples/s lr: 8.38e-04 [09/19 19:05:16] lb.utils.events INFO: eta: 1 day, 17:38:41 iteration: 99599/375342 consumed_samples: 101990400 total_loss: 0.4348 time: 0.5389 s/iter data_time: 0.0508 s/iter total_throughput: 1900.21 samples/s lr: 8.38e-04 [09/19 19:06:10] lb.utils.events INFO: eta: 1 day, 17:37:25 iteration: 99699/375342 consumed_samples: 102092800 total_loss: 0.4424 time: 0.5389 s/iter data_time: 0.0473 s/iter total_throughput: 1900.19 samples/s lr: 8.37e-04 [09/19 19:07:05] lb.utils.events INFO: eta: 1 day, 17:36:12 iteration: 99799/375342 consumed_samples: 102195200 total_loss: 0.4424 time: 0.5389 s/iter data_time: 0.0513 s/iter total_throughput: 1900.17 samples/s lr: 8.37e-04 [09/19 19:07:59] lb.utils.events INFO: eta: 1 day, 17:34:44 iteration: 99899/375342 consumed_samples: 102297600 total_loss: 0.4443 time: 0.5389 s/iter data_time: 0.0497 s/iter total_throughput: 1900.15 samples/s lr: 8.37e-04 [09/19 19:08:53] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0099999 [09/19 19:08:54] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 19:08:54] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 19:08:59] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0523 s/iter. Inference: 0.2548 s/iter. Eval: 0.0028 s/iter. Total: 0.3099 s/iter. ETA=0:00:11 [09/19 19:09:04] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0729 s/iter. Inference: 0.2572 s/iter. Eval: 0.0027 s/iter. Total: 0.3329 s/iter. ETA=0:00:07 [09/19 19:09:09] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0728 s/iter. Inference: 0.2554 s/iter. Eval: 0.0026 s/iter. Total: 0.3311 s/iter. ETA=0:00:01 [09/19 19:09:11] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 19:09:11] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.216451 (0.000284 s / iter per device, on 8 devices) [09/19 19:09:11] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000223 s / iter per device, on 8 devices) [09/19 19:09:11] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 19:09:11] lb.evaluation.utils INFO: copypaste: Acc@1=71.454 [09/19 19:09:11] lb.evaluation.utils INFO: copypaste: Acc@5=90.622 [09/19 19:09:11] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 71.45400, better than last best score 71.35200 @ iteration 89999. [09/19 19:09:11] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 19:09:12] lb.utils.events INFO: eta: 1 day, 17:34:23 iteration: 99999/375342 consumed_samples: 102400000 total_loss: 0.4396 time: 0.5389 s/iter data_time: 0.0503 s/iter total_throughput: 1900.13 samples/s lr: 8.37e-04 [09/19 19:10:07] lb.utils.events INFO: eta: 1 day, 17:34:23 iteration: 100099/375342 consumed_samples: 102502400 total_loss: 0.4345 time: 0.5389 s/iter data_time: 0.0540 s/iter total_throughput: 1900.10 samples/s lr: 8.36e-04 [09/19 19:11:01] lb.utils.events INFO: eta: 1 day, 17:34:12 iteration: 100199/375342 consumed_samples: 102604800 total_loss: 0.4363 time: 0.5389 s/iter data_time: 0.0571 s/iter total_throughput: 1900.08 samples/s lr: 8.36e-04 [09/19 19:11:56] lb.utils.events INFO: eta: 1 day, 17:33:44 iteration: 100299/375342 consumed_samples: 102707200 total_loss: 0.4322 time: 0.5389 s/iter data_time: 0.0538 s/iter total_throughput: 1900.06 samples/s lr: 8.36e-04 [09/19 19:12:50] lb.utils.events INFO: eta: 1 day, 17:33:43 iteration: 100399/375342 consumed_samples: 102809600 total_loss: 0.4352 time: 0.5389 s/iter data_time: 0.0546 s/iter total_throughput: 1900.04 samples/s lr: 8.35e-04 [09/19 19:13:45] lb.utils.events INFO: eta: 1 day, 17:34:31 iteration: 100499/375342 consumed_samples: 102912000 total_loss: 0.4397 time: 0.5389 s/iter data_time: 0.0553 s/iter total_throughput: 1900.02 samples/s lr: 8.35e-04 [09/19 19:14:39] lb.utils.events INFO: eta: 1 day, 17:34:44 iteration: 100599/375342 consumed_samples: 103014400 total_loss: 0.4449 time: 0.5389 s/iter data_time: 0.0562 s/iter total_throughput: 1899.99 samples/s lr: 8.35e-04 [09/19 19:15:34] lb.utils.events INFO: eta: 1 day, 17:34:48 iteration: 100699/375342 consumed_samples: 103116800 total_loss: 0.4373 time: 0.5390 s/iter data_time: 0.0555 s/iter total_throughput: 1899.97 samples/s lr: 8.34e-04 [09/19 19:16:29] lb.utils.events INFO: eta: 1 day, 17:34:54 iteration: 100799/375342 consumed_samples: 103219200 total_loss: 0.4362 time: 0.5390 s/iter data_time: 0.0541 s/iter total_throughput: 1899.94 samples/s lr: 8.34e-04 [09/19 19:17:23] lb.utils.events INFO: eta: 1 day, 17:34:18 iteration: 100899/375342 consumed_samples: 103321600 total_loss: 0.4365 time: 0.5390 s/iter data_time: 0.0544 s/iter total_throughput: 1899.92 samples/s lr: 8.34e-04 [09/19 19:18:18] lb.utils.events INFO: eta: 1 day, 17:34:17 iteration: 100999/375342 consumed_samples: 103424000 total_loss: 0.4342 time: 0.5390 s/iter data_time: 0.0547 s/iter total_throughput: 1899.89 samples/s lr: 8.33e-04 [09/19 19:19:13] lb.utils.events INFO: eta: 1 day, 17:33:38 iteration: 101099/375342 consumed_samples: 103526400 total_loss: 0.4266 time: 0.5390 s/iter data_time: 0.0530 s/iter total_throughput: 1899.87 samples/s lr: 8.33e-04 [09/19 19:20:07] lb.utils.events INFO: eta: 1 day, 17:32:48 iteration: 101199/375342 consumed_samples: 103628800 total_loss: 0.4342 time: 0.5390 s/iter data_time: 0.0542 s/iter total_throughput: 1899.85 samples/s lr: 8.33e-04 [09/19 19:21:02] lb.utils.events INFO: eta: 1 day, 17:31:42 iteration: 101299/375342 consumed_samples: 103731200 total_loss: 0.4423 time: 0.5390 s/iter data_time: 0.0544 s/iter total_throughput: 1899.82 samples/s lr: 8.32e-04 [09/19 19:21:56] lb.utils.events INFO: eta: 1 day, 17:31:01 iteration: 101399/375342 consumed_samples: 103833600 total_loss: 0.4334 time: 0.5390 s/iter data_time: 0.0491 s/iter total_throughput: 1899.80 samples/s lr: 8.32e-04 [09/19 19:22:51] lb.utils.events INFO: eta: 1 day, 17:30:23 iteration: 101499/375342 consumed_samples: 103936000 total_loss: 0.4375 time: 0.5390 s/iter data_time: 0.0505 s/iter total_throughput: 1899.78 samples/s lr: 8.32e-04 [09/19 19:23:45] lb.utils.events INFO: eta: 1 day, 17:30:01 iteration: 101599/375342 consumed_samples: 104038400 total_loss: 0.44 time: 0.5390 s/iter data_time: 0.0506 s/iter total_throughput: 1899.75 samples/s lr: 8.32e-04 [09/19 19:24:40] lb.utils.events INFO: eta: 1 day, 17:28:49 iteration: 101699/375342 consumed_samples: 104140800 total_loss: 0.4387 time: 0.5390 s/iter data_time: 0.0506 s/iter total_throughput: 1899.73 samples/s lr: 8.31e-04 [09/19 19:25:35] lb.utils.events INFO: eta: 1 day, 17:27:15 iteration: 101799/375342 consumed_samples: 104243200 total_loss: 0.4386 time: 0.5390 s/iter data_time: 0.0502 s/iter total_throughput: 1899.71 samples/s lr: 8.31e-04 [09/19 19:26:29] lb.utils.events INFO: eta: 1 day, 17:26:05 iteration: 101899/375342 consumed_samples: 104345600 total_loss: 0.4386 time: 0.5390 s/iter data_time: 0.0506 s/iter total_throughput: 1899.69 samples/s lr: 8.31e-04 [09/19 19:27:24] lb.utils.events INFO: eta: 1 day, 17:24:56 iteration: 101999/375342 consumed_samples: 104448000 total_loss: 0.4386 time: 0.5390 s/iter data_time: 0.0497 s/iter total_throughput: 1899.67 samples/s lr: 8.30e-04 [09/19 19:28:18] lb.utils.events INFO: eta: 1 day, 17:23:56 iteration: 102099/375342 consumed_samples: 104550400 total_loss: 0.4367 time: 0.5390 s/iter data_time: 0.0508 s/iter total_throughput: 1899.65 samples/s lr: 8.30e-04 [09/19 19:29:13] lb.utils.events INFO: eta: 1 day, 17:22:00 iteration: 102199/375342 consumed_samples: 104652800 total_loss: 0.4384 time: 0.5391 s/iter data_time: 0.0511 s/iter total_throughput: 1899.63 samples/s lr: 8.30e-04 [09/19 19:30:07] lb.utils.events INFO: eta: 1 day, 17:21:02 iteration: 102299/375342 consumed_samples: 104755200 total_loss: 0.4359 time: 0.5391 s/iter data_time: 0.0509 s/iter total_throughput: 1899.61 samples/s lr: 8.29e-04 [09/19 19:31:02] lb.utils.events INFO: eta: 1 day, 17:19:39 iteration: 102399/375342 consumed_samples: 104857600 total_loss: 0.4353 time: 0.5391 s/iter data_time: 0.0508 s/iter total_throughput: 1899.59 samples/s lr: 8.29e-04 [09/19 19:31:56] lb.utils.events INFO: eta: 1 day, 17:17:56 iteration: 102499/375342 consumed_samples: 104960000 total_loss: 0.4397 time: 0.5391 s/iter data_time: 0.0520 s/iter total_throughput: 1899.57 samples/s lr: 8.29e-04 [09/19 19:32:51] lb.utils.events INFO: eta: 1 day, 17:16:03 iteration: 102599/375342 consumed_samples: 105062400 total_loss: 0.4397 time: 0.5391 s/iter data_time: 0.0506 s/iter total_throughput: 1899.55 samples/s lr: 8.28e-04 [09/19 19:33:45] lb.utils.events INFO: eta: 1 day, 17:14:05 iteration: 102699/375342 consumed_samples: 105164800 total_loss: 0.4397 time: 0.5391 s/iter data_time: 0.0516 s/iter total_throughput: 1899.54 samples/s lr: 8.28e-04 [09/19 19:34:39] lb.utils.events INFO: eta: 1 day, 17:11:53 iteration: 102799/375342 consumed_samples: 105267200 total_loss: 0.4375 time: 0.5391 s/iter data_time: 0.0513 s/iter total_throughput: 1899.52 samples/s lr: 8.28e-04 [09/19 19:35:34] lb.utils.events INFO: eta: 1 day, 17:10:28 iteration: 102899/375342 consumed_samples: 105369600 total_loss: 0.4303 time: 0.5391 s/iter data_time: 0.0522 s/iter total_throughput: 1899.50 samples/s lr: 8.27e-04 [09/19 19:36:28] lb.utils.events INFO: eta: 1 day, 17:08:43 iteration: 102999/375342 consumed_samples: 105472000 total_loss: 0.4351 time: 0.5391 s/iter data_time: 0.0528 s/iter total_throughput: 1899.49 samples/s lr: 8.27e-04 [09/19 19:37:22] lb.utils.events INFO: eta: 1 day, 17:06:49 iteration: 103099/375342 consumed_samples: 105574400 total_loss: 0.4369 time: 0.5391 s/iter data_time: 0.0504 s/iter total_throughput: 1899.48 samples/s lr: 8.27e-04 [09/19 19:38:17] lb.utils.events INFO: eta: 1 day, 17:04:58 iteration: 103199/375342 consumed_samples: 105676800 total_loss: 0.432 time: 0.5391 s/iter data_time: 0.0506 s/iter total_throughput: 1899.46 samples/s lr: 8.27e-04 [09/19 19:39:11] lb.utils.events INFO: eta: 1 day, 17:03:48 iteration: 103299/375342 consumed_samples: 105779200 total_loss: 0.4365 time: 0.5391 s/iter data_time: 0.0511 s/iter total_throughput: 1899.44 samples/s lr: 8.26e-04 [09/19 19:40:06] lb.utils.events INFO: eta: 1 day, 17:02:24 iteration: 103399/375342 consumed_samples: 105881600 total_loss: 0.4435 time: 0.5391 s/iter data_time: 0.0497 s/iter total_throughput: 1899.43 samples/s lr: 8.26e-04 [09/19 19:41:00] lb.utils.events INFO: eta: 1 day, 17:01:02 iteration: 103499/375342 consumed_samples: 105984000 total_loss: 0.4428 time: 0.5391 s/iter data_time: 0.0580 s/iter total_throughput: 1899.40 samples/s lr: 8.26e-04 [09/19 19:41:55] lb.utils.events INFO: eta: 1 day, 17:00:47 iteration: 103599/375342 consumed_samples: 106086400 total_loss: 0.4415 time: 0.5391 s/iter data_time: 0.0552 s/iter total_throughput: 1899.38 samples/s lr: 8.25e-04 [09/19 19:42:50] lb.utils.events INFO: eta: 1 day, 17:00:31 iteration: 103699/375342 consumed_samples: 106188800 total_loss: 0.4341 time: 0.5391 s/iter data_time: 0.0533 s/iter total_throughput: 1899.36 samples/s lr: 8.25e-04 [09/19 19:43:44] lb.utils.events INFO: eta: 1 day, 17:00:59 iteration: 103799/375342 consumed_samples: 106291200 total_loss: 0.4358 time: 0.5391 s/iter data_time: 0.0562 s/iter total_throughput: 1899.34 samples/s lr: 8.25e-04 [09/19 19:44:38] lb.utils.events INFO: eta: 1 day, 17:00:25 iteration: 103899/375342 consumed_samples: 106393600 total_loss: 0.4434 time: 0.5391 s/iter data_time: 0.0550 s/iter total_throughput: 1899.32 samples/s lr: 8.24e-04 [09/19 19:45:33] lb.utils.events INFO: eta: 1 day, 17:00:55 iteration: 103999/375342 consumed_samples: 106496000 total_loss: 0.4368 time: 0.5391 s/iter data_time: 0.0565 s/iter total_throughput: 1899.30 samples/s lr: 8.24e-04 [09/19 19:46:28] lb.utils.events INFO: eta: 1 day, 17:01:29 iteration: 104099/375342 consumed_samples: 106598400 total_loss: 0.429 time: 0.5392 s/iter data_time: 0.0552 s/iter total_throughput: 1899.28 samples/s lr: 8.24e-04 [09/19 19:47:22] lb.utils.events INFO: eta: 1 day, 17:01:17 iteration: 104199/375342 consumed_samples: 106700800 total_loss: 0.4321 time: 0.5392 s/iter data_time: 0.0565 s/iter total_throughput: 1899.26 samples/s lr: 8.23e-04 [09/19 19:48:17] lb.utils.events INFO: eta: 1 day, 17:01:16 iteration: 104299/375342 consumed_samples: 106803200 total_loss: 0.4387 time: 0.5392 s/iter data_time: 0.0554 s/iter total_throughput: 1899.24 samples/s lr: 8.23e-04 [09/19 19:49:11] lb.utils.events INFO: eta: 1 day, 17:01:00 iteration: 104399/375342 consumed_samples: 106905600 total_loss: 0.4404 time: 0.5392 s/iter data_time: 0.0559 s/iter total_throughput: 1899.22 samples/s lr: 8.23e-04 [09/19 19:50:06] lb.utils.events INFO: eta: 1 day, 17:01:19 iteration: 104499/375342 consumed_samples: 107008000 total_loss: 0.4395 time: 0.5392 s/iter data_time: 0.0557 s/iter total_throughput: 1899.20 samples/s lr: 8.22e-04 [09/19 19:51:00] lb.utils.events INFO: eta: 1 day, 17:00:28 iteration: 104599/375342 consumed_samples: 107110400 total_loss: 0.4361 time: 0.5392 s/iter data_time: 0.0563 s/iter total_throughput: 1899.18 samples/s lr: 8.22e-04 [09/19 19:51:55] lb.utils.events INFO: eta: 1 day, 16:59:54 iteration: 104699/375342 consumed_samples: 107212800 total_loss: 0.4405 time: 0.5392 s/iter data_time: 0.0535 s/iter total_throughput: 1899.16 samples/s lr: 8.22e-04 [09/19 19:52:49] lb.utils.events INFO: eta: 1 day, 16:58:51 iteration: 104799/375342 consumed_samples: 107315200 total_loss: 0.4424 time: 0.5392 s/iter data_time: 0.0506 s/iter total_throughput: 1899.14 samples/s lr: 8.21e-04 [09/19 19:53:44] lb.utils.events INFO: eta: 1 day, 16:58:00 iteration: 104899/375342 consumed_samples: 107417600 total_loss: 0.4388 time: 0.5392 s/iter data_time: 0.0510 s/iter total_throughput: 1899.12 samples/s lr: 8.21e-04 [09/19 19:54:38] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0104999 [09/19 19:54:39] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 19:54:39] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 19:54:44] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0470 s/iter. Inference: 0.2494 s/iter. Eval: 0.0022 s/iter. Total: 0.2987 s/iter. ETA=0:00:11 [09/19 19:54:49] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0594 s/iter. Inference: 0.2688 s/iter. Eval: 0.0027 s/iter. Total: 0.3311 s/iter. ETA=0:00:07 [09/19 19:54:54] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0612 s/iter. Inference: 0.2621 s/iter. Eval: 0.0025 s/iter. Total: 0.3261 s/iter. ETA=0:00:01 [09/19 19:54:56] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 19:54:56] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.208367 (0.000284 s / iter per device, on 8 devices) [09/19 19:54:56] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000228 s / iter per device, on 8 devices) [09/19 19:54:56] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 19:54:56] lb.evaluation.utils INFO: copypaste: Acc@1=71.574 [09/19 19:54:56] lb.evaluation.utils INFO: copypaste: Acc@5=90.77 [09/19 19:54:56] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 71.57400, better than last best score 71.45400 @ iteration 99999. [09/19 19:54:56] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 19:54:57] lb.utils.events INFO: eta: 1 day, 16:57:15 iteration: 104999/375342 consumed_samples: 107520000 total_loss: 0.4379 time: 0.5392 s/iter data_time: 0.0495 s/iter total_throughput: 1899.10 samples/s lr: 8.21e-04 [09/19 19:55:51] lb.utils.events INFO: eta: 1 day, 16:55:56 iteration: 105099/375342 consumed_samples: 107622400 total_loss: 0.4352 time: 0.5392 s/iter data_time: 0.0526 s/iter total_throughput: 1899.08 samples/s lr: 8.21e-04 [09/19 19:56:46] lb.utils.events INFO: eta: 1 day, 16:54:20 iteration: 105199/375342 consumed_samples: 107724800 total_loss: 0.4298 time: 0.5392 s/iter data_time: 0.0484 s/iter total_throughput: 1899.06 samples/s lr: 8.20e-04 [09/19 19:57:40] lb.utils.events INFO: eta: 1 day, 16:52:22 iteration: 105299/375342 consumed_samples: 107827200 total_loss: 0.4379 time: 0.5392 s/iter data_time: 0.0511 s/iter total_throughput: 1899.05 samples/s lr: 8.20e-04 [09/19 19:58:35] lb.utils.events INFO: eta: 1 day, 16:50:48 iteration: 105399/375342 consumed_samples: 107929600 total_loss: 0.4474 time: 0.5392 s/iter data_time: 0.0520 s/iter total_throughput: 1899.03 samples/s lr: 8.20e-04 [09/19 19:59:29] lb.utils.events INFO: eta: 1 day, 16:47:33 iteration: 105499/375342 consumed_samples: 108032000 total_loss: 0.4429 time: 0.5392 s/iter data_time: 0.0498 s/iter total_throughput: 1899.02 samples/s lr: 8.19e-04 [09/19 20:00:23] lb.utils.events INFO: eta: 1 day, 16:46:07 iteration: 105599/375342 consumed_samples: 108134400 total_loss: 0.436 time: 0.5392 s/iter data_time: 0.0491 s/iter total_throughput: 1899.00 samples/s lr: 8.19e-04 [09/19 20:01:18] lb.utils.events INFO: eta: 1 day, 16:44:17 iteration: 105699/375342 consumed_samples: 108236800 total_loss: 0.4351 time: 0.5392 s/iter data_time: 0.0491 s/iter total_throughput: 1898.99 samples/s lr: 8.19e-04 [09/19 20:02:12] lb.utils.events INFO: eta: 1 day, 16:43:51 iteration: 105799/375342 consumed_samples: 108339200 total_loss: 0.4413 time: 0.5392 s/iter data_time: 0.0505 s/iter total_throughput: 1898.97 samples/s lr: 8.18e-04 [09/19 20:03:07] lb.utils.events INFO: eta: 1 day, 16:42:20 iteration: 105899/375342 consumed_samples: 108441600 total_loss: 0.4408 time: 0.5392 s/iter data_time: 0.0504 s/iter total_throughput: 1898.96 samples/s lr: 8.18e-04 [09/19 20:04:01] lb.utils.events INFO: eta: 1 day, 16:40:54 iteration: 105999/375342 consumed_samples: 108544000 total_loss: 0.4415 time: 0.5392 s/iter data_time: 0.0526 s/iter total_throughput: 1898.94 samples/s lr: 8.18e-04 [09/19 20:04:55] lb.utils.events INFO: eta: 1 day, 16:39:01 iteration: 106099/375342 consumed_samples: 108646400 total_loss: 0.4338 time: 0.5393 s/iter data_time: 0.0521 s/iter total_throughput: 1898.93 samples/s lr: 8.17e-04 [09/19 20:05:50] lb.utils.events INFO: eta: 1 day, 16:37:47 iteration: 106199/375342 consumed_samples: 108748800 total_loss: 0.4288 time: 0.5393 s/iter data_time: 0.0522 s/iter total_throughput: 1898.91 samples/s lr: 8.17e-04 [09/19 20:06:44] lb.utils.events INFO: eta: 1 day, 16:36:58 iteration: 106299/375342 consumed_samples: 108851200 total_loss: 0.4388 time: 0.5393 s/iter data_time: 0.0514 s/iter total_throughput: 1898.90 samples/s lr: 8.17e-04 [09/19 20:07:38] lb.utils.events INFO: eta: 1 day, 16:35:48 iteration: 106399/375342 consumed_samples: 108953600 total_loss: 0.4345 time: 0.5393 s/iter data_time: 0.0520 s/iter total_throughput: 1898.89 samples/s lr: 8.16e-04 [09/19 20:08:33] lb.utils.events INFO: eta: 1 day, 16:34:24 iteration: 106499/375342 consumed_samples: 109056000 total_loss: 0.4306 time: 0.5393 s/iter data_time: 0.0509 s/iter total_throughput: 1898.88 samples/s lr: 8.16e-04 [09/19 20:09:27] lb.utils.events INFO: eta: 1 day, 16:33:30 iteration: 106599/375342 consumed_samples: 109158400 total_loss: 0.4272 time: 0.5393 s/iter data_time: 0.0496 s/iter total_throughput: 1898.86 samples/s lr: 8.16e-04 [09/19 20:10:22] lb.utils.events INFO: eta: 1 day, 16:33:16 iteration: 106699/375342 consumed_samples: 109260800 total_loss: 0.4281 time: 0.5393 s/iter data_time: 0.0493 s/iter total_throughput: 1898.84 samples/s lr: 8.15e-04 [09/19 20:11:16] lb.utils.events INFO: eta: 1 day, 16:32:33 iteration: 106799/375342 consumed_samples: 109363200 total_loss: 0.4326 time: 0.5393 s/iter data_time: 0.0506 s/iter total_throughput: 1898.82 samples/s lr: 8.15e-04 [09/19 20:12:11] lb.utils.events INFO: eta: 1 day, 16:32:03 iteration: 106899/375342 consumed_samples: 109465600 total_loss: 0.4333 time: 0.5393 s/iter data_time: 0.0498 s/iter total_throughput: 1898.80 samples/s lr: 8.15e-04 [09/19 20:13:06] lb.utils.events INFO: eta: 1 day, 16:32:36 iteration: 106999/375342 consumed_samples: 109568000 total_loss: 0.4398 time: 0.5393 s/iter data_time: 0.0546 s/iter total_throughput: 1898.77 samples/s lr: 8.14e-04 [09/19 20:14:00] lb.utils.events INFO: eta: 1 day, 16:33:12 iteration: 107099/375342 consumed_samples: 109670400 total_loss: 0.4385 time: 0.5393 s/iter data_time: 0.0565 s/iter total_throughput: 1898.75 samples/s lr: 8.14e-04 [09/19 20:14:55] lb.utils.events INFO: eta: 1 day, 16:34:12 iteration: 107199/375342 consumed_samples: 109772800 total_loss: 0.4416 time: 0.5393 s/iter data_time: 0.0541 s/iter total_throughput: 1898.72 samples/s lr: 8.14e-04 [09/19 20:15:50] lb.utils.events INFO: eta: 1 day, 16:34:03 iteration: 107299/375342 consumed_samples: 109875200 total_loss: 0.4378 time: 0.5393 s/iter data_time: 0.0554 s/iter total_throughput: 1898.70 samples/s lr: 8.13e-04 [09/19 20:16:44] lb.utils.events INFO: eta: 1 day, 16:34:57 iteration: 107399/375342 consumed_samples: 109977600 total_loss: 0.4284 time: 0.5393 s/iter data_time: 0.0550 s/iter total_throughput: 1898.68 samples/s lr: 8.13e-04 [09/19 20:17:39] lb.utils.events INFO: eta: 1 day, 16:36:48 iteration: 107499/375342 consumed_samples: 110080000 total_loss: 0.4388 time: 0.5393 s/iter data_time: 0.0563 s/iter total_throughput: 1898.65 samples/s lr: 8.13e-04 [09/19 20:18:34] lb.utils.events INFO: eta: 1 day, 16:37:20 iteration: 107599/375342 consumed_samples: 110182400 total_loss: 0.4428 time: 0.5393 s/iter data_time: 0.0561 s/iter total_throughput: 1898.62 samples/s lr: 8.12e-04 [09/19 20:19:29] lb.utils.events INFO: eta: 1 day, 16:37:09 iteration: 107699/375342 consumed_samples: 110284800 total_loss: 0.4357 time: 0.5393 s/iter data_time: 0.0535 s/iter total_throughput: 1898.60 samples/s lr: 8.12e-04 [09/19 20:20:23] lb.utils.events INFO: eta: 1 day, 16:37:05 iteration: 107799/375342 consumed_samples: 110387200 total_loss: 0.4329 time: 0.5394 s/iter data_time: 0.0562 s/iter total_throughput: 1898.58 samples/s lr: 8.12e-04 [09/19 20:21:18] lb.utils.events INFO: eta: 1 day, 16:36:53 iteration: 107899/375342 consumed_samples: 110489600 total_loss: 0.4285 time: 0.5394 s/iter data_time: 0.0546 s/iter total_throughput: 1898.56 samples/s lr: 8.11e-04 [09/19 20:22:12] lb.utils.events INFO: eta: 1 day, 16:35:31 iteration: 107999/375342 consumed_samples: 110592000 total_loss: 0.428 time: 0.5394 s/iter data_time: 0.0561 s/iter total_throughput: 1898.54 samples/s lr: 8.11e-04 [09/19 20:23:07] lb.utils.events INFO: eta: 1 day, 16:34:34 iteration: 108099/375342 consumed_samples: 110694400 total_loss: 0.4283 time: 0.5394 s/iter data_time: 0.0564 s/iter total_throughput: 1898.51 samples/s lr: 8.11e-04 [09/19 20:24:02] lb.utils.events INFO: eta: 1 day, 16:33:47 iteration: 108199/375342 consumed_samples: 110796800 total_loss: 0.4369 time: 0.5394 s/iter data_time: 0.0539 s/iter total_throughput: 1898.49 samples/s lr: 8.11e-04 [09/19 20:24:56] lb.utils.events INFO: eta: 1 day, 16:32:47 iteration: 108299/375342 consumed_samples: 110899200 total_loss: 0.4428 time: 0.5394 s/iter data_time: 0.0499 s/iter total_throughput: 1898.47 samples/s lr: 8.10e-04 [09/19 20:25:51] lb.utils.events INFO: eta: 1 day, 16:31:03 iteration: 108399/375342 consumed_samples: 111001600 total_loss: 0.4351 time: 0.5394 s/iter data_time: 0.0520 s/iter total_throughput: 1898.45 samples/s lr: 8.10e-04 [09/19 20:26:46] lb.utils.events INFO: eta: 1 day, 16:29:58 iteration: 108499/375342 consumed_samples: 111104000 total_loss: 0.4349 time: 0.5394 s/iter data_time: 0.0507 s/iter total_throughput: 1898.42 samples/s lr: 8.10e-04 [09/19 20:27:40] lb.utils.events INFO: eta: 1 day, 16:28:28 iteration: 108599/375342 consumed_samples: 111206400 total_loss: 0.4403 time: 0.5394 s/iter data_time: 0.0515 s/iter total_throughput: 1898.40 samples/s lr: 8.09e-04 [09/19 20:28:35] lb.utils.events INFO: eta: 1 day, 16:26:32 iteration: 108699/375342 consumed_samples: 111308800 total_loss: 0.4421 time: 0.5394 s/iter data_time: 0.0507 s/iter total_throughput: 1898.38 samples/s lr: 8.09e-04 [09/19 20:29:29] lb.utils.events INFO: eta: 1 day, 16:25:06 iteration: 108799/375342 consumed_samples: 111411200 total_loss: 0.4419 time: 0.5394 s/iter data_time: 0.0506 s/iter total_throughput: 1898.36 samples/s lr: 8.09e-04 [09/19 20:30:24] lb.utils.events INFO: eta: 1 day, 16:23:03 iteration: 108899/375342 consumed_samples: 111513600 total_loss: 0.442 time: 0.5394 s/iter data_time: 0.0492 s/iter total_throughput: 1898.35 samples/s lr: 8.08e-04 [09/19 20:31:19] lb.utils.events INFO: eta: 1 day, 16:22:41 iteration: 108999/375342 consumed_samples: 111616000 total_loss: 0.4423 time: 0.5394 s/iter data_time: 0.0505 s/iter total_throughput: 1898.32 samples/s lr: 8.08e-04 [09/19 20:32:13] lb.utils.events INFO: eta: 1 day, 16:21:34 iteration: 109099/375342 consumed_samples: 111718400 total_loss: 0.4366 time: 0.5394 s/iter data_time: 0.0515 s/iter total_throughput: 1898.31 samples/s lr: 8.08e-04 [09/19 20:33:08] lb.utils.events INFO: eta: 1 day, 16:19:10 iteration: 109199/375342 consumed_samples: 111820800 total_loss: 0.4335 time: 0.5394 s/iter data_time: 0.0509 s/iter total_throughput: 1898.29 samples/s lr: 8.07e-04 [09/19 20:34:02] lb.utils.events INFO: eta: 1 day, 16:17:22 iteration: 109299/375342 consumed_samples: 111923200 total_loss: 0.4402 time: 0.5394 s/iter data_time: 0.0525 s/iter total_throughput: 1898.27 samples/s lr: 8.07e-04 [09/19 20:34:57] lb.utils.events INFO: eta: 1 day, 16:16:36 iteration: 109399/375342 consumed_samples: 112025600 total_loss: 0.4403 time: 0.5394 s/iter data_time: 0.0533 s/iter total_throughput: 1898.25 samples/s lr: 8.07e-04 [09/19 20:35:51] lb.utils.events INFO: eta: 1 day, 16:15:05 iteration: 109499/375342 consumed_samples: 112128000 total_loss: 0.4388 time: 0.5394 s/iter data_time: 0.0523 s/iter total_throughput: 1898.23 samples/s lr: 8.06e-04 [09/19 20:36:46] lb.utils.events INFO: eta: 1 day, 16:13:37 iteration: 109599/375342 consumed_samples: 112230400 total_loss: 0.4345 time: 0.5395 s/iter data_time: 0.0515 s/iter total_throughput: 1898.22 samples/s lr: 8.06e-04 [09/19 20:37:40] lb.utils.events INFO: eta: 1 day, 16:12:33 iteration: 109699/375342 consumed_samples: 112332800 total_loss: 0.4291 time: 0.5395 s/iter data_time: 0.0511 s/iter total_throughput: 1898.20 samples/s lr: 8.06e-04 [09/19 20:38:35] lb.utils.events INFO: eta: 1 day, 16:11:13 iteration: 109799/375342 consumed_samples: 112435200 total_loss: 0.4302 time: 0.5395 s/iter data_time: 0.0519 s/iter total_throughput: 1898.19 samples/s lr: 8.05e-04 [09/19 20:39:29] lb.utils.events INFO: eta: 1 day, 16:10:18 iteration: 109899/375342 consumed_samples: 112537600 total_loss: 0.4275 time: 0.5395 s/iter data_time: 0.0520 s/iter total_throughput: 1898.17 samples/s lr: 8.05e-04 [09/19 20:40:23] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0109999 [09/19 20:40:24] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 20:40:24] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 20:40:29] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0571 s/iter. Inference: 0.2509 s/iter. Eval: 0.0023 s/iter. Total: 0.3103 s/iter. ETA=0:00:11 [09/19 20:40:34] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0691 s/iter. Inference: 0.2622 s/iter. Eval: 0.0025 s/iter. Total: 0.3340 s/iter. ETA=0:00:07 [09/19 20:40:39] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0701 s/iter. Inference: 0.2561 s/iter. Eval: 0.0025 s/iter. Total: 0.3290 s/iter. ETA=0:00:01 [09/19 20:40:41] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 20:40:41] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.230827 (0.000285 s / iter per device, on 8 devices) [09/19 20:40:41] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000225 s / iter per device, on 8 devices) [09/19 20:40:41] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 20:40:41] lb.evaluation.utils INFO: copypaste: Acc@1=71.97 [09/19 20:40:41] lb.evaluation.utils INFO: copypaste: Acc@5=90.878 [09/19 20:40:41] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 71.97000, better than last best score 71.57400 @ iteration 104999. [09/19 20:40:41] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 20:40:42] lb.utils.events INFO: eta: 1 day, 16:08:21 iteration: 109999/375342 consumed_samples: 112640000 total_loss: 0.4235 time: 0.5395 s/iter data_time: 0.0521 s/iter total_throughput: 1898.16 samples/s lr: 8.05e-04 [09/19 20:41:36] lb.utils.events INFO: eta: 1 day, 16:07:01 iteration: 110099/375342 consumed_samples: 112742400 total_loss: 0.4321 time: 0.5395 s/iter data_time: 0.0496 s/iter total_throughput: 1898.14 samples/s lr: 8.04e-04 [09/19 20:42:31] lb.utils.events INFO: eta: 1 day, 16:05:51 iteration: 110199/375342 consumed_samples: 112844800 total_loss: 0.4263 time: 0.5395 s/iter data_time: 0.0495 s/iter total_throughput: 1898.13 samples/s lr: 8.04e-04 [09/19 20:43:25] lb.utils.events INFO: eta: 1 day, 16:04:33 iteration: 110299/375342 consumed_samples: 112947200 total_loss: 0.4288 time: 0.5395 s/iter data_time: 0.0499 s/iter total_throughput: 1898.11 samples/s lr: 8.04e-04 [09/19 20:44:20] lb.utils.events INFO: eta: 1 day, 16:03:00 iteration: 110399/375342 consumed_samples: 113049600 total_loss: 0.4355 time: 0.5395 s/iter data_time: 0.0575 s/iter total_throughput: 1898.08 samples/s lr: 8.03e-04 [09/19 20:45:14] lb.utils.events INFO: eta: 1 day, 16:02:01 iteration: 110499/375342 consumed_samples: 113152000 total_loss: 0.436 time: 0.5395 s/iter data_time: 0.0563 s/iter total_throughput: 1898.07 samples/s lr: 8.03e-04 [09/19 20:46:09] lb.utils.events INFO: eta: 1 day, 16:01:22 iteration: 110599/375342 consumed_samples: 113254400 total_loss: 0.4377 time: 0.5395 s/iter data_time: 0.0537 s/iter total_throughput: 1898.05 samples/s lr: 8.03e-04 [09/19 20:47:04] lb.utils.events INFO: eta: 1 day, 16:01:21 iteration: 110699/375342 consumed_samples: 113356800 total_loss: 0.4384 time: 0.5395 s/iter data_time: 0.0538 s/iter total_throughput: 1898.03 samples/s lr: 8.02e-04 [09/19 20:47:58] lb.utils.events INFO: eta: 1 day, 16:00:58 iteration: 110799/375342 consumed_samples: 113459200 total_loss: 0.433 time: 0.5395 s/iter data_time: 0.0554 s/iter total_throughput: 1898.01 samples/s lr: 8.02e-04 [09/19 20:48:53] lb.utils.events INFO: eta: 1 day, 16:01:00 iteration: 110899/375342 consumed_samples: 113561600 total_loss: 0.4313 time: 0.5395 s/iter data_time: 0.0562 s/iter total_throughput: 1897.99 samples/s lr: 8.02e-04 [09/19 20:49:47] lb.utils.events INFO: eta: 1 day, 16:01:30 iteration: 110999/375342 consumed_samples: 113664000 total_loss: 0.4315 time: 0.5395 s/iter data_time: 0.0572 s/iter total_throughput: 1897.97 samples/s lr: 8.01e-04 [09/19 20:50:42] lb.utils.events INFO: eta: 1 day, 16:02:20 iteration: 111099/375342 consumed_samples: 113766400 total_loss: 0.4363 time: 0.5395 s/iter data_time: 0.0565 s/iter total_throughput: 1897.94 samples/s lr: 8.01e-04 [09/19 20:51:37] lb.utils.events INFO: eta: 1 day, 16:02:13 iteration: 111199/375342 consumed_samples: 113868800 total_loss: 0.4408 time: 0.5395 s/iter data_time: 0.0556 s/iter total_throughput: 1897.92 samples/s lr: 8.01e-04 [09/19 20:52:32] lb.utils.events INFO: eta: 1 day, 16:01:51 iteration: 111299/375342 consumed_samples: 113971200 total_loss: 0.4357 time: 0.5395 s/iter data_time: 0.0558 s/iter total_throughput: 1897.90 samples/s lr: 8.00e-04 [09/19 20:53:26] lb.utils.events INFO: eta: 1 day, 16:02:32 iteration: 111399/375342 consumed_samples: 114073600 total_loss: 0.4359 time: 0.5395 s/iter data_time: 0.0553 s/iter total_throughput: 1897.88 samples/s lr: 8.00e-04 [09/19 20:54:21] lb.utils.events INFO: eta: 1 day, 16:01:46 iteration: 111499/375342 consumed_samples: 114176000 total_loss: 0.4364 time: 0.5396 s/iter data_time: 0.0554 s/iter total_throughput: 1897.86 samples/s lr: 8.00e-04 [09/19 20:55:15] lb.utils.events INFO: eta: 1 day, 16:01:22 iteration: 111599/375342 consumed_samples: 114278400 total_loss: 0.4391 time: 0.5396 s/iter data_time: 0.0547 s/iter total_throughput: 1897.84 samples/s lr: 7.99e-04 [09/19 20:56:10] lb.utils.events INFO: eta: 1 day, 16:00:46 iteration: 111699/375342 consumed_samples: 114380800 total_loss: 0.4495 time: 0.5396 s/iter data_time: 0.0521 s/iter total_throughput: 1897.82 samples/s lr: 7.99e-04 [09/19 20:57:04] lb.utils.events INFO: eta: 1 day, 15:58:55 iteration: 111799/375342 consumed_samples: 114483200 total_loss: 0.4426 time: 0.5396 s/iter data_time: 0.0502 s/iter total_throughput: 1897.80 samples/s lr: 7.99e-04 [09/19 20:57:59] lb.utils.events INFO: eta: 1 day, 15:58:14 iteration: 111899/375342 consumed_samples: 114585600 total_loss: 0.4384 time: 0.5396 s/iter data_time: 0.0515 s/iter total_throughput: 1897.78 samples/s lr: 7.98e-04 [09/19 20:58:54] lb.utils.events INFO: eta: 1 day, 15:57:02 iteration: 111999/375342 consumed_samples: 114688000 total_loss: 0.4367 time: 0.5396 s/iter data_time: 0.0502 s/iter total_throughput: 1897.76 samples/s lr: 7.98e-04 [09/19 20:59:48] lb.utils.events INFO: eta: 1 day, 15:55:06 iteration: 112099/375342 consumed_samples: 114790400 total_loss: 0.4394 time: 0.5396 s/iter data_time: 0.0496 s/iter total_throughput: 1897.74 samples/s lr: 7.98e-04 [09/19 21:00:43] lb.utils.events INFO: eta: 1 day, 15:53:28 iteration: 112199/375342 consumed_samples: 114892800 total_loss: 0.4443 time: 0.5396 s/iter data_time: 0.0496 s/iter total_throughput: 1897.72 samples/s lr: 7.97e-04 [09/19 21:01:37] lb.utils.events INFO: eta: 1 day, 15:51:44 iteration: 112299/375342 consumed_samples: 114995200 total_loss: 0.4345 time: 0.5396 s/iter data_time: 0.0526 s/iter total_throughput: 1897.71 samples/s lr: 7.97e-04 [09/19 21:02:32] lb.utils.events INFO: eta: 1 day, 15:50:26 iteration: 112399/375342 consumed_samples: 115097600 total_loss: 0.4327 time: 0.5396 s/iter data_time: 0.0519 s/iter total_throughput: 1897.69 samples/s lr: 7.97e-04 [09/19 21:03:26] lb.utils.events INFO: eta: 1 day, 15:49:19 iteration: 112499/375342 consumed_samples: 115200000 total_loss: 0.4417 time: 0.5396 s/iter data_time: 0.0497 s/iter total_throughput: 1897.67 samples/s lr: 7.96e-04 [09/19 21:04:21] lb.utils.events INFO: eta: 1 day, 15:48:11 iteration: 112599/375342 consumed_samples: 115302400 total_loss: 0.4352 time: 0.5396 s/iter data_time: 0.0518 s/iter total_throughput: 1897.65 samples/s lr: 7.96e-04 [09/19 21:05:16] lb.utils.events INFO: eta: 1 day, 15:47:08 iteration: 112699/375342 consumed_samples: 115404800 total_loss: 0.4327 time: 0.5396 s/iter data_time: 0.0513 s/iter total_throughput: 1897.64 samples/s lr: 7.96e-04 [09/19 21:06:10] lb.utils.events INFO: eta: 1 day, 15:45:36 iteration: 112799/375342 consumed_samples: 115507200 total_loss: 0.4405 time: 0.5396 s/iter data_time: 0.0519 s/iter total_throughput: 1897.63 samples/s lr: 7.95e-04 [09/19 21:07:04] lb.utils.events INFO: eta: 1 day, 15:44:32 iteration: 112899/375342 consumed_samples: 115609600 total_loss: 0.4338 time: 0.5396 s/iter data_time: 0.0522 s/iter total_throughput: 1897.61 samples/s lr: 7.95e-04 [09/19 21:07:59] lb.utils.events INFO: eta: 1 day, 15:42:45 iteration: 112999/375342 consumed_samples: 115712000 total_loss: 0.4323 time: 0.5396 s/iter data_time: 0.0528 s/iter total_throughput: 1897.59 samples/s lr: 7.95e-04 [09/19 21:08:53] lb.utils.events INFO: eta: 1 day, 15:41:04 iteration: 113099/375342 consumed_samples: 115814400 total_loss: 0.4303 time: 0.5396 s/iter data_time: 0.0514 s/iter total_throughput: 1897.58 samples/s lr: 7.94e-04 [09/19 21:09:48] lb.utils.events INFO: eta: 1 day, 15:38:54 iteration: 113199/375342 consumed_samples: 115916800 total_loss: 0.4271 time: 0.5396 s/iter data_time: 0.0526 s/iter total_throughput: 1897.57 samples/s lr: 7.94e-04 [09/19 21:10:42] lb.utils.events INFO: eta: 1 day, 15:37:46 iteration: 113299/375342 consumed_samples: 116019200 total_loss: 0.4277 time: 0.5396 s/iter data_time: 0.0515 s/iter total_throughput: 1897.55 samples/s lr: 7.94e-04 [09/19 21:11:37] lb.utils.events INFO: eta: 1 day, 15:36:29 iteration: 113399/375342 consumed_samples: 116121600 total_loss: 0.4346 time: 0.5396 s/iter data_time: 0.0516 s/iter total_throughput: 1897.54 samples/s lr: 7.93e-04 [09/19 21:12:31] lb.utils.events INFO: eta: 1 day, 15:35:27 iteration: 113499/375342 consumed_samples: 116224000 total_loss: 0.4333 time: 0.5396 s/iter data_time: 0.0487 s/iter total_throughput: 1897.53 samples/s lr: 7.93e-04 [09/19 21:13:25] lb.utils.events INFO: eta: 1 day, 15:33:49 iteration: 113599/375342 consumed_samples: 116326400 total_loss: 0.4353 time: 0.5397 s/iter data_time: 0.0492 s/iter total_throughput: 1897.51 samples/s lr: 7.93e-04 [09/19 21:14:20] lb.utils.events INFO: eta: 1 day, 15:32:57 iteration: 113699/375342 consumed_samples: 116428800 total_loss: 0.4392 time: 0.5397 s/iter data_time: 0.0492 s/iter total_throughput: 1897.50 samples/s lr: 7.92e-04 [09/19 21:15:14] lb.utils.events INFO: eta: 1 day, 15:32:15 iteration: 113799/375342 consumed_samples: 116531200 total_loss: 0.4316 time: 0.5397 s/iter data_time: 0.0483 s/iter total_throughput: 1897.48 samples/s lr: 7.92e-04 [09/19 21:16:09] lb.utils.events INFO: eta: 1 day, 15:31:24 iteration: 113899/375342 consumed_samples: 116633600 total_loss: 0.4225 time: 0.5397 s/iter data_time: 0.0557 s/iter total_throughput: 1897.45 samples/s lr: 7.92e-04 [09/19 21:17:04] lb.utils.events INFO: eta: 1 day, 15:30:26 iteration: 113999/375342 consumed_samples: 116736000 total_loss: 0.4431 time: 0.5397 s/iter data_time: 0.0551 s/iter total_throughput: 1897.44 samples/s lr: 7.91e-04 [09/19 21:17:58] lb.utils.events INFO: eta: 1 day, 15:30:03 iteration: 114099/375342 consumed_samples: 116838400 total_loss: 0.4386 time: 0.5397 s/iter data_time: 0.0555 s/iter total_throughput: 1897.42 samples/s lr: 7.91e-04 [09/19 21:18:53] lb.utils.events INFO: eta: 1 day, 15:30:24 iteration: 114199/375342 consumed_samples: 116940800 total_loss: 0.4348 time: 0.5397 s/iter data_time: 0.0542 s/iter total_throughput: 1897.40 samples/s lr: 7.91e-04 [09/19 21:19:47] lb.utils.events INFO: eta: 1 day, 15:30:39 iteration: 114299/375342 consumed_samples: 117043200 total_loss: 0.4325 time: 0.5397 s/iter data_time: 0.0533 s/iter total_throughput: 1897.39 samples/s lr: 7.90e-04 [09/19 21:20:42] lb.utils.events INFO: eta: 1 day, 15:31:10 iteration: 114399/375342 consumed_samples: 117145600 total_loss: 0.4319 time: 0.5397 s/iter data_time: 0.0548 s/iter total_throughput: 1897.37 samples/s lr: 7.90e-04 [09/19 21:21:37] lb.utils.events INFO: eta: 1 day, 15:31:54 iteration: 114499/375342 consumed_samples: 117248000 total_loss: 0.4321 time: 0.5397 s/iter data_time: 0.0567 s/iter total_throughput: 1897.35 samples/s lr: 7.90e-04 [09/19 21:22:31] lb.utils.events INFO: eta: 1 day, 15:31:26 iteration: 114599/375342 consumed_samples: 117350400 total_loss: 0.437 time: 0.5397 s/iter data_time: 0.0558 s/iter total_throughput: 1897.33 samples/s lr: 7.89e-04 [09/19 21:23:26] lb.utils.events INFO: eta: 1 day, 15:30:49 iteration: 114699/375342 consumed_samples: 117452800 total_loss: 0.4399 time: 0.5397 s/iter data_time: 0.0548 s/iter total_throughput: 1897.31 samples/s lr: 7.89e-04 [09/19 21:24:21] lb.utils.events INFO: eta: 1 day, 15:30:51 iteration: 114799/375342 consumed_samples: 117555200 total_loss: 0.432 time: 0.5397 s/iter data_time: 0.0544 s/iter total_throughput: 1897.29 samples/s lr: 7.89e-04 [09/19 21:25:15] lb.utils.events INFO: eta: 1 day, 15:30:16 iteration: 114899/375342 consumed_samples: 117657600 total_loss: 0.4346 time: 0.5397 s/iter data_time: 0.0566 s/iter total_throughput: 1897.27 samples/s lr: 7.88e-04 [09/19 21:26:10] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0114999 [09/19 21:26:10] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 21:26:10] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 21:26:15] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0705 s/iter. Inference: 0.2483 s/iter. Eval: 0.0021 s/iter. Total: 0.3210 s/iter. ETA=0:00:11 [09/19 21:26:20] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0744 s/iter. Inference: 0.2583 s/iter. Eval: 0.0023 s/iter. Total: 0.3353 s/iter. ETA=0:00:07 [09/19 21:26:25] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0716 s/iter. Inference: 0.2553 s/iter. Eval: 0.0023 s/iter. Total: 0.3296 s/iter. ETA=0:00:01 [09/19 21:26:28] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 21:26:28] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.310499 (0.000286 s / iter per device, on 8 devices) [09/19 21:26:28] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/19 21:26:28] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 21:26:28] lb.evaluation.utils INFO: copypaste: Acc@1=71.666 [09/19 21:26:28] lb.evaluation.utils INFO: copypaste: Acc@5=90.594 [09/19 21:26:28] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 71.66600, not better than best score 71.97000 @ iteration 109999. [09/19 21:26:28] lb.utils.events INFO: eta: 1 day, 15:29:07 iteration: 114999/375342 consumed_samples: 117760000 total_loss: 0.4368 time: 0.5397 s/iter data_time: 0.0555 s/iter total_throughput: 1897.25 samples/s lr: 7.88e-04 [09/19 21:27:22] lb.utils.events INFO: eta: 1 day, 15:28:20 iteration: 115099/375342 consumed_samples: 117862400 total_loss: 0.4359 time: 0.5397 s/iter data_time: 0.0524 s/iter total_throughput: 1897.24 samples/s lr: 7.88e-04 [09/19 21:28:17] lb.utils.events INFO: eta: 1 day, 15:27:23 iteration: 115199/375342 consumed_samples: 117964800 total_loss: 0.4388 time: 0.5397 s/iter data_time: 0.0502 s/iter total_throughput: 1897.22 samples/s lr: 7.87e-04 [09/19 21:29:11] lb.utils.events INFO: eta: 1 day, 15:26:37 iteration: 115299/375342 consumed_samples: 118067200 total_loss: 0.4461 time: 0.5397 s/iter data_time: 0.0512 s/iter total_throughput: 1897.20 samples/s lr: 7.87e-04 [09/19 21:30:06] lb.utils.events INFO: eta: 1 day, 15:25:10 iteration: 115399/375342 consumed_samples: 118169600 total_loss: 0.4425 time: 0.5397 s/iter data_time: 0.0492 s/iter total_throughput: 1897.18 samples/s lr: 7.87e-04 [09/19 21:31:00] lb.utils.events INFO: eta: 1 day, 15:23:05 iteration: 115499/375342 consumed_samples: 118272000 total_loss: 0.4389 time: 0.5398 s/iter data_time: 0.0497 s/iter total_throughput: 1897.17 samples/s lr: 7.86e-04 [09/19 21:31:55] lb.utils.events INFO: eta: 1 day, 15:21:46 iteration: 115599/375342 consumed_samples: 118374400 total_loss: 0.4378 time: 0.5398 s/iter data_time: 0.0513 s/iter total_throughput: 1897.15 samples/s lr: 7.86e-04 [09/19 21:32:49] lb.utils.events INFO: eta: 1 day, 15:20:27 iteration: 115699/375342 consumed_samples: 118476800 total_loss: 0.4352 time: 0.5398 s/iter data_time: 0.0522 s/iter total_throughput: 1897.13 samples/s lr: 7.85e-04 [09/19 21:33:44] lb.utils.events INFO: eta: 1 day, 15:18:47 iteration: 115799/375342 consumed_samples: 118579200 total_loss: 0.4253 time: 0.5398 s/iter data_time: 0.0494 s/iter total_throughput: 1897.12 samples/s lr: 7.85e-04 [09/19 21:34:38] lb.utils.events INFO: eta: 1 day, 15:17:29 iteration: 115899/375342 consumed_samples: 118681600 total_loss: 0.441 time: 0.5398 s/iter data_time: 0.0517 s/iter total_throughput: 1897.10 samples/s lr: 7.85e-04 [09/19 21:35:33] lb.utils.events INFO: eta: 1 day, 15:16:29 iteration: 115999/375342 consumed_samples: 118784000 total_loss: 0.4437 time: 0.5398 s/iter data_time: 0.0507 s/iter total_throughput: 1897.09 samples/s lr: 7.84e-04 [09/19 21:36:28] lb.utils.events INFO: eta: 1 day, 15:15:12 iteration: 116099/375342 consumed_samples: 118886400 total_loss: 0.4289 time: 0.5398 s/iter data_time: 0.0499 s/iter total_throughput: 1897.07 samples/s lr: 7.84e-04 [09/19 21:37:22] lb.utils.events INFO: eta: 1 day, 15:14:03 iteration: 116199/375342 consumed_samples: 118988800 total_loss: 0.4368 time: 0.5398 s/iter data_time: 0.0522 s/iter total_throughput: 1897.05 samples/s lr: 7.84e-04 [09/19 21:38:17] lb.utils.events INFO: eta: 1 day, 15:12:18 iteration: 116299/375342 consumed_samples: 119091200 total_loss: 0.4373 time: 0.5398 s/iter data_time: 0.0516 s/iter total_throughput: 1897.04 samples/s lr: 7.83e-04 [09/19 21:39:11] lb.utils.events INFO: eta: 1 day, 15:10:57 iteration: 116399/375342 consumed_samples: 119193600 total_loss: 0.4373 time: 0.5398 s/iter data_time: 0.0535 s/iter total_throughput: 1897.02 samples/s lr: 7.83e-04 [09/19 21:40:05] lb.utils.events INFO: eta: 1 day, 15:08:56 iteration: 116499/375342 consumed_samples: 119296000 total_loss: 0.4405 time: 0.5398 s/iter data_time: 0.0515 s/iter total_throughput: 1897.01 samples/s lr: 7.83e-04 [09/19 21:41:00] lb.utils.events INFO: eta: 1 day, 15:07:23 iteration: 116599/375342 consumed_samples: 119398400 total_loss: 0.4372 time: 0.5398 s/iter data_time: 0.0507 s/iter total_throughput: 1897.00 samples/s lr: 7.82e-04 [09/19 21:41:54] lb.utils.events INFO: eta: 1 day, 15:05:56 iteration: 116699/375342 consumed_samples: 119500800 total_loss: 0.4368 time: 0.5398 s/iter data_time: 0.0521 s/iter total_throughput: 1896.99 samples/s lr: 7.82e-04 [09/19 21:42:49] lb.utils.events INFO: eta: 1 day, 15:04:13 iteration: 116799/375342 consumed_samples: 119603200 total_loss: 0.4365 time: 0.5398 s/iter data_time: 0.0530 s/iter total_throughput: 1896.98 samples/s lr: 7.82e-04 [09/19 21:43:43] lb.utils.events INFO: eta: 1 day, 15:02:07 iteration: 116899/375342 consumed_samples: 119705600 total_loss: 0.4308 time: 0.5398 s/iter data_time: 0.0516 s/iter total_throughput: 1896.97 samples/s lr: 7.81e-04 [09/19 21:44:37] lb.utils.events INFO: eta: 1 day, 15:00:26 iteration: 116999/375342 consumed_samples: 119808000 total_loss: 0.4226 time: 0.5398 s/iter data_time: 0.0489 s/iter total_throughput: 1896.96 samples/s lr: 7.81e-04 [09/19 21:45:32] lb.utils.events INFO: eta: 1 day, 14:59:30 iteration: 117099/375342 consumed_samples: 119910400 total_loss: 0.4281 time: 0.5398 s/iter data_time: 0.0491 s/iter total_throughput: 1896.94 samples/s lr: 7.81e-04 [09/19 21:46:26] lb.utils.events INFO: eta: 1 day, 14:57:34 iteration: 117199/375342 consumed_samples: 120012800 total_loss: 0.4353 time: 0.5398 s/iter data_time: 0.0503 s/iter total_throughput: 1896.93 samples/s lr: 7.80e-04 [09/19 21:47:21] lb.utils.events INFO: eta: 1 day, 14:57:41 iteration: 117299/375342 consumed_samples: 120115200 total_loss: 0.4346 time: 0.5398 s/iter data_time: 0.0512 s/iter total_throughput: 1896.90 samples/s lr: 7.80e-04 [09/19 21:48:16] lb.utils.events INFO: eta: 1 day, 14:56:23 iteration: 117399/375342 consumed_samples: 120217600 total_loss: 0.4354 time: 0.5398 s/iter data_time: 0.0547 s/iter total_throughput: 1896.89 samples/s lr: 7.80e-04 [09/19 21:49:10] lb.utils.events INFO: eta: 1 day, 14:55:56 iteration: 117499/375342 consumed_samples: 120320000 total_loss: 0.437 time: 0.5398 s/iter data_time: 0.0539 s/iter total_throughput: 1896.88 samples/s lr: 7.79e-04 [09/19 21:50:04] lb.utils.events INFO: eta: 1 day, 14:55:59 iteration: 117599/375342 consumed_samples: 120422400 total_loss: 0.4371 time: 0.5398 s/iter data_time: 0.0529 s/iter total_throughput: 1896.86 samples/s lr: 7.79e-04 [09/19 21:50:59] lb.utils.events INFO: eta: 1 day, 14:55:04 iteration: 117699/375342 consumed_samples: 120524800 total_loss: 0.4321 time: 0.5398 s/iter data_time: 0.0548 s/iter total_throughput: 1896.85 samples/s lr: 7.79e-04 [09/19 21:51:53] lb.utils.events INFO: eta: 1 day, 14:55:34 iteration: 117799/375342 consumed_samples: 120627200 total_loss: 0.4338 time: 0.5398 s/iter data_time: 0.0565 s/iter total_throughput: 1896.83 samples/s lr: 7.78e-04 [09/19 21:52:48] lb.utils.events INFO: eta: 1 day, 14:56:11 iteration: 117899/375342 consumed_samples: 120729600 total_loss: 0.4364 time: 0.5399 s/iter data_time: 0.0569 s/iter total_throughput: 1896.81 samples/s lr: 7.78e-04 [09/19 21:53:43] lb.utils.events INFO: eta: 1 day, 14:56:05 iteration: 117999/375342 consumed_samples: 120832000 total_loss: 0.4374 time: 0.5399 s/iter data_time: 0.0550 s/iter total_throughput: 1896.79 samples/s lr: 7.78e-04 [09/19 21:54:37] lb.utils.events INFO: eta: 1 day, 14:56:11 iteration: 118099/375342 consumed_samples: 120934400 total_loss: 0.4427 time: 0.5399 s/iter data_time: 0.0535 s/iter total_throughput: 1896.78 samples/s lr: 7.77e-04 [09/19 21:55:32] lb.utils.events INFO: eta: 1 day, 14:56:28 iteration: 118199/375342 consumed_samples: 121036800 total_loss: 0.4331 time: 0.5399 s/iter data_time: 0.0557 s/iter total_throughput: 1896.76 samples/s lr: 7.77e-04 [09/19 21:56:27] lb.utils.events INFO: eta: 1 day, 14:55:00 iteration: 118299/375342 consumed_samples: 121139200 total_loss: 0.4374 time: 0.5399 s/iter data_time: 0.0534 s/iter total_throughput: 1896.74 samples/s lr: 7.77e-04 [09/19 21:57:21] lb.utils.events INFO: eta: 1 day, 14:54:56 iteration: 118399/375342 consumed_samples: 121241600 total_loss: 0.434 time: 0.5399 s/iter data_time: 0.0558 s/iter total_throughput: 1896.73 samples/s lr: 7.76e-04 [09/19 21:58:15] lb.utils.events INFO: eta: 1 day, 14:53:52 iteration: 118499/375342 consumed_samples: 121344000 total_loss: 0.4368 time: 0.5399 s/iter data_time: 0.0518 s/iter total_throughput: 1896.72 samples/s lr: 7.76e-04 [09/19 21:59:10] lb.utils.events INFO: eta: 1 day, 14:52:59 iteration: 118599/375342 consumed_samples: 121446400 total_loss: 0.4443 time: 0.5399 s/iter data_time: 0.0493 s/iter total_throughput: 1896.70 samples/s lr: 7.75e-04 [09/19 22:00:04] lb.utils.events INFO: eta: 1 day, 14:52:03 iteration: 118699/375342 consumed_samples: 121548800 total_loss: 0.4308 time: 0.5399 s/iter data_time: 0.0490 s/iter total_throughput: 1896.69 samples/s lr: 7.75e-04 [09/19 22:00:59] lb.utils.events INFO: eta: 1 day, 14:50:59 iteration: 118799/375342 consumed_samples: 121651200 total_loss: 0.4298 time: 0.5399 s/iter data_time: 0.0498 s/iter total_throughput: 1896.67 samples/s lr: 7.75e-04 [09/19 22:01:54] lb.utils.events INFO: eta: 1 day, 14:49:19 iteration: 118899/375342 consumed_samples: 121753600 total_loss: 0.4382 time: 0.5399 s/iter data_time: 0.0503 s/iter total_throughput: 1896.66 samples/s lr: 7.74e-04 [09/19 22:02:48] lb.utils.events INFO: eta: 1 day, 14:47:59 iteration: 118999/375342 consumed_samples: 121856000 total_loss: 0.4401 time: 0.5399 s/iter data_time: 0.0499 s/iter total_throughput: 1896.64 samples/s lr: 7.74e-04 [09/19 22:03:42] lb.utils.events INFO: eta: 1 day, 14:46:21 iteration: 119099/375342 consumed_samples: 121958400 total_loss: 0.4309 time: 0.5399 s/iter data_time: 0.0491 s/iter total_throughput: 1896.63 samples/s lr: 7.74e-04 [09/19 22:04:37] lb.utils.events INFO: eta: 1 day, 14:44:26 iteration: 119199/375342 consumed_samples: 122060800 total_loss: 0.4244 time: 0.5399 s/iter data_time: 0.0502 s/iter total_throughput: 1896.62 samples/s lr: 7.73e-04 [09/19 22:05:31] lb.utils.events INFO: eta: 1 day, 14:43:35 iteration: 119299/375342 consumed_samples: 122163200 total_loss: 0.4324 time: 0.5399 s/iter data_time: 0.0482 s/iter total_throughput: 1896.61 samples/s lr: 7.73e-04 [09/19 22:06:26] lb.utils.events INFO: eta: 1 day, 14:42:40 iteration: 119399/375342 consumed_samples: 122265600 total_loss: 0.4346 time: 0.5399 s/iter data_time: 0.0519 s/iter total_throughput: 1896.59 samples/s lr: 7.73e-04 [09/19 22:07:20] lb.utils.events INFO: eta: 1 day, 14:41:39 iteration: 119499/375342 consumed_samples: 122368000 total_loss: 0.4268 time: 0.5399 s/iter data_time: 0.0491 s/iter total_throughput: 1896.58 samples/s lr: 7.72e-04 [09/19 22:08:15] lb.utils.events INFO: eta: 1 day, 14:39:36 iteration: 119599/375342 consumed_samples: 122470400 total_loss: 0.4295 time: 0.5399 s/iter data_time: 0.0503 s/iter total_throughput: 1896.57 samples/s lr: 7.72e-04 [09/19 22:09:09] lb.utils.events INFO: eta: 1 day, 14:38:14 iteration: 119699/375342 consumed_samples: 122572800 total_loss: 0.4374 time: 0.5399 s/iter data_time: 0.0509 s/iter total_throughput: 1896.56 samples/s lr: 7.72e-04 [09/19 22:10:03] lb.utils.events INFO: eta: 1 day, 14:36:54 iteration: 119799/375342 consumed_samples: 122675200 total_loss: 0.4379 time: 0.5399 s/iter data_time: 0.0504 s/iter total_throughput: 1896.55 samples/s lr: 7.71e-04 [09/19 22:10:58] lb.utils.events INFO: eta: 1 day, 14:35:15 iteration: 119899/375342 consumed_samples: 122777600 total_loss: 0.4389 time: 0.5399 s/iter data_time: 0.0510 s/iter total_throughput: 1896.53 samples/s lr: 7.71e-04 [09/19 22:11:52] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0119999 [09/19 22:11:53] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 22:11:53] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 22:11:58] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0676 s/iter. Inference: 0.2443 s/iter. Eval: 0.0029 s/iter. Total: 0.3148 s/iter. ETA=0:00:11 [09/19 22:12:03] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0818 s/iter. Inference: 0.2533 s/iter. Eval: 0.0027 s/iter. Total: 0.3380 s/iter. ETA=0:00:07 [09/19 22:12:08] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0769 s/iter. Inference: 0.2531 s/iter. Eval: 0.0026 s/iter. Total: 0.3328 s/iter. ETA=0:00:01 [09/19 22:12:10] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 22:12:10] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.303092 (0.000286 s / iter per device, on 8 devices) [09/19 22:12:10] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000221 s / iter per device, on 8 devices) [09/19 22:12:10] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 22:12:10] lb.evaluation.utils INFO: copypaste: Acc@1=72.18599999999999 [09/19 22:12:10] lb.evaluation.utils INFO: copypaste: Acc@5=90.9 [09/19 22:12:10] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 72.18600, better than last best score 71.97000 @ iteration 109999. [09/19 22:12:10] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 22:12:11] lb.utils.events INFO: eta: 1 day, 14:34:13 iteration: 119999/375342 consumed_samples: 122880000 total_loss: 0.4409 time: 0.5399 s/iter data_time: 0.0524 s/iter total_throughput: 1896.52 samples/s lr: 7.71e-04 [09/19 22:13:05] lb.utils.events INFO: eta: 1 day, 14:33:42 iteration: 120099/375342 consumed_samples: 122982400 total_loss: 0.443 time: 0.5399 s/iter data_time: 0.0516 s/iter total_throughput: 1896.51 samples/s lr: 7.70e-04 [09/19 22:14:00] lb.utils.events INFO: eta: 1 day, 14:32:12 iteration: 120199/375342 consumed_samples: 123084800 total_loss: 0.4366 time: 0.5399 s/iter data_time: 0.0523 s/iter total_throughput: 1896.50 samples/s lr: 7.70e-04 [09/19 22:14:54] lb.utils.events INFO: eta: 1 day, 14:30:50 iteration: 120299/375342 consumed_samples: 123187200 total_loss: 0.4313 time: 0.5399 s/iter data_time: 0.0522 s/iter total_throughput: 1896.49 samples/s lr: 7.70e-04 [09/19 22:15:48] lb.utils.events INFO: eta: 1 day, 14:29:45 iteration: 120399/375342 consumed_samples: 123289600 total_loss: 0.427 time: 0.5400 s/iter data_time: 0.0518 s/iter total_throughput: 1896.47 samples/s lr: 7.69e-04 [09/19 22:16:43] lb.utils.events INFO: eta: 1 day, 14:29:13 iteration: 120499/375342 consumed_samples: 123392000 total_loss: 0.4381 time: 0.5400 s/iter data_time: 0.0493 s/iter total_throughput: 1896.45 samples/s lr: 7.69e-04 [09/19 22:17:38] lb.utils.events INFO: eta: 1 day, 14:30:17 iteration: 120599/375342 consumed_samples: 123494400 total_loss: 0.4417 time: 0.5400 s/iter data_time: 0.0518 s/iter total_throughput: 1896.43 samples/s lr: 7.69e-04 [09/19 22:18:32] lb.utils.events INFO: eta: 1 day, 14:30:19 iteration: 120699/375342 consumed_samples: 123596800 total_loss: 0.439 time: 0.5400 s/iter data_time: 0.0520 s/iter total_throughput: 1896.41 samples/s lr: 7.68e-04 [09/19 22:19:28] lb.utils.events INFO: eta: 1 day, 14:30:48 iteration: 120799/375342 consumed_samples: 123699200 total_loss: 0.4299 time: 0.5400 s/iter data_time: 0.0560 s/iter total_throughput: 1896.38 samples/s lr: 7.68e-04 [09/19 22:20:22] lb.utils.events INFO: eta: 1 day, 14:30:13 iteration: 120899/375342 consumed_samples: 123801600 total_loss: 0.4311 time: 0.5400 s/iter data_time: 0.0566 s/iter total_throughput: 1896.36 samples/s lr: 7.67e-04 [09/19 22:21:17] lb.utils.events INFO: eta: 1 day, 14:30:19 iteration: 120999/375342 consumed_samples: 123904000 total_loss: 0.4351 time: 0.5400 s/iter data_time: 0.0547 s/iter total_throughput: 1896.35 samples/s lr: 7.67e-04 [09/19 22:22:11] lb.utils.events INFO: eta: 1 day, 14:29:59 iteration: 121099/375342 consumed_samples: 124006400 total_loss: 0.4373 time: 0.5400 s/iter data_time: 0.0542 s/iter total_throughput: 1896.33 samples/s lr: 7.67e-04 [09/19 22:23:06] lb.utils.events INFO: eta: 1 day, 14:30:11 iteration: 121199/375342 consumed_samples: 124108800 total_loss: 0.4311 time: 0.5400 s/iter data_time: 0.0571 s/iter total_throughput: 1896.32 samples/s lr: 7.66e-04 [09/19 22:24:01] lb.utils.events INFO: eta: 1 day, 14:30:05 iteration: 121299/375342 consumed_samples: 124211200 total_loss: 0.4406 time: 0.5400 s/iter data_time: 0.0554 s/iter total_throughput: 1896.30 samples/s lr: 7.66e-04 [09/19 22:24:55] lb.utils.events INFO: eta: 1 day, 14:29:54 iteration: 121399/375342 consumed_samples: 124313600 total_loss: 0.4401 time: 0.5400 s/iter data_time: 0.0543 s/iter total_throughput: 1896.28 samples/s lr: 7.66e-04 [09/19 22:25:50] lb.utils.events INFO: eta: 1 day, 14:29:44 iteration: 121499/375342 consumed_samples: 124416000 total_loss: 0.435 time: 0.5400 s/iter data_time: 0.0545 s/iter total_throughput: 1896.26 samples/s lr: 7.65e-04 [09/19 22:26:45] lb.utils.events INFO: eta: 1 day, 14:28:49 iteration: 121599/375342 consumed_samples: 124518400 total_loss: 0.4326 time: 0.5400 s/iter data_time: 0.0547 s/iter total_throughput: 1896.24 samples/s lr: 7.65e-04 [09/19 22:27:39] lb.utils.events INFO: eta: 1 day, 14:27:57 iteration: 121699/375342 consumed_samples: 124620800 total_loss: 0.4369 time: 0.5400 s/iter data_time: 0.0546 s/iter total_throughput: 1896.23 samples/s lr: 7.65e-04 [09/19 22:28:34] lb.utils.events INFO: eta: 1 day, 14:25:29 iteration: 121799/375342 consumed_samples: 124723200 total_loss: 0.4429 time: 0.5400 s/iter data_time: 0.0560 s/iter total_throughput: 1896.21 samples/s lr: 7.64e-04 [09/19 22:29:28] lb.utils.events INFO: eta: 1 day, 14:24:35 iteration: 121899/375342 consumed_samples: 124825600 total_loss: 0.4382 time: 0.5400 s/iter data_time: 0.0548 s/iter total_throughput: 1896.20 samples/s lr: 7.64e-04 [09/19 22:30:23] lb.utils.events INFO: eta: 1 day, 14:23:18 iteration: 121999/375342 consumed_samples: 124928000 total_loss: 0.4326 time: 0.5400 s/iter data_time: 0.0527 s/iter total_throughput: 1896.19 samples/s lr: 7.64e-04 [09/19 22:31:17] lb.utils.events INFO: eta: 1 day, 14:22:51 iteration: 122099/375342 consumed_samples: 125030400 total_loss: 0.4369 time: 0.5400 s/iter data_time: 0.0503 s/iter total_throughput: 1896.17 samples/s lr: 7.63e-04 [09/19 22:32:12] lb.utils.events INFO: eta: 1 day, 14:21:44 iteration: 122199/375342 consumed_samples: 125132800 total_loss: 0.4399 time: 0.5400 s/iter data_time: 0.0497 s/iter total_throughput: 1896.16 samples/s lr: 7.63e-04 [09/19 22:33:06] lb.utils.events INFO: eta: 1 day, 14:20:24 iteration: 122299/375342 consumed_samples: 125235200 total_loss: 0.4334 time: 0.5400 s/iter data_time: 0.0499 s/iter total_throughput: 1896.14 samples/s lr: 7.63e-04 [09/19 22:34:01] lb.utils.events INFO: eta: 1 day, 14:19:13 iteration: 122399/375342 consumed_samples: 125337600 total_loss: 0.4373 time: 0.5400 s/iter data_time: 0.0519 s/iter total_throughput: 1896.13 samples/s lr: 7.62e-04 [09/19 22:34:55] lb.utils.events INFO: eta: 1 day, 14:18:23 iteration: 122499/375342 consumed_samples: 125440000 total_loss: 0.4391 time: 0.5401 s/iter data_time: 0.0503 s/iter total_throughput: 1896.11 samples/s lr: 7.62e-04 [09/19 22:35:50] lb.utils.events INFO: eta: 1 day, 14:16:55 iteration: 122599/375342 consumed_samples: 125542400 total_loss: 0.4336 time: 0.5401 s/iter data_time: 0.0495 s/iter total_throughput: 1896.10 samples/s lr: 7.61e-04 [09/19 22:36:44] lb.utils.events INFO: eta: 1 day, 14:15:50 iteration: 122699/375342 consumed_samples: 125644800 total_loss: 0.4358 time: 0.5401 s/iter data_time: 0.0483 s/iter total_throughput: 1896.09 samples/s lr: 7.61e-04 [09/19 22:37:39] lb.utils.events INFO: eta: 1 day, 14:14:57 iteration: 122799/375342 consumed_samples: 125747200 total_loss: 0.4377 time: 0.5401 s/iter data_time: 0.0501 s/iter total_throughput: 1896.07 samples/s lr: 7.61e-04 [09/19 22:38:33] lb.utils.events INFO: eta: 1 day, 14:13:52 iteration: 122899/375342 consumed_samples: 125849600 total_loss: 0.4424 time: 0.5401 s/iter data_time: 0.0488 s/iter total_throughput: 1896.06 samples/s lr: 7.60e-04 [09/19 22:39:28] lb.utils.events INFO: eta: 1 day, 14:13:03 iteration: 122999/375342 consumed_samples: 125952000 total_loss: 0.4424 time: 0.5401 s/iter data_time: 0.0508 s/iter total_throughput: 1896.05 samples/s lr: 7.60e-04 [09/19 22:40:22] lb.utils.events INFO: eta: 1 day, 14:10:49 iteration: 123099/375342 consumed_samples: 126054400 total_loss: 0.4375 time: 0.5401 s/iter data_time: 0.0508 s/iter total_throughput: 1896.03 samples/s lr: 7.60e-04 [09/19 22:41:17] lb.utils.events INFO: eta: 1 day, 14:10:21 iteration: 123199/375342 consumed_samples: 126156800 total_loss: 0.4345 time: 0.5401 s/iter data_time: 0.0509 s/iter total_throughput: 1896.02 samples/s lr: 7.59e-04 [09/19 22:42:11] lb.utils.events INFO: eta: 1 day, 14:08:57 iteration: 123299/375342 consumed_samples: 126259200 total_loss: 0.4324 time: 0.5401 s/iter data_time: 0.0520 s/iter total_throughput: 1896.01 samples/s lr: 7.59e-04 [09/19 22:43:06] lb.utils.events INFO: eta: 1 day, 14:06:08 iteration: 123399/375342 consumed_samples: 126361600 total_loss: 0.4325 time: 0.5401 s/iter data_time: 0.0502 s/iter total_throughput: 1896.00 samples/s lr: 7.59e-04 [09/19 22:44:00] lb.utils.events INFO: eta: 1 day, 14:04:18 iteration: 123499/375342 consumed_samples: 126464000 total_loss: 0.4376 time: 0.5401 s/iter data_time: 0.0512 s/iter total_throughput: 1895.99 samples/s lr: 7.58e-04 [09/19 22:44:54] lb.utils.events INFO: eta: 1 day, 14:03:06 iteration: 123599/375342 consumed_samples: 126566400 total_loss: 0.442 time: 0.5401 s/iter data_time: 0.0515 s/iter total_throughput: 1895.98 samples/s lr: 7.58e-04 [09/19 22:45:49] lb.utils.events INFO: eta: 1 day, 14:01:20 iteration: 123699/375342 consumed_samples: 126668800 total_loss: 0.4389 time: 0.5401 s/iter data_time: 0.0523 s/iter total_throughput: 1895.97 samples/s lr: 7.58e-04 [09/19 22:46:43] lb.utils.events INFO: eta: 1 day, 13:59:25 iteration: 123799/375342 consumed_samples: 126771200 total_loss: 0.4308 time: 0.5401 s/iter data_time: 0.0522 s/iter total_throughput: 1895.97 samples/s lr: 7.57e-04 [09/19 22:47:37] lb.utils.events INFO: eta: 1 day, 13:58:28 iteration: 123899/375342 consumed_samples: 126873600 total_loss: 0.4297 time: 0.5401 s/iter data_time: 0.0487 s/iter total_throughput: 1895.95 samples/s lr: 7.57e-04 [09/19 22:48:32] lb.utils.events INFO: eta: 1 day, 13:57:22 iteration: 123999/375342 consumed_samples: 126976000 total_loss: 0.4362 time: 0.5401 s/iter data_time: 0.0496 s/iter total_throughput: 1895.94 samples/s lr: 7.56e-04 [09/19 22:49:26] lb.utils.events INFO: eta: 1 day, 13:56:18 iteration: 124099/375342 consumed_samples: 127078400 total_loss: 0.4398 time: 0.5401 s/iter data_time: 0.0498 s/iter total_throughput: 1895.93 samples/s lr: 7.56e-04 [09/19 22:50:21] lb.utils.events INFO: eta: 1 day, 13:54:55 iteration: 124199/375342 consumed_samples: 127180800 total_loss: 0.4351 time: 0.5401 s/iter data_time: 0.0547 s/iter total_throughput: 1895.91 samples/s lr: 7.56e-04 [09/19 22:51:15] lb.utils.events INFO: eta: 1 day, 13:54:39 iteration: 124299/375342 consumed_samples: 127283200 total_loss: 0.4303 time: 0.5401 s/iter data_time: 0.0539 s/iter total_throughput: 1895.90 samples/s lr: 7.55e-04 [09/19 22:52:10] lb.utils.events INFO: eta: 1 day, 13:54:41 iteration: 124399/375342 consumed_samples: 127385600 total_loss: 0.427 time: 0.5401 s/iter data_time: 0.0544 s/iter total_throughput: 1895.89 samples/s lr: 7.55e-04 [09/19 22:53:04] lb.utils.events INFO: eta: 1 day, 13:54:13 iteration: 124499/375342 consumed_samples: 127488000 total_loss: 0.4378 time: 0.5401 s/iter data_time: 0.0530 s/iter total_throughput: 1895.87 samples/s lr: 7.55e-04 [09/19 22:53:59] lb.utils.events INFO: eta: 1 day, 13:53:54 iteration: 124599/375342 consumed_samples: 127590400 total_loss: 0.4381 time: 0.5401 s/iter data_time: 0.0559 s/iter total_throughput: 1895.86 samples/s lr: 7.54e-04 [09/19 22:54:53] lb.utils.events INFO: eta: 1 day, 13:54:26 iteration: 124699/375342 consumed_samples: 127692800 total_loss: 0.4305 time: 0.5401 s/iter data_time: 0.0547 s/iter total_throughput: 1895.85 samples/s lr: 7.54e-04 [09/19 22:55:48] lb.utils.events INFO: eta: 1 day, 13:55:00 iteration: 124799/375342 consumed_samples: 127795200 total_loss: 0.4308 time: 0.5401 s/iter data_time: 0.0555 s/iter total_throughput: 1895.83 samples/s lr: 7.54e-04 [09/19 22:56:43] lb.utils.events INFO: eta: 1 day, 13:54:58 iteration: 124899/375342 consumed_samples: 127897600 total_loss: 0.4364 time: 0.5401 s/iter data_time: 0.0560 s/iter total_throughput: 1895.81 samples/s lr: 7.53e-04 [09/19 22:57:37] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0124999 [09/19 22:57:38] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 22:57:38] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 22:57:43] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0520 s/iter. Inference: 0.2537 s/iter. Eval: 0.0020 s/iter. Total: 0.3078 s/iter. ETA=0:00:11 [09/19 22:57:48] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0654 s/iter. Inference: 0.2677 s/iter. Eval: 0.0022 s/iter. Total: 0.3356 s/iter. ETA=0:00:07 [09/19 22:57:53] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0662 s/iter. Inference: 0.2606 s/iter. Eval: 0.0022 s/iter. Total: 0.3294 s/iter. ETA=0:00:01 [09/19 22:57:55] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 22:57:55] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.247068 (0.000285 s / iter per device, on 8 devices) [09/19 22:57:55] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000227 s / iter per device, on 8 devices) [09/19 22:57:55] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 22:57:55] lb.evaluation.utils INFO: copypaste: Acc@1=71.726 [09/19 22:57:55] lb.evaluation.utils INFO: copypaste: Acc@5=90.9 [09/19 22:57:55] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 71.72600, not better than best score 72.18600 @ iteration 119999. [09/19 22:57:55] lb.utils.events INFO: eta: 1 day, 13:54:55 iteration: 124999/375342 consumed_samples: 128000000 total_loss: 0.4359 time: 0.5401 s/iter data_time: 0.0551 s/iter total_throughput: 1895.80 samples/s lr: 7.53e-04 [09/19 22:58:50] lb.utils.events INFO: eta: 1 day, 13:54:34 iteration: 125099/375342 consumed_samples: 128102400 total_loss: 0.436 time: 0.5401 s/iter data_time: 0.0565 s/iter total_throughput: 1895.78 samples/s lr: 7.53e-04 [09/19 22:59:44] lb.utils.events INFO: eta: 1 day, 13:53:51 iteration: 125199/375342 consumed_samples: 128204800 total_loss: 0.4371 time: 0.5402 s/iter data_time: 0.0547 s/iter total_throughput: 1895.77 samples/s lr: 7.52e-04 [09/19 23:00:39] lb.utils.events INFO: eta: 1 day, 13:54:02 iteration: 125299/375342 consumed_samples: 128307200 total_loss: 0.4378 time: 0.5402 s/iter data_time: 0.0546 s/iter total_throughput: 1895.75 samples/s lr: 7.52e-04 [09/19 23:01:33] lb.utils.events INFO: eta: 1 day, 13:52:48 iteration: 125399/375342 consumed_samples: 128409600 total_loss: 0.4375 time: 0.5402 s/iter data_time: 0.0544 s/iter total_throughput: 1895.74 samples/s lr: 7.51e-04 [09/19 23:02:28] lb.utils.events INFO: eta: 1 day, 13:52:50 iteration: 125499/375342 consumed_samples: 128512000 total_loss: 0.4376 time: 0.5402 s/iter data_time: 0.0512 s/iter total_throughput: 1895.72 samples/s lr: 7.51e-04 [09/19 23:03:22] lb.utils.events INFO: eta: 1 day, 13:52:03 iteration: 125599/375342 consumed_samples: 128614400 total_loss: 0.4386 time: 0.5402 s/iter data_time: 0.0490 s/iter total_throughput: 1895.71 samples/s lr: 7.51e-04 [09/19 23:04:17] lb.utils.events INFO: eta: 1 day, 13:51:09 iteration: 125699/375342 consumed_samples: 128716800 total_loss: 0.4389 time: 0.5402 s/iter data_time: 0.0496 s/iter total_throughput: 1895.69 samples/s lr: 7.50e-04 [09/19 23:05:12] lb.utils.events INFO: eta: 1 day, 13:49:11 iteration: 125799/375342 consumed_samples: 128819200 total_loss: 0.4393 time: 0.5402 s/iter data_time: 0.0509 s/iter total_throughput: 1895.68 samples/s lr: 7.50e-04 [09/19 23:06:06] lb.utils.events INFO: eta: 1 day, 13:47:45 iteration: 125899/375342 consumed_samples: 128921600 total_loss: 0.4441 time: 0.5402 s/iter data_time: 0.0506 s/iter total_throughput: 1895.66 samples/s lr: 7.50e-04 [09/19 23:07:01] lb.utils.events INFO: eta: 1 day, 13:46:31 iteration: 125999/375342 consumed_samples: 129024000 total_loss: 0.4434 time: 0.5402 s/iter data_time: 0.0513 s/iter total_throughput: 1895.65 samples/s lr: 7.49e-04 [09/19 23:07:55] lb.utils.events INFO: eta: 1 day, 13:45:36 iteration: 126099/375342 consumed_samples: 129126400 total_loss: 0.4379 time: 0.5402 s/iter data_time: 0.0507 s/iter total_throughput: 1895.63 samples/s lr: 7.49e-04 [09/19 23:08:50] lb.utils.events INFO: eta: 1 day, 13:44:25 iteration: 126199/375342 consumed_samples: 129228800 total_loss: 0.4384 time: 0.5402 s/iter data_time: 0.0494 s/iter total_throughput: 1895.62 samples/s lr: 7.49e-04 [09/19 23:09:44] lb.utils.events INFO: eta: 1 day, 13:42:49 iteration: 126299/375342 consumed_samples: 129331200 total_loss: 0.4378 time: 0.5402 s/iter data_time: 0.0514 s/iter total_throughput: 1895.61 samples/s lr: 7.48e-04 [09/19 23:10:39] lb.utils.events INFO: eta: 1 day, 13:41:58 iteration: 126399/375342 consumed_samples: 129433600 total_loss: 0.4358 time: 0.5402 s/iter data_time: 0.0499 s/iter total_throughput: 1895.59 samples/s lr: 7.48e-04 [09/19 23:11:33] lb.utils.events INFO: eta: 1 day, 13:40:42 iteration: 126499/375342 consumed_samples: 129536000 total_loss: 0.4332 time: 0.5402 s/iter data_time: 0.0516 s/iter total_throughput: 1895.58 samples/s lr: 7.48e-04 [09/19 23:12:28] lb.utils.events INFO: eta: 1 day, 13:39:31 iteration: 126599/375342 consumed_samples: 129638400 total_loss: 0.4331 time: 0.5402 s/iter data_time: 0.0511 s/iter total_throughput: 1895.57 samples/s lr: 7.47e-04 [09/19 23:13:22] lb.utils.events INFO: eta: 1 day, 13:37:46 iteration: 126699/375342 consumed_samples: 129740800 total_loss: 0.4312 time: 0.5402 s/iter data_time: 0.0515 s/iter total_throughput: 1895.56 samples/s lr: 7.47e-04 [09/19 23:14:17] lb.utils.events INFO: eta: 1 day, 13:36:29 iteration: 126799/375342 consumed_samples: 129843200 total_loss: 0.4246 time: 0.5402 s/iter data_time: 0.0506 s/iter total_throughput: 1895.54 samples/s lr: 7.46e-04 [09/19 23:15:11] lb.utils.events INFO: eta: 1 day, 13:35:08 iteration: 126899/375342 consumed_samples: 129945600 total_loss: 0.4281 time: 0.5402 s/iter data_time: 0.0519 s/iter total_throughput: 1895.53 samples/s lr: 7.46e-04 [09/19 23:16:06] lb.utils.events INFO: eta: 1 day, 13:33:41 iteration: 126999/375342 consumed_samples: 130048000 total_loss: 0.4309 time: 0.5402 s/iter data_time: 0.0505 s/iter total_throughput: 1895.53 samples/s lr: 7.46e-04 [09/19 23:17:00] lb.utils.events INFO: eta: 1 day, 13:32:22 iteration: 127099/375342 consumed_samples: 130150400 total_loss: 0.4337 time: 0.5402 s/iter data_time: 0.0508 s/iter total_throughput: 1895.52 samples/s lr: 7.45e-04 [09/19 23:17:54] lb.utils.events INFO: eta: 1 day, 13:31:13 iteration: 127199/375342 consumed_samples: 130252800 total_loss: 0.4405 time: 0.5402 s/iter data_time: 0.0511 s/iter total_throughput: 1895.51 samples/s lr: 7.45e-04 [09/19 23:18:49] lb.utils.events INFO: eta: 1 day, 13:29:57 iteration: 127299/375342 consumed_samples: 130355200 total_loss: 0.4336 time: 0.5402 s/iter data_time: 0.0490 s/iter total_throughput: 1895.50 samples/s lr: 7.45e-04 [09/19 23:19:43] lb.utils.events INFO: eta: 1 day, 13:28:57 iteration: 127399/375342 consumed_samples: 130457600 total_loss: 0.4263 time: 0.5402 s/iter data_time: 0.0474 s/iter total_throughput: 1895.48 samples/s lr: 7.44e-04 [09/19 23:20:38] lb.utils.events INFO: eta: 1 day, 13:27:43 iteration: 127499/375342 consumed_samples: 130560000 total_loss: 0.4252 time: 0.5402 s/iter data_time: 0.0493 s/iter total_throughput: 1895.47 samples/s lr: 7.44e-04 [09/19 23:21:32] lb.utils.events INFO: eta: 1 day, 13:26:18 iteration: 127599/375342 consumed_samples: 130662400 total_loss: 0.4187 time: 0.5402 s/iter data_time: 0.0489 s/iter total_throughput: 1895.46 samples/s lr: 7.44e-04 [09/19 23:22:27] lb.utils.events INFO: eta: 1 day, 13:25:21 iteration: 127699/375342 consumed_samples: 130764800 total_loss: 0.4176 time: 0.5402 s/iter data_time: 0.0516 s/iter total_throughput: 1895.44 samples/s lr: 7.43e-04 [09/19 23:23:22] lb.utils.events INFO: eta: 1 day, 13:25:00 iteration: 127799/375342 consumed_samples: 130867200 total_loss: 0.4271 time: 0.5402 s/iter data_time: 0.0539 s/iter total_throughput: 1895.43 samples/s lr: 7.43e-04 [09/19 23:24:16] lb.utils.events INFO: eta: 1 day, 13:24:28 iteration: 127899/375342 consumed_samples: 130969600 total_loss: 0.4272 time: 0.5403 s/iter data_time: 0.0544 s/iter total_throughput: 1895.41 samples/s lr: 7.42e-04 [09/19 23:25:11] lb.utils.events INFO: eta: 1 day, 13:24:52 iteration: 127999/375342 consumed_samples: 131072000 total_loss: 0.4265 time: 0.5403 s/iter data_time: 0.0571 s/iter total_throughput: 1895.40 samples/s lr: 7.42e-04 [09/19 23:26:05] lb.utils.events INFO: eta: 1 day, 13:25:04 iteration: 128099/375342 consumed_samples: 131174400 total_loss: 0.4311 time: 0.5403 s/iter data_time: 0.0560 s/iter total_throughput: 1895.38 samples/s lr: 7.42e-04 [09/19 23:27:00] lb.utils.events INFO: eta: 1 day, 13:25:35 iteration: 128199/375342 consumed_samples: 131276800 total_loss: 0.4392 time: 0.5403 s/iter data_time: 0.0554 s/iter total_throughput: 1895.36 samples/s lr: 7.41e-04 [09/19 23:27:55] lb.utils.events INFO: eta: 1 day, 13:26:11 iteration: 128299/375342 consumed_samples: 131379200 total_loss: 0.4352 time: 0.5403 s/iter data_time: 0.0545 s/iter total_throughput: 1895.35 samples/s lr: 7.41e-04 [09/19 23:28:49] lb.utils.events INFO: eta: 1 day, 13:25:58 iteration: 128399/375342 consumed_samples: 131481600 total_loss: 0.4308 time: 0.5403 s/iter data_time: 0.0524 s/iter total_throughput: 1895.33 samples/s lr: 7.41e-04 [09/19 23:29:44] lb.utils.events INFO: eta: 1 day, 13:25:38 iteration: 128499/375342 consumed_samples: 131584000 total_loss: 0.4333 time: 0.5403 s/iter data_time: 0.0551 s/iter total_throughput: 1895.31 samples/s lr: 7.40e-04 [09/19 23:30:39] lb.utils.events INFO: eta: 1 day, 13:25:06 iteration: 128599/375342 consumed_samples: 131686400 total_loss: 0.4408 time: 0.5403 s/iter data_time: 0.0549 s/iter total_throughput: 1895.30 samples/s lr: 7.40e-04 [09/19 23:31:33] lb.utils.events INFO: eta: 1 day, 13:24:57 iteration: 128699/375342 consumed_samples: 131788800 total_loss: 0.4336 time: 0.5403 s/iter data_time: 0.0541 s/iter total_throughput: 1895.28 samples/s lr: 7.40e-04 [09/19 23:32:28] lb.utils.events INFO: eta: 1 day, 13:24:10 iteration: 128799/375342 consumed_samples: 131891200 total_loss: 0.4219 time: 0.5403 s/iter data_time: 0.0527 s/iter total_throughput: 1895.27 samples/s lr: 7.39e-04 [09/19 23:33:22] lb.utils.events INFO: eta: 1 day, 13:23:17 iteration: 128899/375342 consumed_samples: 131993600 total_loss: 0.4332 time: 0.5403 s/iter data_time: 0.0510 s/iter total_throughput: 1895.26 samples/s lr: 7.39e-04 [09/19 23:34:17] lb.utils.events INFO: eta: 1 day, 13:22:27 iteration: 128999/375342 consumed_samples: 132096000 total_loss: 0.4383 time: 0.5403 s/iter data_time: 0.0506 s/iter total_throughput: 1895.24 samples/s lr: 7.38e-04 [09/19 23:35:11] lb.utils.events INFO: eta: 1 day, 13:21:24 iteration: 129099/375342 consumed_samples: 132198400 total_loss: 0.433 time: 0.5403 s/iter data_time: 0.0485 s/iter total_throughput: 1895.23 samples/s lr: 7.38e-04 [09/19 23:36:06] lb.utils.events INFO: eta: 1 day, 13:19:28 iteration: 129199/375342 consumed_samples: 132300800 total_loss: 0.433 time: 0.5403 s/iter data_time: 0.0494 s/iter total_throughput: 1895.21 samples/s lr: 7.38e-04 [09/19 23:37:01] lb.utils.events INFO: eta: 1 day, 13:17:55 iteration: 129299/375342 consumed_samples: 132403200 total_loss: 0.4355 time: 0.5403 s/iter data_time: 0.0507 s/iter total_throughput: 1895.20 samples/s lr: 7.37e-04 [09/19 23:37:55] lb.utils.events INFO: eta: 1 day, 13:16:19 iteration: 129399/375342 consumed_samples: 132505600 total_loss: 0.434 time: 0.5403 s/iter data_time: 0.0518 s/iter total_throughput: 1895.19 samples/s lr: 7.37e-04 [09/19 23:38:50] lb.utils.events INFO: eta: 1 day, 13:14:48 iteration: 129499/375342 consumed_samples: 132608000 total_loss: 0.4293 time: 0.5403 s/iter data_time: 0.0500 s/iter total_throughput: 1895.18 samples/s lr: 7.37e-04 [09/19 23:39:44] lb.utils.events INFO: eta: 1 day, 13:13:32 iteration: 129599/375342 consumed_samples: 132710400 total_loss: 0.4349 time: 0.5403 s/iter data_time: 0.0500 s/iter total_throughput: 1895.16 samples/s lr: 7.36e-04 [09/19 23:40:39] lb.utils.events INFO: eta: 1 day, 13:11:53 iteration: 129699/375342 consumed_samples: 132812800 total_loss: 0.441 time: 0.5403 s/iter data_time: 0.0492 s/iter total_throughput: 1895.15 samples/s lr: 7.36e-04 [09/19 23:41:33] lb.utils.events INFO: eta: 1 day, 13:11:05 iteration: 129799/375342 consumed_samples: 132915200 total_loss: 0.4262 time: 0.5403 s/iter data_time: 0.0501 s/iter total_throughput: 1895.14 samples/s lr: 7.36e-04 [09/19 23:42:28] lb.utils.events INFO: eta: 1 day, 13:09:48 iteration: 129899/375342 consumed_samples: 133017600 total_loss: 0.4276 time: 0.5403 s/iter data_time: 0.0517 s/iter total_throughput: 1895.13 samples/s lr: 7.35e-04 [09/19 23:43:22] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0129999 [09/19 23:43:23] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/19 23:43:23] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/19 23:43:27] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0577 s/iter. Inference: 0.2467 s/iter. Eval: 0.0033 s/iter. Total: 0.3078 s/iter. ETA=0:00:11 [09/19 23:43:32] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0617 s/iter. Inference: 0.2660 s/iter. Eval: 0.0025 s/iter. Total: 0.3305 s/iter. ETA=0:00:07 [09/19 23:43:37] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0633 s/iter. Inference: 0.2613 s/iter. Eval: 0.0025 s/iter. Total: 0.3274 s/iter. ETA=0:00:01 [09/19 23:43:40] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/19 23:43:40] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.200768 (0.000284 s / iter per device, on 8 devices) [09/19 23:43:40] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000230 s / iter per device, on 8 devices) [09/19 23:43:40] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/19 23:43:40] lb.evaluation.utils INFO: copypaste: Acc@1=72.548 [09/19 23:43:40] lb.evaluation.utils INFO: copypaste: Acc@5=91.33200000000001 [09/19 23:43:40] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 72.54800, better than last best score 72.18600 @ iteration 119999. [09/19 23:43:40] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/19 23:43:40] lb.utils.events INFO: eta: 1 day, 13:08:08 iteration: 129999/375342 consumed_samples: 133120000 total_loss: 0.4379 time: 0.5403 s/iter data_time: 0.0511 s/iter total_throughput: 1895.12 samples/s lr: 7.35e-04 [09/19 23:44:35] lb.utils.events INFO: eta: 1 day, 13:06:34 iteration: 130099/375342 consumed_samples: 133222400 total_loss: 0.4356 time: 0.5403 s/iter data_time: 0.0499 s/iter total_throughput: 1895.11 samples/s lr: 7.34e-04 [09/19 23:45:29] lb.utils.events INFO: eta: 1 day, 13:05:24 iteration: 130199/375342 consumed_samples: 133324800 total_loss: 0.4308 time: 0.5403 s/iter data_time: 0.0508 s/iter total_throughput: 1895.10 samples/s lr: 7.34e-04 [09/19 23:46:24] lb.utils.events INFO: eta: 1 day, 13:03:42 iteration: 130299/375342 consumed_samples: 133427200 total_loss: 0.4279 time: 0.5403 s/iter data_time: 0.0504 s/iter total_throughput: 1895.09 samples/s lr: 7.34e-04 [09/19 23:47:18] lb.utils.events INFO: eta: 1 day, 13:02:36 iteration: 130399/375342 consumed_samples: 133529600 total_loss: 0.4259 time: 0.5403 s/iter data_time: 0.0522 s/iter total_throughput: 1895.08 samples/s lr: 7.33e-04 [09/19 23:48:12] lb.utils.events INFO: eta: 1 day, 13:01:08 iteration: 130499/375342 consumed_samples: 133632000 total_loss: 0.4338 time: 0.5403 s/iter data_time: 0.0503 s/iter total_throughput: 1895.07 samples/s lr: 7.33e-04 [09/19 23:49:07] lb.utils.events INFO: eta: 1 day, 12:59:27 iteration: 130599/375342 consumed_samples: 133734400 total_loss: 0.4356 time: 0.5404 s/iter data_time: 0.0501 s/iter total_throughput: 1895.06 samples/s lr: 7.33e-04 [09/19 23:50:01] lb.utils.events INFO: eta: 1 day, 12:57:43 iteration: 130699/375342 consumed_samples: 133836800 total_loss: 0.4414 time: 0.5404 s/iter data_time: 0.0514 s/iter total_throughput: 1895.05 samples/s lr: 7.32e-04 [09/19 23:50:55] lb.utils.events INFO: eta: 1 day, 12:56:37 iteration: 130799/375342 consumed_samples: 133939200 total_loss: 0.4407 time: 0.5404 s/iter data_time: 0.0494 s/iter total_throughput: 1895.04 samples/s lr: 7.32e-04 [09/19 23:51:50] lb.utils.events INFO: eta: 1 day, 12:55:34 iteration: 130899/375342 consumed_samples: 134041600 total_loss: 0.4327 time: 0.5404 s/iter data_time: 0.0491 s/iter total_throughput: 1895.03 samples/s lr: 7.31e-04 [09/19 23:52:44] lb.utils.events INFO: eta: 1 day, 12:54:40 iteration: 130999/375342 consumed_samples: 134144000 total_loss: 0.4331 time: 0.5404 s/iter data_time: 0.0482 s/iter total_throughput: 1895.02 samples/s lr: 7.31e-04 [09/19 23:53:39] lb.utils.events INFO: eta: 1 day, 12:53:26 iteration: 131099/375342 consumed_samples: 134246400 total_loss: 0.4286 time: 0.5404 s/iter data_time: 0.0526 s/iter total_throughput: 1895.01 samples/s lr: 7.31e-04 [09/19 23:54:33] lb.utils.events INFO: eta: 1 day, 12:52:33 iteration: 131199/375342 consumed_samples: 134348800 total_loss: 0.4256 time: 0.5404 s/iter data_time: 0.0535 s/iter total_throughput: 1895.00 samples/s lr: 7.30e-04 [09/19 23:55:28] lb.utils.events INFO: eta: 1 day, 12:52:17 iteration: 131299/375342 consumed_samples: 134451200 total_loss: 0.4315 time: 0.5404 s/iter data_time: 0.0549 s/iter total_throughput: 1894.99 samples/s lr: 7.30e-04 [09/19 23:56:22] lb.utils.events INFO: eta: 1 day, 12:51:30 iteration: 131399/375342 consumed_samples: 134553600 total_loss: 0.4345 time: 0.5404 s/iter data_time: 0.0536 s/iter total_throughput: 1894.98 samples/s lr: 7.30e-04 [09/19 23:57:17] lb.utils.events INFO: eta: 1 day, 12:51:03 iteration: 131499/375342 consumed_samples: 134656000 total_loss: 0.4284 time: 0.5404 s/iter data_time: 0.0564 s/iter total_throughput: 1894.96 samples/s lr: 7.29e-04 [09/19 23:58:11] lb.utils.events INFO: eta: 1 day, 12:50:55 iteration: 131599/375342 consumed_samples: 134758400 total_loss: 0.4209 time: 0.5404 s/iter data_time: 0.0556 s/iter total_throughput: 1894.95 samples/s lr: 7.29e-04 [09/19 23:59:06] lb.utils.events INFO: eta: 1 day, 12:51:45 iteration: 131699/375342 consumed_samples: 134860800 total_loss: 0.4283 time: 0.5404 s/iter data_time: 0.0552 s/iter total_throughput: 1894.93 samples/s lr: 7.29e-04 [09/20 00:00:01] lb.utils.events INFO: eta: 1 day, 12:51:08 iteration: 131799/375342 consumed_samples: 134963200 total_loss: 0.4296 time: 0.5404 s/iter data_time: 0.0546 s/iter total_throughput: 1894.92 samples/s lr: 7.28e-04 [09/20 00:00:55] lb.utils.events INFO: eta: 1 day, 12:51:16 iteration: 131899/375342 consumed_samples: 135065600 total_loss: 0.4209 time: 0.5404 s/iter data_time: 0.0543 s/iter total_throughput: 1894.91 samples/s lr: 7.28e-04 [09/20 00:01:50] lb.utils.events INFO: eta: 1 day, 12:50:54 iteration: 131999/375342 consumed_samples: 135168000 total_loss: 0.4296 time: 0.5404 s/iter data_time: 0.0533 s/iter total_throughput: 1894.89 samples/s lr: 7.27e-04 [09/20 00:02:44] lb.utils.events INFO: eta: 1 day, 12:50:08 iteration: 132099/375342 consumed_samples: 135270400 total_loss: 0.4358 time: 0.5404 s/iter data_time: 0.0538 s/iter total_throughput: 1894.88 samples/s lr: 7.27e-04 [09/20 00:03:39] lb.utils.events INFO: eta: 1 day, 12:49:52 iteration: 132199/375342 consumed_samples: 135372800 total_loss: 0.4326 time: 0.5404 s/iter data_time: 0.0533 s/iter total_throughput: 1894.87 samples/s lr: 7.27e-04 [09/20 00:04:33] lb.utils.events INFO: eta: 1 day, 12:49:25 iteration: 132299/375342 consumed_samples: 135475200 total_loss: 0.4327 time: 0.5404 s/iter data_time: 0.0537 s/iter total_throughput: 1894.86 samples/s lr: 7.26e-04 [09/20 00:05:28] lb.utils.events INFO: eta: 1 day, 12:48:51 iteration: 132399/375342 consumed_samples: 135577600 total_loss: 0.4384 time: 0.5404 s/iter data_time: 0.0507 s/iter total_throughput: 1894.84 samples/s lr: 7.26e-04 [09/20 00:06:23] lb.utils.events INFO: eta: 1 day, 12:47:39 iteration: 132499/375342 consumed_samples: 135680000 total_loss: 0.4354 time: 0.5404 s/iter data_time: 0.0501 s/iter total_throughput: 1894.83 samples/s lr: 7.26e-04 [09/20 00:07:17] lb.utils.events INFO: eta: 1 day, 12:46:20 iteration: 132599/375342 consumed_samples: 135782400 total_loss: 0.4334 time: 0.5404 s/iter data_time: 0.0487 s/iter total_throughput: 1894.82 samples/s lr: 7.25e-04 [09/20 00:08:12] lb.utils.events INFO: eta: 1 day, 12:44:43 iteration: 132699/375342 consumed_samples: 135884800 total_loss: 0.4349 time: 0.5404 s/iter data_time: 0.0497 s/iter total_throughput: 1894.81 samples/s lr: 7.25e-04 [09/20 00:09:06] lb.utils.events INFO: eta: 1 day, 12:43:29 iteration: 132799/375342 consumed_samples: 135987200 total_loss: 0.429 time: 0.5404 s/iter data_time: 0.0481 s/iter total_throughput: 1894.80 samples/s lr: 7.24e-04 [09/20 00:10:00] lb.utils.events INFO: eta: 1 day, 12:41:58 iteration: 132899/375342 consumed_samples: 136089600 total_loss: 0.4296 time: 0.5404 s/iter data_time: 0.0486 s/iter total_throughput: 1894.79 samples/s lr: 7.24e-04 [09/20 00:10:55] lb.utils.events INFO: eta: 1 day, 12:40:43 iteration: 132999/375342 consumed_samples: 136192000 total_loss: 0.4389 time: 0.5404 s/iter data_time: 0.0507 s/iter total_throughput: 1894.78 samples/s lr: 7.24e-04 [09/20 00:11:49] lb.utils.events INFO: eta: 1 day, 12:39:35 iteration: 133099/375342 consumed_samples: 136294400 total_loss: 0.4364 time: 0.5404 s/iter data_time: 0.0487 s/iter total_throughput: 1894.77 samples/s lr: 7.23e-04 [09/20 00:12:44] lb.utils.events INFO: eta: 1 day, 12:38:32 iteration: 133199/375342 consumed_samples: 136396800 total_loss: 0.4309 time: 0.5404 s/iter data_time: 0.0500 s/iter total_throughput: 1894.76 samples/s lr: 7.23e-04 [09/20 00:13:38] lb.utils.events INFO: eta: 1 day, 12:37:42 iteration: 133299/375342 consumed_samples: 136499200 total_loss: 0.4254 time: 0.5404 s/iter data_time: 0.0510 s/iter total_throughput: 1894.74 samples/s lr: 7.23e-04 [09/20 00:14:33] lb.utils.events INFO: eta: 1 day, 12:36:43 iteration: 133399/375342 consumed_samples: 136601600 total_loss: 0.4221 time: 0.5404 s/iter data_time: 0.0506 s/iter total_throughput: 1894.73 samples/s lr: 7.22e-04 [09/20 00:15:28] lb.utils.events INFO: eta: 1 day, 12:35:53 iteration: 133499/375342 consumed_samples: 136704000 total_loss: 0.4266 time: 0.5405 s/iter data_time: 0.0523 s/iter total_throughput: 1894.72 samples/s lr: 7.22e-04 [09/20 00:16:22] lb.utils.events INFO: eta: 1 day, 12:35:02 iteration: 133599/375342 consumed_samples: 136806400 total_loss: 0.4352 time: 0.5405 s/iter data_time: 0.0517 s/iter total_throughput: 1894.71 samples/s lr: 7.21e-04 [09/20 00:17:17] lb.utils.events INFO: eta: 1 day, 12:34:09 iteration: 133699/375342 consumed_samples: 136908800 total_loss: 0.4338 time: 0.5405 s/iter data_time: 0.0543 s/iter total_throughput: 1894.69 samples/s lr: 7.21e-04 [09/20 00:18:11] lb.utils.events INFO: eta: 1 day, 12:33:34 iteration: 133799/375342 consumed_samples: 137011200 total_loss: 0.4273 time: 0.5405 s/iter data_time: 0.0513 s/iter total_throughput: 1894.68 samples/s lr: 7.21e-04 [09/20 00:19:06] lb.utils.events INFO: eta: 1 day, 12:33:26 iteration: 133899/375342 consumed_samples: 137113600 total_loss: 0.4241 time: 0.5405 s/iter data_time: 0.0534 s/iter total_throughput: 1894.66 samples/s lr: 7.20e-04 [09/20 00:20:00] lb.utils.events INFO: eta: 1 day, 12:33:25 iteration: 133999/375342 consumed_samples: 137216000 total_loss: 0.4194 time: 0.5405 s/iter data_time: 0.0542 s/iter total_throughput: 1894.65 samples/s lr: 7.20e-04 [09/20 00:20:55] lb.utils.events INFO: eta: 1 day, 12:32:35 iteration: 134099/375342 consumed_samples: 137318400 total_loss: 0.4286 time: 0.5405 s/iter data_time: 0.0525 s/iter total_throughput: 1894.64 samples/s lr: 7.20e-04 [09/20 00:21:49] lb.utils.events INFO: eta: 1 day, 12:31:57 iteration: 134199/375342 consumed_samples: 137420800 total_loss: 0.4283 time: 0.5405 s/iter data_time: 0.0504 s/iter total_throughput: 1894.63 samples/s lr: 7.19e-04 [09/20 00:22:44] lb.utils.events INFO: eta: 1 day, 12:30:46 iteration: 134299/375342 consumed_samples: 137523200 total_loss: 0.4275 time: 0.5405 s/iter data_time: 0.0526 s/iter total_throughput: 1894.61 samples/s lr: 7.19e-04 [09/20 00:23:39] lb.utils.events INFO: eta: 1 day, 12:29:44 iteration: 134399/375342 consumed_samples: 137625600 total_loss: 0.4368 time: 0.5405 s/iter data_time: 0.0495 s/iter total_throughput: 1894.60 samples/s lr: 7.18e-04 [09/20 00:24:33] lb.utils.events INFO: eta: 1 day, 12:28:25 iteration: 134499/375342 consumed_samples: 137728000 total_loss: 0.4361 time: 0.5405 s/iter data_time: 0.0498 s/iter total_throughput: 1894.59 samples/s lr: 7.18e-04 [09/20 00:25:28] lb.utils.events INFO: eta: 1 day, 12:27:23 iteration: 134599/375342 consumed_samples: 137830400 total_loss: 0.4296 time: 0.5405 s/iter data_time: 0.0548 s/iter total_throughput: 1894.57 samples/s lr: 7.18e-04 [09/20 00:26:22] lb.utils.events INFO: eta: 1 day, 12:27:03 iteration: 134699/375342 consumed_samples: 137932800 total_loss: 0.4305 time: 0.5405 s/iter data_time: 0.0540 s/iter total_throughput: 1894.56 samples/s lr: 7.17e-04 [09/20 00:27:17] lb.utils.events INFO: eta: 1 day, 12:26:11 iteration: 134799/375342 consumed_samples: 138035200 total_loss: 0.4338 time: 0.5405 s/iter data_time: 0.0539 s/iter total_throughput: 1894.54 samples/s lr: 7.17e-04 [09/20 00:28:12] lb.utils.events INFO: eta: 1 day, 12:25:19 iteration: 134899/375342 consumed_samples: 138137600 total_loss: 0.4329 time: 0.5405 s/iter data_time: 0.0542 s/iter total_throughput: 1894.53 samples/s lr: 7.17e-04 [09/20 00:29:06] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0134999 [09/20 00:29:07] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 00:29:07] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 00:29:12] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0543 s/iter. Inference: 0.2563 s/iter. Eval: 0.0022 s/iter. Total: 0.3129 s/iter. ETA=0:00:11 [09/20 00:29:17] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0815 s/iter. Inference: 0.2568 s/iter. Eval: 0.0023 s/iter. Total: 0.3408 s/iter. ETA=0:00:07 [09/20 00:29:22] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0764 s/iter. Inference: 0.2534 s/iter. Eval: 0.0022 s/iter. Total: 0.3323 s/iter. ETA=0:00:01 [09/20 00:29:24] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 00:29:24] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.251105 (0.000285 s / iter per device, on 8 devices) [09/20 00:29:24] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000222 s / iter per device, on 8 devices) [09/20 00:29:24] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 00:29:24] lb.evaluation.utils INFO: copypaste: Acc@1=72.684 [09/20 00:29:24] lb.evaluation.utils INFO: copypaste: Acc@5=91.45 [09/20 00:29:24] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 72.68400, better than last best score 72.54800 @ iteration 129999. [09/20 00:29:24] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 00:29:25] lb.utils.events INFO: eta: 1 day, 12:24:45 iteration: 134999/375342 consumed_samples: 138240000 total_loss: 0.4221 time: 0.5405 s/iter data_time: 0.0554 s/iter total_throughput: 1894.51 samples/s lr: 7.16e-04 [09/20 00:30:19] lb.utils.events INFO: eta: 1 day, 12:24:34 iteration: 135099/375342 consumed_samples: 138342400 total_loss: 0.4264 time: 0.5405 s/iter data_time: 0.0547 s/iter total_throughput: 1894.50 samples/s lr: 7.16e-04 [09/20 00:31:14] lb.utils.events INFO: eta: 1 day, 12:24:01 iteration: 135199/375342 consumed_samples: 138444800 total_loss: 0.434 time: 0.5405 s/iter data_time: 0.0575 s/iter total_throughput: 1894.48 samples/s lr: 7.15e-04 [09/20 00:32:09] lb.utils.events INFO: eta: 1 day, 12:23:56 iteration: 135299/375342 consumed_samples: 138547200 total_loss: 0.4294 time: 0.5405 s/iter data_time: 0.0550 s/iter total_throughput: 1894.47 samples/s lr: 7.15e-04 [09/20 00:33:03] lb.utils.events INFO: eta: 1 day, 12:23:41 iteration: 135399/375342 consumed_samples: 138649600 total_loss: 0.421 time: 0.5405 s/iter data_time: 0.0560 s/iter total_throughput: 1894.45 samples/s lr: 7.15e-04 [09/20 00:33:58] lb.utils.events INFO: eta: 1 day, 12:23:01 iteration: 135499/375342 consumed_samples: 138752000 total_loss: 0.4228 time: 0.5405 s/iter data_time: 0.0557 s/iter total_throughput: 1894.44 samples/s lr: 7.14e-04 [09/20 00:34:53] lb.utils.events INFO: eta: 1 day, 12:21:43 iteration: 135599/375342 consumed_samples: 138854400 total_loss: 0.4216 time: 0.5405 s/iter data_time: 0.0540 s/iter total_throughput: 1894.42 samples/s lr: 7.14e-04 [09/20 00:35:47] lb.utils.events INFO: eta: 1 day, 12:20:55 iteration: 135699/375342 consumed_samples: 138956800 total_loss: 0.429 time: 0.5405 s/iter data_time: 0.0552 s/iter total_throughput: 1894.41 samples/s lr: 7.14e-04 [09/20 00:36:42] lb.utils.events INFO: eta: 1 day, 12:20:27 iteration: 135799/375342 consumed_samples: 139059200 total_loss: 0.4333 time: 0.5405 s/iter data_time: 0.0505 s/iter total_throughput: 1894.40 samples/s lr: 7.13e-04 [09/20 00:37:36] lb.utils.events INFO: eta: 1 day, 12:19:24 iteration: 135899/375342 consumed_samples: 139161600 total_loss: 0.4318 time: 0.5405 s/iter data_time: 0.0498 s/iter total_throughput: 1894.39 samples/s lr: 7.13e-04 [09/20 00:38:31] lb.utils.events INFO: eta: 1 day, 12:18:03 iteration: 135999/375342 consumed_samples: 139264000 total_loss: 0.4327 time: 0.5405 s/iter data_time: 0.0499 s/iter total_throughput: 1894.37 samples/s lr: 7.12e-04 [09/20 00:39:26] lb.utils.events INFO: eta: 1 day, 12:16:48 iteration: 136099/375342 consumed_samples: 139366400 total_loss: 0.4289 time: 0.5406 s/iter data_time: 0.0493 s/iter total_throughput: 1894.36 samples/s lr: 7.12e-04 [09/20 00:40:20] lb.utils.events INFO: eta: 1 day, 12:15:46 iteration: 136199/375342 consumed_samples: 139468800 total_loss: 0.4251 time: 0.5406 s/iter data_time: 0.0495 s/iter total_throughput: 1894.34 samples/s lr: 7.12e-04 [09/20 00:41:15] lb.utils.events INFO: eta: 1 day, 12:14:47 iteration: 136299/375342 consumed_samples: 139571200 total_loss: 0.4291 time: 0.5406 s/iter data_time: 0.0517 s/iter total_throughput: 1894.33 samples/s lr: 7.11e-04 [09/20 00:42:09] lb.utils.events INFO: eta: 1 day, 12:13:13 iteration: 136399/375342 consumed_samples: 139673600 total_loss: 0.4333 time: 0.5406 s/iter data_time: 0.0501 s/iter total_throughput: 1894.32 samples/s lr: 7.11e-04 [09/20 00:43:04] lb.utils.events INFO: eta: 1 day, 12:12:28 iteration: 136499/375342 consumed_samples: 139776000 total_loss: 0.4352 time: 0.5406 s/iter data_time: 0.0519 s/iter total_throughput: 1894.31 samples/s lr: 7.11e-04 [09/20 00:43:58] lb.utils.events INFO: eta: 1 day, 12:11:57 iteration: 136599/375342 consumed_samples: 139878400 total_loss: 0.4342 time: 0.5406 s/iter data_time: 0.0520 s/iter total_throughput: 1894.29 samples/s lr: 7.10e-04 [09/20 00:44:53] lb.utils.events INFO: eta: 1 day, 12:10:53 iteration: 136699/375342 consumed_samples: 139980800 total_loss: 0.4282 time: 0.5406 s/iter data_time: 0.0512 s/iter total_throughput: 1894.28 samples/s lr: 7.10e-04 [09/20 00:45:48] lb.utils.events INFO: eta: 1 day, 12:09:41 iteration: 136799/375342 consumed_samples: 140083200 total_loss: 0.4266 time: 0.5406 s/iter data_time: 0.0507 s/iter total_throughput: 1894.27 samples/s lr: 7.09e-04 [09/20 00:46:42] lb.utils.events INFO: eta: 1 day, 12:08:52 iteration: 136899/375342 consumed_samples: 140185600 total_loss: 0.4361 time: 0.5406 s/iter data_time: 0.0529 s/iter total_throughput: 1894.25 samples/s lr: 7.09e-04 [09/20 00:47:37] lb.utils.events INFO: eta: 1 day, 12:07:34 iteration: 136999/375342 consumed_samples: 140288000 total_loss: 0.4302 time: 0.5406 s/iter data_time: 0.0537 s/iter total_throughput: 1894.24 samples/s lr: 7.09e-04 [09/20 00:48:31] lb.utils.events INFO: eta: 1 day, 12:05:58 iteration: 137099/375342 consumed_samples: 140390400 total_loss: 0.4295 time: 0.5406 s/iter data_time: 0.0513 s/iter total_throughput: 1894.23 samples/s lr: 7.08e-04 [09/20 00:49:26] lb.utils.events INFO: eta: 1 day, 12:04:35 iteration: 137199/375342 consumed_samples: 140492800 total_loss: 0.4301 time: 0.5406 s/iter data_time: 0.0508 s/iter total_throughput: 1894.22 samples/s lr: 7.08e-04 [09/20 00:50:20] lb.utils.events INFO: eta: 1 day, 12:02:27 iteration: 137299/375342 consumed_samples: 140595200 total_loss: 0.4263 time: 0.5406 s/iter data_time: 0.0517 s/iter total_throughput: 1894.21 samples/s lr: 7.08e-04 [09/20 00:51:15] lb.utils.events INFO: eta: 1 day, 12:01:21 iteration: 137399/375342 consumed_samples: 140697600 total_loss: 0.4236 time: 0.5406 s/iter data_time: 0.0517 s/iter total_throughput: 1894.20 samples/s lr: 7.07e-04 [09/20 00:52:09] lb.utils.events INFO: eta: 1 day, 11:59:36 iteration: 137499/375342 consumed_samples: 140800000 total_loss: 0.4282 time: 0.5406 s/iter data_time: 0.0517 s/iter total_throughput: 1894.19 samples/s lr: 7.07e-04 [09/20 00:53:03] lb.utils.events INFO: eta: 1 day, 11:58:05 iteration: 137599/375342 consumed_samples: 140902400 total_loss: 0.4322 time: 0.5406 s/iter data_time: 0.0503 s/iter total_throughput: 1894.18 samples/s lr: 7.06e-04 [09/20 00:53:58] lb.utils.events INFO: eta: 1 day, 11:57:12 iteration: 137699/375342 consumed_samples: 141004800 total_loss: 0.4303 time: 0.5406 s/iter data_time: 0.0497 s/iter total_throughput: 1894.17 samples/s lr: 7.06e-04 [09/20 00:54:53] lb.utils.events INFO: eta: 1 day, 11:56:12 iteration: 137799/375342 consumed_samples: 141107200 total_loss: 0.4325 time: 0.5406 s/iter data_time: 0.0491 s/iter total_throughput: 1894.16 samples/s lr: 7.06e-04 [09/20 00:55:47] lb.utils.events INFO: eta: 1 day, 11:55:00 iteration: 137899/375342 consumed_samples: 141209600 total_loss: 0.4301 time: 0.5406 s/iter data_time: 0.0500 s/iter total_throughput: 1894.15 samples/s lr: 7.05e-04 [09/20 00:56:42] lb.utils.events INFO: eta: 1 day, 11:53:59 iteration: 137999/375342 consumed_samples: 141312000 total_loss: 0.4292 time: 0.5406 s/iter data_time: 0.0542 s/iter total_throughput: 1894.13 samples/s lr: 7.05e-04 [09/20 00:57:37] lb.utils.events INFO: eta: 1 day, 11:53:25 iteration: 138099/375342 consumed_samples: 141414400 total_loss: 0.4296 time: 0.5406 s/iter data_time: 0.0531 s/iter total_throughput: 1894.12 samples/s lr: 7.05e-04 [09/20 00:58:31] lb.utils.events INFO: eta: 1 day, 11:52:31 iteration: 138199/375342 consumed_samples: 141516800 total_loss: 0.4247 time: 0.5406 s/iter data_time: 0.0526 s/iter total_throughput: 1894.11 samples/s lr: 7.04e-04 [09/20 00:59:26] lb.utils.events INFO: eta: 1 day, 11:52:06 iteration: 138299/375342 consumed_samples: 141619200 total_loss: 0.4298 time: 0.5406 s/iter data_time: 0.0552 s/iter total_throughput: 1894.10 samples/s lr: 7.04e-04 [09/20 01:00:20] lb.utils.events INFO: eta: 1 day, 11:51:48 iteration: 138399/375342 consumed_samples: 141721600 total_loss: 0.4313 time: 0.5406 s/iter data_time: 0.0555 s/iter total_throughput: 1894.08 samples/s lr: 7.03e-04 [09/20 01:01:15] lb.utils.events INFO: eta: 1 day, 11:51:58 iteration: 138499/375342 consumed_samples: 141824000 total_loss: 0.4331 time: 0.5406 s/iter data_time: 0.0569 s/iter total_throughput: 1894.07 samples/s lr: 7.03e-04 [09/20 01:02:10] lb.utils.events INFO: eta: 1 day, 11:52:48 iteration: 138599/375342 consumed_samples: 141926400 total_loss: 0.4293 time: 0.5406 s/iter data_time: 0.0580 s/iter total_throughput: 1894.05 samples/s lr: 7.03e-04 [09/20 01:03:04] lb.utils.events INFO: eta: 1 day, 11:52:01 iteration: 138699/375342 consumed_samples: 142028800 total_loss: 0.4254 time: 0.5406 s/iter data_time: 0.0554 s/iter total_throughput: 1894.04 samples/s lr: 7.02e-04 [09/20 01:03:59] lb.utils.events INFO: eta: 1 day, 11:51:43 iteration: 138799/375342 consumed_samples: 142131200 total_loss: 0.43 time: 0.5406 s/iter data_time: 0.0546 s/iter total_throughput: 1894.02 samples/s lr: 7.02e-04 [09/20 01:04:54] lb.utils.events INFO: eta: 1 day, 11:51:42 iteration: 138899/375342 consumed_samples: 142233600 total_loss: 0.4301 time: 0.5407 s/iter data_time: 0.0546 s/iter total_throughput: 1894.01 samples/s lr: 7.02e-04 [09/20 01:05:48] lb.utils.events INFO: eta: 1 day, 11:50:41 iteration: 138999/375342 consumed_samples: 142336000 total_loss: 0.4255 time: 0.5407 s/iter data_time: 0.0558 s/iter total_throughput: 1894.00 samples/s lr: 7.01e-04 [09/20 01:06:43] lb.utils.events INFO: eta: 1 day, 11:49:20 iteration: 139099/375342 consumed_samples: 142438400 total_loss: 0.4258 time: 0.5407 s/iter data_time: 0.0546 s/iter total_throughput: 1893.99 samples/s lr: 7.01e-04 [09/20 01:07:37] lb.utils.events INFO: eta: 1 day, 11:48:54 iteration: 139199/375342 consumed_samples: 142540800 total_loss: 0.4295 time: 0.5407 s/iter data_time: 0.0547 s/iter total_throughput: 1893.97 samples/s lr: 7.00e-04 [09/20 01:08:32] lb.utils.events INFO: eta: 1 day, 11:48:39 iteration: 139299/375342 consumed_samples: 142643200 total_loss: 0.4269 time: 0.5407 s/iter data_time: 0.0508 s/iter total_throughput: 1893.96 samples/s lr: 7.00e-04 [09/20 01:09:26] lb.utils.events INFO: eta: 1 day, 11:47:47 iteration: 139399/375342 consumed_samples: 142745600 total_loss: 0.4285 time: 0.5407 s/iter data_time: 0.0504 s/iter total_throughput: 1893.95 samples/s lr: 7.00e-04 [09/20 01:10:21] lb.utils.events INFO: eta: 1 day, 11:46:50 iteration: 139499/375342 consumed_samples: 142848000 total_loss: 0.4255 time: 0.5407 s/iter data_time: 0.0485 s/iter total_throughput: 1893.94 samples/s lr: 6.99e-04 [09/20 01:11:16] lb.utils.events INFO: eta: 1 day, 11:45:10 iteration: 139599/375342 consumed_samples: 142950400 total_loss: 0.4174 time: 0.5407 s/iter data_time: 0.0502 s/iter total_throughput: 1893.92 samples/s lr: 6.99e-04 [09/20 01:12:10] lb.utils.events INFO: eta: 1 day, 11:43:37 iteration: 139699/375342 consumed_samples: 143052800 total_loss: 0.4211 time: 0.5407 s/iter data_time: 0.0507 s/iter total_throughput: 1893.91 samples/s lr: 6.98e-04 [09/20 01:13:05] lb.utils.events INFO: eta: 1 day, 11:42:20 iteration: 139799/375342 consumed_samples: 143155200 total_loss: 0.4283 time: 0.5407 s/iter data_time: 0.0501 s/iter total_throughput: 1893.90 samples/s lr: 6.98e-04 [09/20 01:13:59] lb.utils.events INFO: eta: 1 day, 11:40:47 iteration: 139899/375342 consumed_samples: 143257600 total_loss: 0.4293 time: 0.5407 s/iter data_time: 0.0494 s/iter total_throughput: 1893.89 samples/s lr: 6.98e-04 [09/20 01:14:54] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0139999 [09/20 01:14:54] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 01:14:54] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 01:14:59] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0550 s/iter. Inference: 0.2495 s/iter. Eval: 0.0030 s/iter. Total: 0.3075 s/iter. ETA=0:00:11 [09/20 01:15:04] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0676 s/iter. Inference: 0.2633 s/iter. Eval: 0.0024 s/iter. Total: 0.3336 s/iter. ETA=0:00:07 [09/20 01:15:09] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0681 s/iter. Inference: 0.2587 s/iter. Eval: 0.0025 s/iter. Total: 0.3296 s/iter. ETA=0:00:01 [09/20 01:15:11] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 01:15:11] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.187975 (0.000284 s / iter per device, on 8 devices) [09/20 01:15:11] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000225 s / iter per device, on 8 devices) [09/20 01:15:11] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 01:15:11] lb.evaluation.utils INFO: copypaste: Acc@1=72.614 [09/20 01:15:11] lb.evaluation.utils INFO: copypaste: Acc@5=91.304 [09/20 01:15:11] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 72.61400, not better than best score 72.68400 @ iteration 134999. [09/20 01:15:11] lb.utils.events INFO: eta: 1 day, 11:39:43 iteration: 139999/375342 consumed_samples: 143360000 total_loss: 0.4238 time: 0.5407 s/iter data_time: 0.0508 s/iter total_throughput: 1893.88 samples/s lr: 6.97e-04 [09/20 01:16:06] lb.utils.events INFO: eta: 1 day, 11:38:57 iteration: 140099/375342 consumed_samples: 143462400 total_loss: 0.4236 time: 0.5407 s/iter data_time: 0.0508 s/iter total_throughput: 1893.87 samples/s lr: 6.97e-04 [09/20 01:17:01] lb.utils.events INFO: eta: 1 day, 11:38:03 iteration: 140199/375342 consumed_samples: 143564800 total_loss: 0.4279 time: 0.5407 s/iter data_time: 0.0527 s/iter total_throughput: 1893.86 samples/s lr: 6.97e-04 [09/20 01:17:55] lb.utils.events INFO: eta: 1 day, 11:36:05 iteration: 140299/375342 consumed_samples: 143667200 total_loss: 0.4246 time: 0.5407 s/iter data_time: 0.0519 s/iter total_throughput: 1893.85 samples/s lr: 6.96e-04 [09/20 01:18:49] lb.utils.events INFO: eta: 1 day, 11:34:26 iteration: 140399/375342 consumed_samples: 143769600 total_loss: 0.4248 time: 0.5407 s/iter data_time: 0.0511 s/iter total_throughput: 1893.84 samples/s lr: 6.96e-04 [09/20 01:19:44] lb.utils.events INFO: eta: 1 day, 11:33:24 iteration: 140499/375342 consumed_samples: 143872000 total_loss: 0.4335 time: 0.5407 s/iter data_time: 0.0522 s/iter total_throughput: 1893.82 samples/s lr: 6.95e-04 [09/20 01:20:38] lb.utils.events INFO: eta: 1 day, 11:32:09 iteration: 140599/375342 consumed_samples: 143974400 total_loss: 0.4338 time: 0.5407 s/iter data_time: 0.0517 s/iter total_throughput: 1893.82 samples/s lr: 6.95e-04 [09/20 01:21:33] lb.utils.events INFO: eta: 1 day, 11:30:47 iteration: 140699/375342 consumed_samples: 144076800 total_loss: 0.4369 time: 0.5407 s/iter data_time: 0.0499 s/iter total_throughput: 1893.81 samples/s lr: 6.95e-04 [09/20 01:22:27] lb.utils.events INFO: eta: 1 day, 11:29:42 iteration: 140799/375342 consumed_samples: 144179200 total_loss: 0.4428 time: 0.5407 s/iter data_time: 0.0527 s/iter total_throughput: 1893.80 samples/s lr: 6.94e-04 [09/20 01:23:22] lb.utils.events INFO: eta: 1 day, 11:28:35 iteration: 140899/375342 consumed_samples: 144281600 total_loss: 0.4328 time: 0.5407 s/iter data_time: 0.0508 s/iter total_throughput: 1893.79 samples/s lr: 6.94e-04 [09/20 01:24:16] lb.utils.events INFO: eta: 1 day, 11:27:12 iteration: 140999/375342 consumed_samples: 144384000 total_loss: 0.4297 time: 0.5407 s/iter data_time: 0.0519 s/iter total_throughput: 1893.78 samples/s lr: 6.93e-04 [09/20 01:25:11] lb.utils.events INFO: eta: 1 day, 11:25:16 iteration: 141099/375342 consumed_samples: 144486400 total_loss: 0.4322 time: 0.5407 s/iter data_time: 0.0483 s/iter total_throughput: 1893.77 samples/s lr: 6.93e-04 [09/20 01:26:05] lb.utils.events INFO: eta: 1 day, 11:24:37 iteration: 141199/375342 consumed_samples: 144588800 total_loss: 0.4324 time: 0.5407 s/iter data_time: 0.0490 s/iter total_throughput: 1893.76 samples/s lr: 6.93e-04 [09/20 01:27:00] lb.utils.events INFO: eta: 1 day, 11:23:45 iteration: 141299/375342 consumed_samples: 144691200 total_loss: 0.443 time: 0.5407 s/iter data_time: 0.0485 s/iter total_throughput: 1893.75 samples/s lr: 6.92e-04 [09/20 01:27:54] lb.utils.events INFO: eta: 1 day, 11:22:49 iteration: 141399/375342 consumed_samples: 144793600 total_loss: 0.4337 time: 0.5407 s/iter data_time: 0.0494 s/iter total_throughput: 1893.74 samples/s lr: 6.92e-04 [09/20 01:28:49] lb.utils.events INFO: eta: 1 day, 11:22:11 iteration: 141499/375342 consumed_samples: 144896000 total_loss: 0.4156 time: 0.5407 s/iter data_time: 0.0540 s/iter total_throughput: 1893.72 samples/s lr: 6.92e-04 [09/20 01:29:44] lb.utils.events INFO: eta: 1 day, 11:21:37 iteration: 141599/375342 consumed_samples: 144998400 total_loss: 0.4141 time: 0.5407 s/iter data_time: 0.0550 s/iter total_throughput: 1893.71 samples/s lr: 6.91e-04 [09/20 01:30:38] lb.utils.events INFO: eta: 1 day, 11:21:05 iteration: 141699/375342 consumed_samples: 145100800 total_loss: 0.4268 time: 0.5407 s/iter data_time: 0.0536 s/iter total_throughput: 1893.70 samples/s lr: 6.91e-04 [09/20 01:31:33] lb.utils.events INFO: eta: 1 day, 11:20:11 iteration: 141799/375342 consumed_samples: 145203200 total_loss: 0.4297 time: 0.5407 s/iter data_time: 0.0573 s/iter total_throughput: 1893.69 samples/s lr: 6.90e-04 [09/20 01:32:27] lb.utils.events INFO: eta: 1 day, 11:20:09 iteration: 141899/375342 consumed_samples: 145305600 total_loss: 0.4254 time: 0.5407 s/iter data_time: 0.0549 s/iter total_throughput: 1893.67 samples/s lr: 6.90e-04 [09/20 01:33:22] lb.utils.events INFO: eta: 1 day, 11:20:37 iteration: 141999/375342 consumed_samples: 145408000 total_loss: 0.4242 time: 0.5408 s/iter data_time: 0.0578 s/iter total_throughput: 1893.66 samples/s lr: 6.90e-04 [09/20 01:34:17] lb.utils.events INFO: eta: 1 day, 11:21:24 iteration: 142099/375342 consumed_samples: 145510400 total_loss: 0.427 time: 0.5408 s/iter data_time: 0.0538 s/iter total_throughput: 1893.64 samples/s lr: 6.89e-04 [09/20 01:35:12] lb.utils.events INFO: eta: 1 day, 11:20:43 iteration: 142199/375342 consumed_samples: 145612800 total_loss: 0.4335 time: 0.5408 s/iter data_time: 0.0543 s/iter total_throughput: 1893.63 samples/s lr: 6.89e-04 [09/20 01:36:06] lb.utils.events INFO: eta: 1 day, 11:20:49 iteration: 142299/375342 consumed_samples: 145715200 total_loss: 0.4362 time: 0.5408 s/iter data_time: 0.0563 s/iter total_throughput: 1893.62 samples/s lr: 6.88e-04 [09/20 01:37:01] lb.utils.events INFO: eta: 1 day, 11:20:43 iteration: 142399/375342 consumed_samples: 145817600 total_loss: 0.431 time: 0.5408 s/iter data_time: 0.0555 s/iter total_throughput: 1893.61 samples/s lr: 6.88e-04 [09/20 01:37:55] lb.utils.events INFO: eta: 1 day, 11:19:49 iteration: 142499/375342 consumed_samples: 145920000 total_loss: 0.4307 time: 0.5408 s/iter data_time: 0.0549 s/iter total_throughput: 1893.59 samples/s lr: 6.88e-04 [09/20 01:38:50] lb.utils.events INFO: eta: 1 day, 11:18:49 iteration: 142599/375342 consumed_samples: 146022400 total_loss: 0.4339 time: 0.5408 s/iter data_time: 0.0559 s/iter total_throughput: 1893.58 samples/s lr: 6.87e-04 [09/20 01:39:45] lb.utils.events INFO: eta: 1 day, 11:17:47 iteration: 142699/375342 consumed_samples: 146124800 total_loss: 0.436 time: 0.5408 s/iter data_time: 0.0522 s/iter total_throughput: 1893.57 samples/s lr: 6.87e-04 [09/20 01:40:39] lb.utils.events INFO: eta: 1 day, 11:17:10 iteration: 142799/375342 consumed_samples: 146227200 total_loss: 0.4325 time: 0.5408 s/iter data_time: 0.0513 s/iter total_throughput: 1893.56 samples/s lr: 6.87e-04 [09/20 01:41:34] lb.utils.events INFO: eta: 1 day, 11:15:49 iteration: 142899/375342 consumed_samples: 146329600 total_loss: 0.4266 time: 0.5408 s/iter data_time: 0.0505 s/iter total_throughput: 1893.55 samples/s lr: 6.86e-04 [09/20 01:42:28] lb.utils.events INFO: eta: 1 day, 11:14:03 iteration: 142999/375342 consumed_samples: 146432000 total_loss: 0.4237 time: 0.5408 s/iter data_time: 0.0492 s/iter total_throughput: 1893.54 samples/s lr: 6.86e-04 [09/20 01:43:23] lb.utils.events INFO: eta: 1 day, 11:12:43 iteration: 143099/375342 consumed_samples: 146534400 total_loss: 0.4272 time: 0.5408 s/iter data_time: 0.0494 s/iter total_throughput: 1893.53 samples/s lr: 6.85e-04 [09/20 01:44:17] lb.utils.events INFO: eta: 1 day, 11:11:44 iteration: 143199/375342 consumed_samples: 146636800 total_loss: 0.4284 time: 0.5408 s/iter data_time: 0.0500 s/iter total_throughput: 1893.51 samples/s lr: 6.85e-04 [09/20 01:45:12] lb.utils.events INFO: eta: 1 day, 11:10:17 iteration: 143299/375342 consumed_samples: 146739200 total_loss: 0.4161 time: 0.5408 s/iter data_time: 0.0506 s/iter total_throughput: 1893.50 samples/s lr: 6.85e-04 [09/20 01:46:06] lb.utils.events INFO: eta: 1 day, 11:08:45 iteration: 143399/375342 consumed_samples: 146841600 total_loss: 0.42 time: 0.5408 s/iter data_time: 0.0493 s/iter total_throughput: 1893.49 samples/s lr: 6.84e-04 [09/20 01:47:01] lb.utils.events INFO: eta: 1 day, 11:08:03 iteration: 143499/375342 consumed_samples: 146944000 total_loss: 0.4291 time: 0.5408 s/iter data_time: 0.0521 s/iter total_throughput: 1893.48 samples/s lr: 6.84e-04 [09/20 01:47:55] lb.utils.events INFO: eta: 1 day, 11:06:45 iteration: 143599/375342 consumed_samples: 147046400 total_loss: 0.424 time: 0.5408 s/iter data_time: 0.0508 s/iter total_throughput: 1893.47 samples/s lr: 6.83e-04 [09/20 01:48:50] lb.utils.events INFO: eta: 1 day, 11:05:25 iteration: 143699/375342 consumed_samples: 147148800 total_loss: 0.4216 time: 0.5408 s/iter data_time: 0.0509 s/iter total_throughput: 1893.47 samples/s lr: 6.83e-04 [09/20 01:49:44] lb.utils.events INFO: eta: 1 day, 11:03:49 iteration: 143799/375342 consumed_samples: 147251200 total_loss: 0.4233 time: 0.5408 s/iter data_time: 0.0515 s/iter total_throughput: 1893.46 samples/s lr: 6.83e-04 [09/20 01:50:39] lb.utils.events INFO: eta: 1 day, 11:02:18 iteration: 143899/375342 consumed_samples: 147353600 total_loss: 0.4294 time: 0.5408 s/iter data_time: 0.0516 s/iter total_throughput: 1893.45 samples/s lr: 6.82e-04 [09/20 01:51:33] lb.utils.events INFO: eta: 1 day, 11:00:58 iteration: 143999/375342 consumed_samples: 147456000 total_loss: 0.4303 time: 0.5408 s/iter data_time: 0.0511 s/iter total_throughput: 1893.44 samples/s lr: 6.82e-04 [09/20 01:52:28] lb.utils.events INFO: eta: 1 day, 10:59:41 iteration: 144099/375342 consumed_samples: 147558400 total_loss: 0.4266 time: 0.5408 s/iter data_time: 0.0523 s/iter total_throughput: 1893.43 samples/s lr: 6.82e-04 [09/20 01:53:22] lb.utils.events INFO: eta: 1 day, 10:58:22 iteration: 144199/375342 consumed_samples: 147660800 total_loss: 0.4262 time: 0.5408 s/iter data_time: 0.0512 s/iter total_throughput: 1893.42 samples/s lr: 6.81e-04 [09/20 01:54:16] lb.utils.events INFO: eta: 1 day, 10:57:15 iteration: 144299/375342 consumed_samples: 147763200 total_loss: 0.4197 time: 0.5408 s/iter data_time: 0.0525 s/iter total_throughput: 1893.41 samples/s lr: 6.81e-04 [09/20 01:55:11] lb.utils.events INFO: eta: 1 day, 10:55:49 iteration: 144399/375342 consumed_samples: 147865600 total_loss: 0.4194 time: 0.5408 s/iter data_time: 0.0520 s/iter total_throughput: 1893.41 samples/s lr: 6.80e-04 [09/20 01:56:05] lb.utils.events INFO: eta: 1 day, 10:54:12 iteration: 144499/375342 consumed_samples: 147968000 total_loss: 0.4305 time: 0.5408 s/iter data_time: 0.0522 s/iter total_throughput: 1893.40 samples/s lr: 6.80e-04 [09/20 01:57:00] lb.utils.events INFO: eta: 1 day, 10:53:25 iteration: 144599/375342 consumed_samples: 148070400 total_loss: 0.4239 time: 0.5408 s/iter data_time: 0.0496 s/iter total_throughput: 1893.39 samples/s lr: 6.80e-04 [09/20 01:57:54] lb.utils.events INFO: eta: 1 day, 10:52:24 iteration: 144699/375342 consumed_samples: 148172800 total_loss: 0.4212 time: 0.5408 s/iter data_time: 0.0505 s/iter total_throughput: 1893.38 samples/s lr: 6.79e-04 [09/20 01:58:49] lb.utils.events INFO: eta: 1 day, 10:52:00 iteration: 144799/375342 consumed_samples: 148275200 total_loss: 0.423 time: 0.5408 s/iter data_time: 0.0508 s/iter total_throughput: 1893.37 samples/s lr: 6.79e-04 [09/20 01:59:44] lb.utils.events INFO: eta: 1 day, 10:50:51 iteration: 144899/375342 consumed_samples: 148377600 total_loss: 0.4202 time: 0.5408 s/iter data_time: 0.0528 s/iter total_throughput: 1893.36 samples/s lr: 6.78e-04 [09/20 02:00:38] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0144999 [09/20 02:00:39] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 02:00:39] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 02:00:43] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0508 s/iter. Inference: 0.2440 s/iter. Eval: 0.0021 s/iter. Total: 0.2970 s/iter. ETA=0:00:10 [09/20 02:00:48] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0766 s/iter. Inference: 0.2547 s/iter. Eval: 0.0023 s/iter. Total: 0.3339 s/iter. ETA=0:00:07 [09/20 02:00:54] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0758 s/iter. Inference: 0.2527 s/iter. Eval: 0.0023 s/iter. Total: 0.3311 s/iter. ETA=0:00:01 [09/20 02:00:56] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 02:00:56] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.295000 (0.000286 s / iter per device, on 8 devices) [09/20 02:00:56] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000221 s / iter per device, on 8 devices) [09/20 02:00:56] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 02:00:56] lb.evaluation.utils INFO: copypaste: Acc@1=73.22999999999999 [09/20 02:00:56] lb.evaluation.utils INFO: copypaste: Acc@5=91.56800000000001 [09/20 02:00:56] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 73.23000, better than last best score 72.68400 @ iteration 134999. [09/20 02:00:56] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 02:00:56] lb.utils.events INFO: eta: 1 day, 10:50:28 iteration: 144999/375342 consumed_samples: 148480000 total_loss: 0.4253 time: 0.5408 s/iter data_time: 0.0548 s/iter total_throughput: 1893.35 samples/s lr: 6.78e-04 [09/20 02:01:51] lb.utils.events INFO: eta: 1 day, 10:49:08 iteration: 145099/375342 consumed_samples: 148582400 total_loss: 0.4255 time: 0.5408 s/iter data_time: 0.0542 s/iter total_throughput: 1893.34 samples/s lr: 6.78e-04 [09/20 02:02:45] lb.utils.events INFO: eta: 1 day, 10:48:14 iteration: 145199/375342 consumed_samples: 148684800 total_loss: 0.4247 time: 0.5408 s/iter data_time: 0.0564 s/iter total_throughput: 1893.33 samples/s lr: 6.77e-04 [09/20 02:03:40] lb.utils.events INFO: eta: 1 day, 10:47:52 iteration: 145299/375342 consumed_samples: 148787200 total_loss: 0.4247 time: 0.5408 s/iter data_time: 0.0536 s/iter total_throughput: 1893.32 samples/s lr: 6.77e-04 [09/20 02:04:35] lb.utils.events INFO: eta: 1 day, 10:48:09 iteration: 145399/375342 consumed_samples: 148889600 total_loss: 0.4208 time: 0.5409 s/iter data_time: 0.0552 s/iter total_throughput: 1893.30 samples/s lr: 6.77e-04 [09/20 02:05:29] lb.utils.events INFO: eta: 1 day, 10:48:15 iteration: 145499/375342 consumed_samples: 148992000 total_loss: 0.414 time: 0.5409 s/iter data_time: 0.0577 s/iter total_throughput: 1893.29 samples/s lr: 6.76e-04 [09/20 02:06:24] lb.utils.events INFO: eta: 1 day, 10:48:21 iteration: 145599/375342 consumed_samples: 149094400 total_loss: 0.4223 time: 0.5409 s/iter data_time: 0.0545 s/iter total_throughput: 1893.28 samples/s lr: 6.76e-04 [09/20 02:07:19] lb.utils.events INFO: eta: 1 day, 10:48:06 iteration: 145699/375342 consumed_samples: 149196800 total_loss: 0.4328 time: 0.5409 s/iter data_time: 0.0543 s/iter total_throughput: 1893.27 samples/s lr: 6.75e-04 [09/20 02:08:13] lb.utils.events INFO: eta: 1 day, 10:46:32 iteration: 145799/375342 consumed_samples: 149299200 total_loss: 0.4311 time: 0.5409 s/iter data_time: 0.0533 s/iter total_throughput: 1893.25 samples/s lr: 6.75e-04 [09/20 02:09:08] lb.utils.events INFO: eta: 1 day, 10:45:55 iteration: 145899/375342 consumed_samples: 149401600 total_loss: 0.4291 time: 0.5409 s/iter data_time: 0.0538 s/iter total_throughput: 1893.24 samples/s lr: 6.75e-04 [09/20 02:10:02] lb.utils.events INFO: eta: 1 day, 10:45:05 iteration: 145999/375342 consumed_samples: 149504000 total_loss: 0.4294 time: 0.5409 s/iter data_time: 0.0549 s/iter total_throughput: 1893.23 samples/s lr: 6.74e-04 [09/20 02:10:57] lb.utils.events INFO: eta: 1 day, 10:45:36 iteration: 146099/375342 consumed_samples: 149606400 total_loss: 0.4204 time: 0.5409 s/iter data_time: 0.0542 s/iter total_throughput: 1893.22 samples/s lr: 6.74e-04 [09/20 02:11:51] lb.utils.events INFO: eta: 1 day, 10:44:51 iteration: 146199/375342 consumed_samples: 149708800 total_loss: 0.4183 time: 0.5409 s/iter data_time: 0.0514 s/iter total_throughput: 1893.21 samples/s lr: 6.73e-04 [09/20 02:12:46] lb.utils.events INFO: eta: 1 day, 10:43:52 iteration: 146299/375342 consumed_samples: 149811200 total_loss: 0.4247 time: 0.5409 s/iter data_time: 0.0482 s/iter total_throughput: 1893.20 samples/s lr: 6.73e-04 [09/20 02:13:40] lb.utils.events INFO: eta: 1 day, 10:42:04 iteration: 146399/375342 consumed_samples: 149913600 total_loss: 0.4263 time: 0.5409 s/iter data_time: 0.0488 s/iter total_throughput: 1893.19 samples/s lr: 6.73e-04 [09/20 02:14:35] lb.utils.events INFO: eta: 1 day, 10:40:32 iteration: 146499/375342 consumed_samples: 150016000 total_loss: 0.4267 time: 0.5409 s/iter data_time: 0.0519 s/iter total_throughput: 1893.18 samples/s lr: 6.72e-04 [09/20 02:15:30] lb.utils.events INFO: eta: 1 day, 10:39:12 iteration: 146599/375342 consumed_samples: 150118400 total_loss: 0.4335 time: 0.5409 s/iter data_time: 0.0511 s/iter total_throughput: 1893.17 samples/s lr: 6.72e-04 [09/20 02:16:24] lb.utils.events INFO: eta: 1 day, 10:38:13 iteration: 146699/375342 consumed_samples: 150220800 total_loss: 0.4225 time: 0.5409 s/iter data_time: 0.0519 s/iter total_throughput: 1893.16 samples/s lr: 6.71e-04 [09/20 02:17:19] lb.utils.events INFO: eta: 1 day, 10:37:26 iteration: 146799/375342 consumed_samples: 150323200 total_loss: 0.4204 time: 0.5409 s/iter data_time: 0.0518 s/iter total_throughput: 1893.15 samples/s lr: 6.71e-04 [09/20 02:18:13] lb.utils.events INFO: eta: 1 day, 10:36:51 iteration: 146899/375342 consumed_samples: 150425600 total_loss: 0.4295 time: 0.5409 s/iter data_time: 0.0522 s/iter total_throughput: 1893.14 samples/s lr: 6.71e-04 [09/20 02:19:08] lb.utils.events INFO: eta: 1 day, 10:36:36 iteration: 146999/375342 consumed_samples: 150528000 total_loss: 0.4272 time: 0.5409 s/iter data_time: 0.0523 s/iter total_throughput: 1893.12 samples/s lr: 6.70e-04 [09/20 02:20:03] lb.utils.events INFO: eta: 1 day, 10:36:17 iteration: 147099/375342 consumed_samples: 150630400 total_loss: 0.4194 time: 0.5409 s/iter data_time: 0.0533 s/iter total_throughput: 1893.11 samples/s lr: 6.70e-04 [09/20 02:20:57] lb.utils.events INFO: eta: 1 day, 10:35:57 iteration: 147199/375342 consumed_samples: 150732800 total_loss: 0.4255 time: 0.5409 s/iter data_time: 0.0532 s/iter total_throughput: 1893.09 samples/s lr: 6.69e-04 [09/20 02:21:52] lb.utils.events INFO: eta: 1 day, 10:35:47 iteration: 147299/375342 consumed_samples: 150835200 total_loss: 0.4303 time: 0.5409 s/iter data_time: 0.0544 s/iter total_throughput: 1893.08 samples/s lr: 6.69e-04 [09/20 02:22:47] lb.utils.events INFO: eta: 1 day, 10:35:48 iteration: 147399/375342 consumed_samples: 150937600 total_loss: 0.4316 time: 0.5409 s/iter data_time: 0.0530 s/iter total_throughput: 1893.06 samples/s lr: 6.69e-04 [09/20 02:23:42] lb.utils.events INFO: eta: 1 day, 10:35:04 iteration: 147499/375342 consumed_samples: 151040000 total_loss: 0.4326 time: 0.5409 s/iter data_time: 0.0535 s/iter total_throughput: 1893.05 samples/s lr: 6.68e-04 [09/20 02:24:36] lb.utils.events INFO: eta: 1 day, 10:34:02 iteration: 147599/375342 consumed_samples: 151142400 total_loss: 0.4298 time: 0.5409 s/iter data_time: 0.0509 s/iter total_throughput: 1893.04 samples/s lr: 6.68e-04 [09/20 02:25:31] lb.utils.events INFO: eta: 1 day, 10:32:49 iteration: 147699/375342 consumed_samples: 151244800 total_loss: 0.425 time: 0.5409 s/iter data_time: 0.0524 s/iter total_throughput: 1893.03 samples/s lr: 6.68e-04 [09/20 02:26:25] lb.utils.events INFO: eta: 1 day, 10:31:07 iteration: 147799/375342 consumed_samples: 151347200 total_loss: 0.4311 time: 0.5409 s/iter data_time: 0.0521 s/iter total_throughput: 1893.02 samples/s lr: 6.67e-04 [09/20 02:27:19] lb.utils.events INFO: eta: 1 day, 10:29:34 iteration: 147899/375342 consumed_samples: 151449600 total_loss: 0.4325 time: 0.5409 s/iter data_time: 0.0525 s/iter total_throughput: 1893.02 samples/s lr: 6.67e-04 [09/20 02:28:14] lb.utils.events INFO: eta: 1 day, 10:27:50 iteration: 147999/375342 consumed_samples: 151552000 total_loss: 0.4248 time: 0.5409 s/iter data_time: 0.0484 s/iter total_throughput: 1893.01 samples/s lr: 6.66e-04 [09/20 02:29:09] lb.utils.events INFO: eta: 1 day, 10:25:55 iteration: 148099/375342 consumed_samples: 151654400 total_loss: 0.4202 time: 0.5409 s/iter data_time: 0.0483 s/iter total_throughput: 1893.00 samples/s lr: 6.66e-04 [09/20 02:30:03] lb.utils.events INFO: eta: 1 day, 10:24:50 iteration: 148199/375342 consumed_samples: 151756800 total_loss: 0.4217 time: 0.5409 s/iter data_time: 0.0520 s/iter total_throughput: 1892.99 samples/s lr: 6.66e-04 [09/20 02:30:58] lb.utils.events INFO: eta: 1 day, 10:23:04 iteration: 148299/375342 consumed_samples: 151859200 total_loss: 0.421 time: 0.5409 s/iter data_time: 0.0487 s/iter total_throughput: 1892.97 samples/s lr: 6.65e-04 [09/20 02:31:53] lb.utils.events INFO: eta: 1 day, 10:22:05 iteration: 148399/375342 consumed_samples: 151961600 total_loss: 0.4235 time: 0.5410 s/iter data_time: 0.0542 s/iter total_throughput: 1892.95 samples/s lr: 6.65e-04 [09/20 02:32:47] lb.utils.events INFO: eta: 1 day, 10:21:11 iteration: 148499/375342 consumed_samples: 152064000 total_loss: 0.4192 time: 0.5410 s/iter data_time: 0.0555 s/iter total_throughput: 1892.94 samples/s lr: 6.64e-04 [09/20 02:33:42] lb.utils.events INFO: eta: 1 day, 10:20:13 iteration: 148599/375342 consumed_samples: 152166400 total_loss: 0.4112 time: 0.5410 s/iter data_time: 0.0559 s/iter total_throughput: 1892.93 samples/s lr: 6.64e-04 [09/20 02:34:37] lb.utils.events INFO: eta: 1 day, 10:19:50 iteration: 148699/375342 consumed_samples: 152268800 total_loss: 0.4235 time: 0.5410 s/iter data_time: 0.0539 s/iter total_throughput: 1892.92 samples/s lr: 6.64e-04 [09/20 02:35:31] lb.utils.events INFO: eta: 1 day, 10:20:17 iteration: 148799/375342 consumed_samples: 152371200 total_loss: 0.4263 time: 0.5410 s/iter data_time: 0.0557 s/iter total_throughput: 1892.91 samples/s lr: 6.63e-04 [09/20 02:36:26] lb.utils.events INFO: eta: 1 day, 10:20:54 iteration: 148899/375342 consumed_samples: 152473600 total_loss: 0.4286 time: 0.5410 s/iter data_time: 0.0538 s/iter total_throughput: 1892.89 samples/s lr: 6.63e-04 [09/20 02:37:21] lb.utils.events INFO: eta: 1 day, 10:21:09 iteration: 148999/375342 consumed_samples: 152576000 total_loss: 0.4189 time: 0.5410 s/iter data_time: 0.0576 s/iter total_throughput: 1892.88 samples/s lr: 6.62e-04 [09/20 02:38:15] lb.utils.events INFO: eta: 1 day, 10:20:58 iteration: 149099/375342 consumed_samples: 152678400 total_loss: 0.4089 time: 0.5410 s/iter data_time: 0.0566 s/iter total_throughput: 1892.86 samples/s lr: 6.62e-04 [09/20 02:39:10] lb.utils.events INFO: eta: 1 day, 10:19:51 iteration: 149199/375342 consumed_samples: 152780800 total_loss: 0.4169 time: 0.5410 s/iter data_time: 0.0547 s/iter total_throughput: 1892.85 samples/s lr: 6.62e-04 [09/20 02:40:05] lb.utils.events INFO: eta: 1 day, 10:19:13 iteration: 149299/375342 consumed_samples: 152883200 total_loss: 0.4297 time: 0.5410 s/iter data_time: 0.0545 s/iter total_throughput: 1892.84 samples/s lr: 6.61e-04 [09/20 02:40:59] lb.utils.events INFO: eta: 1 day, 10:18:01 iteration: 149399/375342 consumed_samples: 152985600 total_loss: 0.4266 time: 0.5410 s/iter data_time: 0.0549 s/iter total_throughput: 1892.83 samples/s lr: 6.61e-04 [09/20 02:41:54] lb.utils.events INFO: eta: 1 day, 10:17:19 iteration: 149499/375342 consumed_samples: 153088000 total_loss: 0.4205 time: 0.5410 s/iter data_time: 0.0549 s/iter total_throughput: 1892.82 samples/s lr: 6.60e-04 [09/20 02:42:49] lb.utils.events INFO: eta: 1 day, 10:16:32 iteration: 149599/375342 consumed_samples: 153190400 total_loss: 0.4222 time: 0.5410 s/iter data_time: 0.0519 s/iter total_throughput: 1892.80 samples/s lr: 6.60e-04 [09/20 02:43:43] lb.utils.events INFO: eta: 1 day, 10:15:37 iteration: 149699/375342 consumed_samples: 153292800 total_loss: 0.4184 time: 0.5410 s/iter data_time: 0.0504 s/iter total_throughput: 1892.79 samples/s lr: 6.60e-04 [09/20 02:44:38] lb.utils.events INFO: eta: 1 day, 10:14:17 iteration: 149799/375342 consumed_samples: 153395200 total_loss: 0.4159 time: 0.5410 s/iter data_time: 0.0510 s/iter total_throughput: 1892.78 samples/s lr: 6.59e-04 [09/20 02:45:33] lb.utils.events INFO: eta: 1 day, 10:12:21 iteration: 149899/375342 consumed_samples: 153497600 total_loss: 0.4217 time: 0.5410 s/iter data_time: 0.0494 s/iter total_throughput: 1892.77 samples/s lr: 6.59e-04 [09/20 02:46:27] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0149999 [09/20 02:46:28] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 02:46:28] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 02:46:32] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0748 s/iter. Inference: 0.2387 s/iter. Eval: 0.0024 s/iter. Total: 0.3159 s/iter. ETA=0:00:11 [09/20 02:46:38] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0777 s/iter. Inference: 0.2568 s/iter. Eval: 0.0024 s/iter. Total: 0.3371 s/iter. ETA=0:00:07 [09/20 02:46:43] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0771 s/iter. Inference: 0.2526 s/iter. Eval: 0.0024 s/iter. Total: 0.3324 s/iter. ETA=0:00:01 [09/20 02:46:45] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 02:46:45] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.288038 (0.000286 s / iter per device, on 8 devices) [09/20 02:46:45] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000220 s / iter per device, on 8 devices) [09/20 02:46:45] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 02:46:45] lb.evaluation.utils INFO: copypaste: Acc@1=73.396 [09/20 02:46:45] lb.evaluation.utils INFO: copypaste: Acc@5=91.85600000000001 [09/20 02:46:45] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 73.39600, better than last best score 73.23000 @ iteration 144999. [09/20 02:46:45] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 02:46:45] lb.utils.events INFO: eta: 1 day, 10:10:50 iteration: 149999/375342 consumed_samples: 153600000 total_loss: 0.4269 time: 0.5410 s/iter data_time: 0.0506 s/iter total_throughput: 1892.76 samples/s lr: 6.59e-04 [09/20 02:47:40] lb.utils.events INFO: eta: 1 day, 10:09:36 iteration: 150099/375342 consumed_samples: 153702400 total_loss: 0.4251 time: 0.5410 s/iter data_time: 0.0494 s/iter total_throughput: 1892.75 samples/s lr: 6.58e-04 [09/20 02:48:35] lb.utils.events INFO: eta: 1 day, 10:08:32 iteration: 150199/375342 consumed_samples: 153804800 total_loss: 0.4216 time: 0.5410 s/iter data_time: 0.0514 s/iter total_throughput: 1892.74 samples/s lr: 6.58e-04 [09/20 02:49:29] lb.utils.events INFO: eta: 1 day, 10:07:46 iteration: 150299/375342 consumed_samples: 153907200 total_loss: 0.4228 time: 0.5410 s/iter data_time: 0.0511 s/iter total_throughput: 1892.72 samples/s lr: 6.57e-04 [09/20 02:50:24] lb.utils.events INFO: eta: 1 day, 10:06:43 iteration: 150399/375342 consumed_samples: 154009600 total_loss: 0.4274 time: 0.5410 s/iter data_time: 0.0521 s/iter total_throughput: 1892.71 samples/s lr: 6.57e-04 [09/20 02:51:19] lb.utils.events INFO: eta: 1 day, 10:05:49 iteration: 150499/375342 consumed_samples: 154112000 total_loss: 0.4266 time: 0.5410 s/iter data_time: 0.0517 s/iter total_throughput: 1892.69 samples/s lr: 6.57e-04 [09/20 02:52:13] lb.utils.events INFO: eta: 1 day, 10:04:54 iteration: 150599/375342 consumed_samples: 154214400 total_loss: 0.422 time: 0.5410 s/iter data_time: 0.0532 s/iter total_throughput: 1892.68 samples/s lr: 6.56e-04 [09/20 02:53:08] lb.utils.events INFO: eta: 1 day, 10:03:31 iteration: 150699/375342 consumed_samples: 154316800 total_loss: 0.4254 time: 0.5410 s/iter data_time: 0.0527 s/iter total_throughput: 1892.67 samples/s lr: 6.56e-04 [09/20 02:54:03] lb.utils.events INFO: eta: 1 day, 10:02:35 iteration: 150799/375342 consumed_samples: 154419200 total_loss: 0.4212 time: 0.5410 s/iter data_time: 0.0523 s/iter total_throughput: 1892.65 samples/s lr: 6.55e-04 [09/20 02:54:58] lb.utils.events INFO: eta: 1 day, 10:02:05 iteration: 150899/375342 consumed_samples: 154521600 total_loss: 0.4241 time: 0.5410 s/iter data_time: 0.0516 s/iter total_throughput: 1892.64 samples/s lr: 6.55e-04 [09/20 02:55:52] lb.utils.events INFO: eta: 1 day, 10:01:34 iteration: 150999/375342 consumed_samples: 154624000 total_loss: 0.4299 time: 0.5410 s/iter data_time: 0.0530 s/iter total_throughput: 1892.63 samples/s lr: 6.55e-04 [09/20 02:56:47] lb.utils.events INFO: eta: 1 day, 10:01:25 iteration: 151099/375342 consumed_samples: 154726400 total_loss: 0.4281 time: 0.5411 s/iter data_time: 0.0517 s/iter total_throughput: 1892.61 samples/s lr: 6.54e-04 [09/20 02:57:42] lb.utils.events INFO: eta: 1 day, 10:02:10 iteration: 151199/375342 consumed_samples: 154828800 total_loss: 0.4263 time: 0.5411 s/iter data_time: 0.0509 s/iter total_throughput: 1892.59 samples/s lr: 6.54e-04 [09/20 02:58:37] lb.utils.events INFO: eta: 1 day, 10:01:18 iteration: 151299/375342 consumed_samples: 154931200 total_loss: 0.4286 time: 0.5411 s/iter data_time: 0.0522 s/iter total_throughput: 1892.58 samples/s lr: 6.53e-04 [09/20 02:59:31] lb.utils.events INFO: eta: 1 day, 10:00:50 iteration: 151399/375342 consumed_samples: 155033600 total_loss: 0.4235 time: 0.5411 s/iter data_time: 0.0511 s/iter total_throughput: 1892.57 samples/s lr: 6.53e-04 [09/20 03:00:26] lb.utils.events INFO: eta: 1 day, 10:00:00 iteration: 151499/375342 consumed_samples: 155136000 total_loss: 0.4197 time: 0.5411 s/iter data_time: 0.0519 s/iter total_throughput: 1892.55 samples/s lr: 6.53e-04 [09/20 03:01:21] lb.utils.events INFO: eta: 1 day, 9:59:17 iteration: 151599/375342 consumed_samples: 155238400 total_loss: 0.4245 time: 0.5411 s/iter data_time: 0.0537 s/iter total_throughput: 1892.54 samples/s lr: 6.52e-04 [09/20 03:02:16] lb.utils.events INFO: eta: 1 day, 9:59:24 iteration: 151699/375342 consumed_samples: 155340800 total_loss: 0.43 time: 0.5411 s/iter data_time: 0.0535 s/iter total_throughput: 1892.52 samples/s lr: 6.52e-04 [09/20 03:03:11] lb.utils.events INFO: eta: 1 day, 9:59:29 iteration: 151799/375342 consumed_samples: 155443200 total_loss: 0.4245 time: 0.5411 s/iter data_time: 0.0550 s/iter total_throughput: 1892.50 samples/s lr: 6.51e-04 [09/20 03:04:06] lb.utils.events INFO: eta: 1 day, 9:58:19 iteration: 151899/375342 consumed_samples: 155545600 total_loss: 0.4169 time: 0.5411 s/iter data_time: 0.0552 s/iter total_throughput: 1892.49 samples/s lr: 6.51e-04 [09/20 03:05:00] lb.utils.events INFO: eta: 1 day, 9:56:58 iteration: 151999/375342 consumed_samples: 155648000 total_loss: 0.4247 time: 0.5411 s/iter data_time: 0.0550 s/iter total_throughput: 1892.47 samples/s lr: 6.51e-04 [09/20 03:05:55] lb.utils.events INFO: eta: 1 day, 9:55:44 iteration: 152099/375342 consumed_samples: 155750400 total_loss: 0.4251 time: 0.5411 s/iter data_time: 0.0547 s/iter total_throughput: 1892.46 samples/s lr: 6.50e-04 [09/20 03:06:50] lb.utils.events INFO: eta: 1 day, 9:54:38 iteration: 152199/375342 consumed_samples: 155852800 total_loss: 0.4266 time: 0.5411 s/iter data_time: 0.0554 s/iter total_throughput: 1892.45 samples/s lr: 6.50e-04 [09/20 03:07:44] lb.utils.events INFO: eta: 1 day, 9:53:52 iteration: 152299/375342 consumed_samples: 155955200 total_loss: 0.4295 time: 0.5411 s/iter data_time: 0.0534 s/iter total_throughput: 1892.44 samples/s lr: 6.49e-04 [09/20 03:08:39] lb.utils.events INFO: eta: 1 day, 9:52:57 iteration: 152399/375342 consumed_samples: 156057600 total_loss: 0.4261 time: 0.5411 s/iter data_time: 0.0549 s/iter total_throughput: 1892.42 samples/s lr: 6.49e-04 [09/20 03:09:34] lb.utils.events INFO: eta: 1 day, 9:51:53 iteration: 152499/375342 consumed_samples: 156160000 total_loss: 0.4259 time: 0.5411 s/iter data_time: 0.0564 s/iter total_throughput: 1892.41 samples/s lr: 6.49e-04 [09/20 03:10:29] lb.utils.events INFO: eta: 1 day, 9:50:42 iteration: 152599/375342 consumed_samples: 156262400 total_loss: 0.4302 time: 0.5411 s/iter data_time: 0.0561 s/iter total_throughput: 1892.40 samples/s lr: 6.48e-04 [09/20 03:11:23] lb.utils.events INFO: eta: 1 day, 9:49:12 iteration: 152699/375342 consumed_samples: 156364800 total_loss: 0.4211 time: 0.5411 s/iter data_time: 0.0555 s/iter total_throughput: 1892.39 samples/s lr: 6.48e-04 [09/20 03:12:18] lb.utils.events INFO: eta: 1 day, 9:47:59 iteration: 152799/375342 consumed_samples: 156467200 total_loss: 0.418 time: 0.5411 s/iter data_time: 0.0559 s/iter total_throughput: 1892.37 samples/s lr: 6.47e-04 [09/20 03:13:12] lb.utils.events INFO: eta: 1 day, 9:47:20 iteration: 152899/375342 consumed_samples: 156569600 total_loss: 0.4188 time: 0.5411 s/iter data_time: 0.0567 s/iter total_throughput: 1892.36 samples/s lr: 6.47e-04 [09/20 03:14:07] lb.utils.events INFO: eta: 1 day, 9:46:18 iteration: 152999/375342 consumed_samples: 156672000 total_loss: 0.4292 time: 0.5411 s/iter data_time: 0.0541 s/iter total_throughput: 1892.35 samples/s lr: 6.47e-04 [09/20 03:15:02] lb.utils.events INFO: eta: 1 day, 9:45:15 iteration: 153099/375342 consumed_samples: 156774400 total_loss: 0.43 time: 0.5411 s/iter data_time: 0.0545 s/iter total_throughput: 1892.34 samples/s lr: 6.46e-04 [09/20 03:15:56] lb.utils.events INFO: eta: 1 day, 9:44:11 iteration: 153199/375342 consumed_samples: 156876800 total_loss: 0.4224 time: 0.5411 s/iter data_time: 0.0564 s/iter total_throughput: 1892.33 samples/s lr: 6.46e-04 [09/20 03:16:51] lb.utils.events INFO: eta: 1 day, 9:42:44 iteration: 153299/375342 consumed_samples: 156979200 total_loss: 0.4217 time: 0.5411 s/iter data_time: 0.0532 s/iter total_throughput: 1892.32 samples/s lr: 6.45e-04 [09/20 03:17:46] lb.utils.events INFO: eta: 1 day, 9:41:20 iteration: 153399/375342 consumed_samples: 157081600 total_loss: 0.4173 time: 0.5411 s/iter data_time: 0.0553 s/iter total_throughput: 1892.30 samples/s lr: 6.45e-04 [09/20 03:18:40] lb.utils.events INFO: eta: 1 day, 9:40:22 iteration: 153499/375342 consumed_samples: 157184000 total_loss: 0.4139 time: 0.5411 s/iter data_time: 0.0542 s/iter total_throughput: 1892.29 samples/s lr: 6.45e-04 [09/20 03:19:35] lb.utils.events INFO: eta: 1 day, 9:39:03 iteration: 153599/375342 consumed_samples: 157286400 total_loss: 0.4187 time: 0.5411 s/iter data_time: 0.0546 s/iter total_throughput: 1892.28 samples/s lr: 6.44e-04 [09/20 03:20:30] lb.utils.events INFO: eta: 1 day, 9:38:13 iteration: 153699/375342 consumed_samples: 157388800 total_loss: 0.4171 time: 0.5411 s/iter data_time: 0.0514 s/iter total_throughput: 1892.27 samples/s lr: 6.44e-04 [09/20 03:21:24] lb.utils.events INFO: eta: 1 day, 9:37:28 iteration: 153799/375342 consumed_samples: 157491200 total_loss: 0.4202 time: 0.5412 s/iter data_time: 0.0532 s/iter total_throughput: 1892.26 samples/s lr: 6.43e-04 [09/20 03:22:19] lb.utils.events INFO: eta: 1 day, 9:36:28 iteration: 153899/375342 consumed_samples: 157593600 total_loss: 0.4222 time: 0.5412 s/iter data_time: 0.0506 s/iter total_throughput: 1892.24 samples/s lr: 6.43e-04 [09/20 03:23:14] lb.utils.events INFO: eta: 1 day, 9:35:29 iteration: 153999/375342 consumed_samples: 157696000 total_loss: 0.4233 time: 0.5412 s/iter data_time: 0.0537 s/iter total_throughput: 1892.23 samples/s lr: 6.43e-04 [09/20 03:24:09] lb.utils.events INFO: eta: 1 day, 9:35:30 iteration: 154099/375342 consumed_samples: 157798400 total_loss: 0.4195 time: 0.5412 s/iter data_time: 0.0527 s/iter total_throughput: 1892.21 samples/s lr: 6.42e-04 [09/20 03:25:04] lb.utils.events INFO: eta: 1 day, 9:35:02 iteration: 154199/375342 consumed_samples: 157900800 total_loss: 0.4124 time: 0.5412 s/iter data_time: 0.0521 s/iter total_throughput: 1892.20 samples/s lr: 6.42e-04 [09/20 03:25:58] lb.utils.events INFO: eta: 1 day, 9:34:47 iteration: 154299/375342 consumed_samples: 158003200 total_loss: 0.4185 time: 0.5412 s/iter data_time: 0.0521 s/iter total_throughput: 1892.18 samples/s lr: 6.41e-04 [09/20 03:26:53] lb.utils.events INFO: eta: 1 day, 9:34:28 iteration: 154399/375342 consumed_samples: 158105600 total_loss: 0.4185 time: 0.5412 s/iter data_time: 0.0533 s/iter total_throughput: 1892.17 samples/s lr: 6.41e-04 [09/20 03:27:48] lb.utils.events INFO: eta: 1 day, 9:33:55 iteration: 154499/375342 consumed_samples: 158208000 total_loss: 0.4197 time: 0.5412 s/iter data_time: 0.0523 s/iter total_throughput: 1892.16 samples/s lr: 6.41e-04 [09/20 03:28:43] lb.utils.events INFO: eta: 1 day, 9:33:12 iteration: 154599/375342 consumed_samples: 158310400 total_loss: 0.4249 time: 0.5412 s/iter data_time: 0.0530 s/iter total_throughput: 1892.14 samples/s lr: 6.40e-04 [09/20 03:29:37] lb.utils.events INFO: eta: 1 day, 9:33:07 iteration: 154699/375342 consumed_samples: 158412800 total_loss: 0.423 time: 0.5412 s/iter data_time: 0.0531 s/iter total_throughput: 1892.13 samples/s lr: 6.40e-04 [09/20 03:30:32] lb.utils.events INFO: eta: 1 day, 9:32:27 iteration: 154799/375342 consumed_samples: 158515200 total_loss: 0.4179 time: 0.5412 s/iter data_time: 0.0532 s/iter total_throughput: 1892.11 samples/s lr: 6.39e-04 [09/20 03:31:27] lb.utils.events INFO: eta: 1 day, 9:32:56 iteration: 154899/375342 consumed_samples: 158617600 total_loss: 0.4146 time: 0.5412 s/iter data_time: 0.0529 s/iter total_throughput: 1892.09 samples/s lr: 6.39e-04 [09/20 03:32:22] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0154999 [09/20 03:32:23] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 03:32:23] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 03:32:27] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0540 s/iter. Inference: 0.2488 s/iter. Eval: 0.0033 s/iter. Total: 0.3061 s/iter. ETA=0:00:11 [09/20 03:32:32] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0652 s/iter. Inference: 0.2669 s/iter. Eval: 0.0035 s/iter. Total: 0.3357 s/iter. ETA=0:00:07 [09/20 03:32:38] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0688 s/iter. Inference: 0.2590 s/iter. Eval: 0.0035 s/iter. Total: 0.3315 s/iter. ETA=0:00:01 [09/20 03:32:40] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 03:32:40] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.258672 (0.000285 s / iter per device, on 8 devices) [09/20 03:32:40] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000227 s / iter per device, on 8 devices) [09/20 03:32:40] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 03:32:40] lb.evaluation.utils INFO: copypaste: Acc@1=73.24199999999999 [09/20 03:32:40] lb.evaluation.utils INFO: copypaste: Acc@5=91.61399999999999 [09/20 03:32:40] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 73.24200, not better than best score 73.39600 @ iteration 149999. [09/20 03:32:40] lb.utils.events INFO: eta: 1 day, 9:32:34 iteration: 154999/375342 consumed_samples: 158720000 total_loss: 0.4167 time: 0.5412 s/iter data_time: 0.0540 s/iter total_throughput: 1892.08 samples/s lr: 6.39e-04 [09/20 03:33:35] lb.utils.events INFO: eta: 1 day, 9:31:42 iteration: 155099/375342 consumed_samples: 158822400 total_loss: 0.4267 time: 0.5412 s/iter data_time: 0.0539 s/iter total_throughput: 1892.06 samples/s lr: 6.38e-04 [09/20 03:34:29] lb.utils.events INFO: eta: 1 day, 9:30:55 iteration: 155199/375342 consumed_samples: 158924800 total_loss: 0.4234 time: 0.5412 s/iter data_time: 0.0521 s/iter total_throughput: 1892.04 samples/s lr: 6.38e-04 [09/20 03:35:24] lb.utils.events INFO: eta: 1 day, 9:29:54 iteration: 155299/375342 consumed_samples: 159027200 total_loss: 0.418 time: 0.5412 s/iter data_time: 0.0573 s/iter total_throughput: 1892.02 samples/s lr: 6.37e-04 [09/20 03:36:19] lb.utils.events INFO: eta: 1 day, 9:28:51 iteration: 155399/375342 consumed_samples: 159129600 total_loss: 0.4109 time: 0.5412 s/iter data_time: 0.0548 s/iter total_throughput: 1892.01 samples/s lr: 6.37e-04 [09/20 03:37:14] lb.utils.events INFO: eta: 1 day, 9:27:39 iteration: 155499/375342 consumed_samples: 159232000 total_loss: 0.416 time: 0.5412 s/iter data_time: 0.0550 s/iter total_throughput: 1892.00 samples/s lr: 6.37e-04 [09/20 03:38:09] lb.utils.events INFO: eta: 1 day, 9:27:02 iteration: 155599/375342 consumed_samples: 159334400 total_loss: 0.4168 time: 0.5412 s/iter data_time: 0.0571 s/iter total_throughput: 1891.99 samples/s lr: 6.36e-04 [09/20 03:39:03] lb.utils.events INFO: eta: 1 day, 9:25:12 iteration: 155699/375342 consumed_samples: 159436800 total_loss: 0.4208 time: 0.5412 s/iter data_time: 0.0564 s/iter total_throughput: 1891.98 samples/s lr: 6.36e-04 [09/20 03:39:58] lb.utils.events INFO: eta: 1 day, 9:23:45 iteration: 155799/375342 consumed_samples: 159539200 total_loss: 0.4193 time: 0.5412 s/iter data_time: 0.0551 s/iter total_throughput: 1891.96 samples/s lr: 6.35e-04 [09/20 03:40:53] lb.utils.events INFO: eta: 1 day, 9:22:23 iteration: 155899/375342 consumed_samples: 159641600 total_loss: 0.4239 time: 0.5412 s/iter data_time: 0.0549 s/iter total_throughput: 1891.95 samples/s lr: 6.35e-04 [09/20 03:41:47] lb.utils.events INFO: eta: 1 day, 9:20:42 iteration: 155999/375342 consumed_samples: 159744000 total_loss: 0.4237 time: 0.5412 s/iter data_time: 0.0563 s/iter total_throughput: 1891.94 samples/s lr: 6.35e-04 [09/20 03:42:42] lb.utils.events INFO: eta: 1 day, 9:18:04 iteration: 156099/375342 consumed_samples: 159846400 total_loss: 0.4184 time: 0.5412 s/iter data_time: 0.0557 s/iter total_throughput: 1891.93 samples/s lr: 6.34e-04 [09/20 03:43:37] lb.utils.events INFO: eta: 1 day, 9:16:23 iteration: 156199/375342 consumed_samples: 159948800 total_loss: 0.4229 time: 0.5412 s/iter data_time: 0.0549 s/iter total_throughput: 1891.92 samples/s lr: 6.34e-04 [09/20 03:44:31] lb.utils.events INFO: eta: 1 day, 9:14:57 iteration: 156299/375342 consumed_samples: 160051200 total_loss: 0.4226 time: 0.5413 s/iter data_time: 0.0548 s/iter total_throughput: 1891.91 samples/s lr: 6.33e-04 [09/20 03:45:26] lb.utils.events INFO: eta: 1 day, 9:13:46 iteration: 156399/375342 consumed_samples: 160153600 total_loss: 0.4218 time: 0.5413 s/iter data_time: 0.0542 s/iter total_throughput: 1891.90 samples/s lr: 6.33e-04 [09/20 03:46:21] lb.utils.events INFO: eta: 1 day, 9:12:54 iteration: 156499/375342 consumed_samples: 160256000 total_loss: 0.4196 time: 0.5413 s/iter data_time: 0.0558 s/iter total_throughput: 1891.89 samples/s lr: 6.33e-04 [09/20 03:47:15] lb.utils.events INFO: eta: 1 day, 9:11:25 iteration: 156599/375342 consumed_samples: 160358400 total_loss: 0.4253 time: 0.5413 s/iter data_time: 0.0540 s/iter total_throughput: 1891.88 samples/s lr: 6.32e-04 [09/20 03:48:10] lb.utils.events INFO: eta: 1 day, 9:10:34 iteration: 156699/375342 consumed_samples: 160460800 total_loss: 0.4294 time: 0.5413 s/iter data_time: 0.0549 s/iter total_throughput: 1891.87 samples/s lr: 6.32e-04 [09/20 03:49:04] lb.utils.events INFO: eta: 1 day, 9:09:42 iteration: 156799/375342 consumed_samples: 160563200 total_loss: 0.423 time: 0.5413 s/iter data_time: 0.0547 s/iter total_throughput: 1891.86 samples/s lr: 6.31e-04 [09/20 03:49:59] lb.utils.events INFO: eta: 1 day, 9:08:48 iteration: 156899/375342 consumed_samples: 160665600 total_loss: 0.4235 time: 0.5413 s/iter data_time: 0.0539 s/iter total_throughput: 1891.84 samples/s lr: 6.31e-04 [09/20 03:50:54] lb.utils.events INFO: eta: 1 day, 9:09:06 iteration: 156999/375342 consumed_samples: 160768000 total_loss: 0.4221 time: 0.5413 s/iter data_time: 0.0525 s/iter total_throughput: 1891.82 samples/s lr: 6.31e-04 [09/20 03:51:49] lb.utils.events INFO: eta: 1 day, 9:09:06 iteration: 157099/375342 consumed_samples: 160870400 total_loss: 0.4175 time: 0.5413 s/iter data_time: 0.0526 s/iter total_throughput: 1891.81 samples/s lr: 6.30e-04 [09/20 03:52:44] lb.utils.events INFO: eta: 1 day, 9:09:05 iteration: 157199/375342 consumed_samples: 160972800 total_loss: 0.4216 time: 0.5413 s/iter data_time: 0.0522 s/iter total_throughput: 1891.79 samples/s lr: 6.30e-04 [09/20 03:53:39] lb.utils.events INFO: eta: 1 day, 9:09:15 iteration: 157299/375342 consumed_samples: 161075200 total_loss: 0.4259 time: 0.5413 s/iter data_time: 0.0554 s/iter total_throughput: 1891.77 samples/s lr: 6.29e-04 [09/20 03:54:34] lb.utils.events INFO: eta: 1 day, 9:09:02 iteration: 157399/375342 consumed_samples: 161177600 total_loss: 0.4189 time: 0.5413 s/iter data_time: 0.0518 s/iter total_throughput: 1891.76 samples/s lr: 6.29e-04 [09/20 03:55:29] lb.utils.events INFO: eta: 1 day, 9:08:59 iteration: 157499/375342 consumed_samples: 161280000 total_loss: 0.4173 time: 0.5413 s/iter data_time: 0.0545 s/iter total_throughput: 1891.74 samples/s lr: 6.29e-04 [09/20 03:56:24] lb.utils.events INFO: eta: 1 day, 9:08:56 iteration: 157599/375342 consumed_samples: 161382400 total_loss: 0.4218 time: 0.5413 s/iter data_time: 0.0535 s/iter total_throughput: 1891.72 samples/s lr: 6.28e-04 [09/20 03:57:18] lb.utils.events INFO: eta: 1 day, 9:09:23 iteration: 157699/375342 consumed_samples: 161484800 total_loss: 0.4218 time: 0.5413 s/iter data_time: 0.0522 s/iter total_throughput: 1891.71 samples/s lr: 6.28e-04 [09/20 03:58:13] lb.utils.events INFO: eta: 1 day, 9:09:14 iteration: 157799/375342 consumed_samples: 161587200 total_loss: 0.4169 time: 0.5413 s/iter data_time: 0.0534 s/iter total_throughput: 1891.69 samples/s lr: 6.27e-04 [09/20 03:59:08] lb.utils.events INFO: eta: 1 day, 9:09:35 iteration: 157899/375342 consumed_samples: 161689600 total_loss: 0.4144 time: 0.5413 s/iter data_time: 0.0512 s/iter total_throughput: 1891.67 samples/s lr: 6.27e-04 [09/20 04:00:03] lb.utils.events INFO: eta: 1 day, 9:08:40 iteration: 157999/375342 consumed_samples: 161792000 total_loss: 0.4184 time: 0.5413 s/iter data_time: 0.0540 s/iter total_throughput: 1891.65 samples/s lr: 6.27e-04 [09/20 04:00:58] lb.utils.events INFO: eta: 1 day, 9:07:41 iteration: 158099/375342 consumed_samples: 161894400 total_loss: 0.4265 time: 0.5413 s/iter data_time: 0.0533 s/iter total_throughput: 1891.64 samples/s lr: 6.26e-04 [09/20 04:01:53] lb.utils.events INFO: eta: 1 day, 9:05:59 iteration: 158199/375342 consumed_samples: 161996800 total_loss: 0.4241 time: 0.5413 s/iter data_time: 0.0531 s/iter total_throughput: 1891.63 samples/s lr: 6.26e-04 [09/20 04:02:48] lb.utils.events INFO: eta: 1 day, 9:04:06 iteration: 158299/375342 consumed_samples: 162099200 total_loss: 0.417 time: 0.5413 s/iter data_time: 0.0520 s/iter total_throughput: 1891.61 samples/s lr: 6.25e-04 [09/20 04:03:42] lb.utils.events INFO: eta: 1 day, 9:02:46 iteration: 158399/375342 consumed_samples: 162201600 total_loss: 0.4154 time: 0.5413 s/iter data_time: 0.0514 s/iter total_throughput: 1891.60 samples/s lr: 6.25e-04 [09/20 04:04:37] lb.utils.events INFO: eta: 1 day, 9:01:19 iteration: 158499/375342 consumed_samples: 162304000 total_loss: 0.4183 time: 0.5413 s/iter data_time: 0.0531 s/iter total_throughput: 1891.59 samples/s lr: 6.25e-04 [09/20 04:05:32] lb.utils.events INFO: eta: 1 day, 8:58:39 iteration: 158599/375342 consumed_samples: 162406400 total_loss: 0.4281 time: 0.5413 s/iter data_time: 0.0519 s/iter total_throughput: 1891.58 samples/s lr: 6.24e-04 [09/20 04:06:27] lb.utils.events INFO: eta: 1 day, 8:56:47 iteration: 158699/375342 consumed_samples: 162508800 total_loss: 0.4245 time: 0.5414 s/iter data_time: 0.0540 s/iter total_throughput: 1891.56 samples/s lr: 6.24e-04 [09/20 04:07:21] lb.utils.events INFO: eta: 1 day, 8:55:20 iteration: 158799/375342 consumed_samples: 162611200 total_loss: 0.4201 time: 0.5414 s/iter data_time: 0.0547 s/iter total_throughput: 1891.55 samples/s lr: 6.23e-04 [09/20 04:08:16] lb.utils.events INFO: eta: 1 day, 8:53:10 iteration: 158899/375342 consumed_samples: 162713600 total_loss: 0.417 time: 0.5414 s/iter data_time: 0.0564 s/iter total_throughput: 1891.54 samples/s lr: 6.23e-04 [09/20 04:09:10] lb.utils.events INFO: eta: 1 day, 8:51:20 iteration: 158999/375342 consumed_samples: 162816000 total_loss: 0.4287 time: 0.5414 s/iter data_time: 0.0548 s/iter total_throughput: 1891.53 samples/s lr: 6.23e-04 [09/20 04:10:05] lb.utils.events INFO: eta: 1 day, 8:49:14 iteration: 159099/375342 consumed_samples: 162918400 total_loss: 0.4339 time: 0.5414 s/iter data_time: 0.0546 s/iter total_throughput: 1891.52 samples/s lr: 6.22e-04 [09/20 04:11:00] lb.utils.events INFO: eta: 1 day, 8:48:19 iteration: 159199/375342 consumed_samples: 163020800 total_loss: 0.4231 time: 0.5414 s/iter data_time: 0.0565 s/iter total_throughput: 1891.51 samples/s lr: 6.22e-04 [09/20 04:11:54] lb.utils.events INFO: eta: 1 day, 8:47:26 iteration: 159299/375342 consumed_samples: 163123200 total_loss: 0.4163 time: 0.5414 s/iter data_time: 0.0554 s/iter total_throughput: 1891.50 samples/s lr: 6.21e-04 [09/20 04:12:49] lb.utils.events INFO: eta: 1 day, 8:46:18 iteration: 159399/375342 consumed_samples: 163225600 total_loss: 0.414 time: 0.5414 s/iter data_time: 0.0540 s/iter total_throughput: 1891.49 samples/s lr: 6.21e-04 [09/20 04:13:44] lb.utils.events INFO: eta: 1 day, 8:45:12 iteration: 159499/375342 consumed_samples: 163328000 total_loss: 0.4169 time: 0.5414 s/iter data_time: 0.0553 s/iter total_throughput: 1891.48 samples/s lr: 6.21e-04 [09/20 04:14:38] lb.utils.events INFO: eta: 1 day, 8:44:28 iteration: 159599/375342 consumed_samples: 163430400 total_loss: 0.423 time: 0.5414 s/iter data_time: 0.0542 s/iter total_throughput: 1891.47 samples/s lr: 6.20e-04 [09/20 04:15:33] lb.utils.events INFO: eta: 1 day, 8:43:40 iteration: 159699/375342 consumed_samples: 163532800 total_loss: 0.4226 time: 0.5414 s/iter data_time: 0.0547 s/iter total_throughput: 1891.46 samples/s lr: 6.20e-04 [09/20 04:16:28] lb.utils.events INFO: eta: 1 day, 8:42:45 iteration: 159799/375342 consumed_samples: 163635200 total_loss: 0.4209 time: 0.5414 s/iter data_time: 0.0552 s/iter total_throughput: 1891.44 samples/s lr: 6.19e-04 [09/20 04:17:23] lb.utils.events INFO: eta: 1 day, 8:42:32 iteration: 159899/375342 consumed_samples: 163737600 total_loss: 0.4228 time: 0.5414 s/iter data_time: 0.0562 s/iter total_throughput: 1891.43 samples/s lr: 6.19e-04 [09/20 04:18:17] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0159999 [09/20 04:18:18] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 04:18:18] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 04:18:23] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0646 s/iter. Inference: 0.2415 s/iter. Eval: 0.0020 s/iter. Total: 0.3080 s/iter. ETA=0:00:11 [09/20 04:18:28] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0784 s/iter. Inference: 0.2553 s/iter. Eval: 0.0021 s/iter. Total: 0.3360 s/iter. ETA=0:00:07 [09/20 04:18:33] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0767 s/iter. Inference: 0.2525 s/iter. Eval: 0.0022 s/iter. Total: 0.3317 s/iter. ETA=0:00:01 [09/20 04:18:35] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 04:18:35] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.310024 (0.000286 s / iter per device, on 8 devices) [09/20 04:18:35] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000221 s / iter per device, on 8 devices) [09/20 04:18:35] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 04:18:35] lb.evaluation.utils INFO: copypaste: Acc@1=73.368 [09/20 04:18:35] lb.evaluation.utils INFO: copypaste: Acc@5=91.702 [09/20 04:18:35] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 73.36800, not better than best score 73.39600 @ iteration 149999. [09/20 04:18:35] lb.utils.events INFO: eta: 1 day, 8:42:30 iteration: 159999/375342 consumed_samples: 163840000 total_loss: 0.4227 time: 0.5414 s/iter data_time: 0.0540 s/iter total_throughput: 1891.42 samples/s lr: 6.19e-04 [09/20 04:19:30] lb.utils.events INFO: eta: 1 day, 8:43:00 iteration: 160099/375342 consumed_samples: 163942400 total_loss: 0.4194 time: 0.5414 s/iter data_time: 0.0578 s/iter total_throughput: 1891.40 samples/s lr: 6.18e-04 [09/20 04:20:25] lb.utils.events INFO: eta: 1 day, 8:42:41 iteration: 160199/375342 consumed_samples: 164044800 total_loss: 0.4104 time: 0.5414 s/iter data_time: 0.0511 s/iter total_throughput: 1891.39 samples/s lr: 6.18e-04 [09/20 04:21:20] lb.utils.events INFO: eta: 1 day, 8:42:40 iteration: 160299/375342 consumed_samples: 164147200 total_loss: 0.4119 time: 0.5414 s/iter data_time: 0.0517 s/iter total_throughput: 1891.37 samples/s lr: 6.17e-04 [09/20 04:22:15] lb.utils.events INFO: eta: 1 day, 8:43:32 iteration: 160399/375342 consumed_samples: 164249600 total_loss: 0.4169 time: 0.5414 s/iter data_time: 0.0535 s/iter total_throughput: 1891.35 samples/s lr: 6.17e-04 [09/20 04:23:10] lb.utils.events INFO: eta: 1 day, 8:43:16 iteration: 160499/375342 consumed_samples: 164352000 total_loss: 0.4274 time: 0.5414 s/iter data_time: 0.0530 s/iter total_throughput: 1891.33 samples/s lr: 6.17e-04 [09/20 04:24:05] lb.utils.events INFO: eta: 1 day, 8:43:49 iteration: 160599/375342 consumed_samples: 164454400 total_loss: 0.4283 time: 0.5414 s/iter data_time: 0.0530 s/iter total_throughput: 1891.31 samples/s lr: 6.16e-04 [09/20 04:25:00] lb.utils.events INFO: eta: 1 day, 8:43:14 iteration: 160699/375342 consumed_samples: 164556800 total_loss: 0.4169 time: 0.5414 s/iter data_time: 0.0521 s/iter total_throughput: 1891.30 samples/s lr: 6.16e-04 [09/20 04:25:55] lb.utils.events INFO: eta: 1 day, 8:42:53 iteration: 160799/375342 consumed_samples: 164659200 total_loss: 0.4131 time: 0.5414 s/iter data_time: 0.0515 s/iter total_throughput: 1891.28 samples/s lr: 6.15e-04 [09/20 04:26:50] lb.utils.events INFO: eta: 1 day, 8:42:36 iteration: 160899/375342 consumed_samples: 164761600 total_loss: 0.4107 time: 0.5414 s/iter data_time: 0.0521 s/iter total_throughput: 1891.27 samples/s lr: 6.15e-04 [09/20 04:27:45] lb.utils.events INFO: eta: 1 day, 8:42:23 iteration: 160999/375342 consumed_samples: 164864000 total_loss: 0.4177 time: 0.5414 s/iter data_time: 0.0523 s/iter total_throughput: 1891.25 samples/s lr: 6.15e-04 [09/20 04:28:39] lb.utils.events INFO: eta: 1 day, 8:40:11 iteration: 161099/375342 consumed_samples: 164966400 total_loss: 0.4219 time: 0.5414 s/iter data_time: 0.0505 s/iter total_throughput: 1891.24 samples/s lr: 6.14e-04 [09/20 04:29:34] lb.utils.events INFO: eta: 1 day, 8:38:52 iteration: 161199/375342 consumed_samples: 165068800 total_loss: 0.42 time: 0.5414 s/iter data_time: 0.0529 s/iter total_throughput: 1891.22 samples/s lr: 6.14e-04 [09/20 04:30:29] lb.utils.events INFO: eta: 1 day, 8:37:47 iteration: 161299/375342 consumed_samples: 165171200 total_loss: 0.4245 time: 0.5415 s/iter data_time: 0.0511 s/iter total_throughput: 1891.21 samples/s lr: 6.13e-04 [09/20 04:31:24] lb.utils.events INFO: eta: 1 day, 8:36:13 iteration: 161399/375342 consumed_samples: 165273600 total_loss: 0.4253 time: 0.5415 s/iter data_time: 0.0519 s/iter total_throughput: 1891.20 samples/s lr: 6.13e-04 [09/20 04:32:18] lb.utils.events INFO: eta: 1 day, 8:34:31 iteration: 161499/375342 consumed_samples: 165376000 total_loss: 0.4241 time: 0.5415 s/iter data_time: 0.0518 s/iter total_throughput: 1891.18 samples/s lr: 6.13e-04 [09/20 04:33:13] lb.utils.events INFO: eta: 1 day, 8:32:18 iteration: 161599/375342 consumed_samples: 165478400 total_loss: 0.423 time: 0.5415 s/iter data_time: 0.0516 s/iter total_throughput: 1891.17 samples/s lr: 6.12e-04 [09/20 04:34:08] lb.utils.events INFO: eta: 1 day, 8:29:55 iteration: 161699/375342 consumed_samples: 165580800 total_loss: 0.4198 time: 0.5415 s/iter data_time: 0.0516 s/iter total_throughput: 1891.16 samples/s lr: 6.12e-04 [09/20 04:35:02] lb.utils.events INFO: eta: 1 day, 8:27:30 iteration: 161799/375342 consumed_samples: 165683200 total_loss: 0.4169 time: 0.5415 s/iter data_time: 0.0512 s/iter total_throughput: 1891.15 samples/s lr: 6.11e-04 [09/20 04:35:57] lb.utils.events INFO: eta: 1 day, 8:25:27 iteration: 161899/375342 consumed_samples: 165785600 total_loss: 0.4165 time: 0.5415 s/iter data_time: 0.0516 s/iter total_throughput: 1891.15 samples/s lr: 6.11e-04 [09/20 04:36:51] lb.utils.events INFO: eta: 1 day, 8:23:11 iteration: 161999/375342 consumed_samples: 165888000 total_loss: 0.4174 time: 0.5415 s/iter data_time: 0.0534 s/iter total_throughput: 1891.14 samples/s lr: 6.11e-04 [09/20 04:37:46] lb.utils.events INFO: eta: 1 day, 8:21:35 iteration: 162099/375342 consumed_samples: 165990400 total_loss: 0.4252 time: 0.5415 s/iter data_time: 0.0540 s/iter total_throughput: 1891.13 samples/s lr: 6.10e-04 [09/20 04:38:41] lb.utils.events INFO: eta: 1 day, 8:19:56 iteration: 162199/375342 consumed_samples: 166092800 total_loss: 0.426 time: 0.5415 s/iter data_time: 0.0553 s/iter total_throughput: 1891.11 samples/s lr: 6.10e-04 [09/20 04:39:36] lb.utils.events INFO: eta: 1 day, 8:18:23 iteration: 162299/375342 consumed_samples: 166195200 total_loss: 0.4192 time: 0.5415 s/iter data_time: 0.0566 s/iter total_throughput: 1891.10 samples/s lr: 6.09e-04 [09/20 04:40:30] lb.utils.events INFO: eta: 1 day, 8:17:08 iteration: 162399/375342 consumed_samples: 166297600 total_loss: 0.4177 time: 0.5415 s/iter data_time: 0.0551 s/iter total_throughput: 1891.09 samples/s lr: 6.09e-04 [09/20 04:41:25] lb.utils.events INFO: eta: 1 day, 8:16:06 iteration: 162499/375342 consumed_samples: 166400000 total_loss: 0.4206 time: 0.5415 s/iter data_time: 0.0554 s/iter total_throughput: 1891.08 samples/s lr: 6.09e-04 [09/20 04:42:20] lb.utils.events INFO: eta: 1 day, 8:15:23 iteration: 162599/375342 consumed_samples: 166502400 total_loss: 0.4229 time: 0.5415 s/iter data_time: 0.0539 s/iter total_throughput: 1891.07 samples/s lr: 6.08e-04 [09/20 04:43:14] lb.utils.events INFO: eta: 1 day, 8:14:57 iteration: 162699/375342 consumed_samples: 166604800 total_loss: 0.4242 time: 0.5415 s/iter data_time: 0.0555 s/iter total_throughput: 1891.06 samples/s lr: 6.08e-04 [09/20 04:44:09] lb.utils.events INFO: eta: 1 day, 8:15:05 iteration: 162799/375342 consumed_samples: 166707200 total_loss: 0.4199 time: 0.5415 s/iter data_time: 0.0568 s/iter total_throughput: 1891.05 samples/s lr: 6.07e-04 [09/20 04:45:04] lb.utils.events INFO: eta: 1 day, 8:15:14 iteration: 162899/375342 consumed_samples: 166809600 total_loss: 0.4179 time: 0.5415 s/iter data_time: 0.0547 s/iter total_throughput: 1891.03 samples/s lr: 6.07e-04 [09/20 04:45:59] lb.utils.events INFO: eta: 1 day, 8:15:07 iteration: 162999/375342 consumed_samples: 166912000 total_loss: 0.4175 time: 0.5415 s/iter data_time: 0.0562 s/iter total_throughput: 1891.02 samples/s lr: 6.06e-04 [09/20 04:46:54] lb.utils.events INFO: eta: 1 day, 8:15:43 iteration: 163099/375342 consumed_samples: 167014400 total_loss: 0.4091 time: 0.5415 s/iter data_time: 0.0567 s/iter total_throughput: 1891.01 samples/s lr: 6.06e-04 [09/20 04:47:48] lb.utils.events INFO: eta: 1 day, 8:15:09 iteration: 163199/375342 consumed_samples: 167116800 total_loss: 0.4132 time: 0.5415 s/iter data_time: 0.0550 s/iter total_throughput: 1890.99 samples/s lr: 6.06e-04 [09/20 04:48:43] lb.utils.events INFO: eta: 1 day, 8:14:27 iteration: 163299/375342 consumed_samples: 167219200 total_loss: 0.4198 time: 0.5415 s/iter data_time: 0.0561 s/iter total_throughput: 1890.98 samples/s lr: 6.05e-04 [09/20 04:49:38] lb.utils.events INFO: eta: 1 day, 8:13:47 iteration: 163399/375342 consumed_samples: 167321600 total_loss: 0.4124 time: 0.5415 s/iter data_time: 0.0541 s/iter total_throughput: 1890.97 samples/s lr: 6.05e-04 [09/20 04:50:33] lb.utils.events INFO: eta: 1 day, 8:13:02 iteration: 163499/375342 consumed_samples: 167424000 total_loss: 0.4207 time: 0.5415 s/iter data_time: 0.0530 s/iter total_throughput: 1890.96 samples/s lr: 6.04e-04 [09/20 04:51:27] lb.utils.events INFO: eta: 1 day, 8:12:42 iteration: 163599/375342 consumed_samples: 167526400 total_loss: 0.4252 time: 0.5415 s/iter data_time: 0.0508 s/iter total_throughput: 1890.94 samples/s lr: 6.04e-04 [09/20 04:52:22] lb.utils.events INFO: eta: 1 day, 8:12:25 iteration: 163699/375342 consumed_samples: 167628800 total_loss: 0.4274 time: 0.5415 s/iter data_time: 0.0516 s/iter total_throughput: 1890.93 samples/s lr: 6.04e-04 [09/20 04:53:17] lb.utils.events INFO: eta: 1 day, 8:12:41 iteration: 163799/375342 consumed_samples: 167731200 total_loss: 0.431 time: 0.5415 s/iter data_time: 0.0519 s/iter total_throughput: 1890.91 samples/s lr: 6.03e-04 [09/20 04:54:12] lb.utils.events INFO: eta: 1 day, 8:11:56 iteration: 163899/375342 consumed_samples: 167833600 total_loss: 0.4228 time: 0.5415 s/iter data_time: 0.0526 s/iter total_throughput: 1890.90 samples/s lr: 6.03e-04 [09/20 04:55:07] lb.utils.events INFO: eta: 1 day, 8:10:51 iteration: 163999/375342 consumed_samples: 167936000 total_loss: 0.4163 time: 0.5415 s/iter data_time: 0.0506 s/iter total_throughput: 1890.88 samples/s lr: 6.02e-04 [09/20 04:56:02] lb.utils.events INFO: eta: 1 day, 8:09:11 iteration: 164099/375342 consumed_samples: 168038400 total_loss: 0.4215 time: 0.5415 s/iter data_time: 0.0499 s/iter total_throughput: 1890.87 samples/s lr: 6.02e-04 [09/20 04:56:57] lb.utils.events INFO: eta: 1 day, 8:09:09 iteration: 164199/375342 consumed_samples: 168140800 total_loss: 0.4237 time: 0.5416 s/iter data_time: 0.0498 s/iter total_throughput: 1890.86 samples/s lr: 6.02e-04 [09/20 04:57:51] lb.utils.events INFO: eta: 1 day, 8:07:44 iteration: 164299/375342 consumed_samples: 168243200 total_loss: 0.4196 time: 0.5416 s/iter data_time: 0.0505 s/iter total_throughput: 1890.84 samples/s lr: 6.01e-04 [09/20 04:58:46] lb.utils.events INFO: eta: 1 day, 8:06:39 iteration: 164399/375342 consumed_samples: 168345600 total_loss: 0.4166 time: 0.5416 s/iter data_time: 0.0515 s/iter total_throughput: 1890.83 samples/s lr: 6.01e-04 [09/20 04:59:41] lb.utils.events INFO: eta: 1 day, 8:05:39 iteration: 164499/375342 consumed_samples: 168448000 total_loss: 0.4167 time: 0.5416 s/iter data_time: 0.0521 s/iter total_throughput: 1890.82 samples/s lr: 6.00e-04 [09/20 05:00:35] lb.utils.events INFO: eta: 1 day, 8:04:08 iteration: 164599/375342 consumed_samples: 168550400 total_loss: 0.4204 time: 0.5416 s/iter data_time: 0.0518 s/iter total_throughput: 1890.81 samples/s lr: 6.00e-04 [09/20 05:01:30] lb.utils.events INFO: eta: 1 day, 8:02:26 iteration: 164699/375342 consumed_samples: 168652800 total_loss: 0.4199 time: 0.5416 s/iter data_time: 0.0514 s/iter total_throughput: 1890.80 samples/s lr: 6.00e-04 [09/20 05:02:25] lb.utils.events INFO: eta: 1 day, 8:00:04 iteration: 164799/375342 consumed_samples: 168755200 total_loss: 0.4169 time: 0.5416 s/iter data_time: 0.0524 s/iter total_throughput: 1890.79 samples/s lr: 5.99e-04 [09/20 05:03:19] lb.utils.events INFO: eta: 1 day, 7:58:08 iteration: 164899/375342 consumed_samples: 168857600 total_loss: 0.4174 time: 0.5416 s/iter data_time: 0.0519 s/iter total_throughput: 1890.78 samples/s lr: 5.99e-04 [09/20 05:04:14] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0164999 [09/20 05:04:14] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 05:04:14] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 05:04:19] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0499 s/iter. Inference: 0.2466 s/iter. Eval: 0.0031 s/iter. Total: 0.2995 s/iter. ETA=0:00:11 [09/20 05:04:24] lb.evaluation.evaluator INFO: Inference done 25600/50000. Dataloading: 0.0696 s/iter. Inference: 0.2703 s/iter. Eval: 0.0029 s/iter. Total: 0.3431 s/iter. ETA=0:00:07 [09/20 05:04:29] lb.evaluation.evaluator INFO: Inference done 41984/50000. Dataloading: 0.0694 s/iter. Inference: 0.2614 s/iter. Eval: 0.0026 s/iter. Total: 0.3337 s/iter. ETA=0:00:02 [09/20 05:04:32] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 05:04:32] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.304381 (0.000286 s / iter per device, on 8 devices) [09/20 05:04:32] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000227 s / iter per device, on 8 devices) [09/20 05:04:32] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 05:04:32] lb.evaluation.utils INFO: copypaste: Acc@1=73.60600000000001 [09/20 05:04:32] lb.evaluation.utils INFO: copypaste: Acc@5=91.81 [09/20 05:04:32] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 73.60600, better than last best score 73.39600 @ iteration 149999. [09/20 05:04:32] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 05:04:32] lb.utils.events INFO: eta: 1 day, 7:56:24 iteration: 164999/375342 consumed_samples: 168960000 total_loss: 0.4224 time: 0.5416 s/iter data_time: 0.0510 s/iter total_throughput: 1890.77 samples/s lr: 5.98e-04 [09/20 05:05:27] lb.utils.events INFO: eta: 1 day, 7:54:54 iteration: 165099/375342 consumed_samples: 169062400 total_loss: 0.4233 time: 0.5416 s/iter data_time: 0.0515 s/iter total_throughput: 1890.77 samples/s lr: 5.98e-04 [09/20 05:06:21] lb.utils.events INFO: eta: 1 day, 7:53:03 iteration: 165199/375342 consumed_samples: 169164800 total_loss: 0.4198 time: 0.5416 s/iter data_time: 0.0498 s/iter total_throughput: 1890.76 samples/s lr: 5.98e-04 [09/20 05:07:16] lb.utils.events INFO: eta: 1 day, 7:52:05 iteration: 165299/375342 consumed_samples: 169267200 total_loss: 0.4197 time: 0.5416 s/iter data_time: 0.0510 s/iter total_throughput: 1890.75 samples/s lr: 5.97e-04 [09/20 05:08:10] lb.utils.events INFO: eta: 1 day, 7:50:29 iteration: 165399/375342 consumed_samples: 169369600 total_loss: 0.4188 time: 0.5416 s/iter data_time: 0.0483 s/iter total_throughput: 1890.75 samples/s lr: 5.97e-04 [09/20 05:09:05] lb.utils.events INFO: eta: 1 day, 7:48:45 iteration: 165499/375342 consumed_samples: 169472000 total_loss: 0.4185 time: 0.5416 s/iter data_time: 0.0515 s/iter total_throughput: 1890.74 samples/s lr: 5.96e-04 [09/20 05:10:00] lb.utils.events INFO: eta: 1 day, 7:47:17 iteration: 165599/375342 consumed_samples: 169574400 total_loss: 0.415 time: 0.5416 s/iter data_time: 0.0572 s/iter total_throughput: 1890.72 samples/s lr: 5.96e-04 [09/20 05:10:54] lb.utils.events INFO: eta: 1 day, 7:45:06 iteration: 165699/375342 consumed_samples: 169676800 total_loss: 0.4136 time: 0.5416 s/iter data_time: 0.0549 s/iter total_throughput: 1890.71 samples/s lr: 5.96e-04 [09/20 05:11:49] lb.utils.events INFO: eta: 1 day, 7:44:09 iteration: 165799/375342 consumed_samples: 169779200 total_loss: 0.4231 time: 0.5416 s/iter data_time: 0.0549 s/iter total_throughput: 1890.70 samples/s lr: 5.95e-04 [09/20 05:12:44] lb.utils.events INFO: eta: 1 day, 7:43:15 iteration: 165899/375342 consumed_samples: 169881600 total_loss: 0.4189 time: 0.5416 s/iter data_time: 0.0545 s/iter total_throughput: 1890.70 samples/s lr: 5.95e-04 [09/20 05:13:38] lb.utils.events INFO: eta: 1 day, 7:43:22 iteration: 165999/375342 consumed_samples: 169984000 total_loss: 0.4083 time: 0.5416 s/iter data_time: 0.0543 s/iter total_throughput: 1890.69 samples/s lr: 5.94e-04 [09/20 05:14:33] lb.utils.events INFO: eta: 1 day, 7:42:53 iteration: 166099/375342 consumed_samples: 170086400 total_loss: 0.4114 time: 0.5416 s/iter data_time: 0.0551 s/iter total_throughput: 1890.68 samples/s lr: 5.94e-04 [09/20 05:15:28] lb.utils.events INFO: eta: 1 day, 7:42:29 iteration: 166199/375342 consumed_samples: 170188800 total_loss: 0.4149 time: 0.5416 s/iter data_time: 0.0553 s/iter total_throughput: 1890.67 samples/s lr: 5.93e-04 [09/20 05:16:22] lb.utils.events INFO: eta: 1 day, 7:41:57 iteration: 166299/375342 consumed_samples: 170291200 total_loss: 0.417 time: 0.5416 s/iter data_time: 0.0560 s/iter total_throughput: 1890.66 samples/s lr: 5.93e-04 [09/20 05:17:17] lb.utils.events INFO: eta: 1 day, 7:41:59 iteration: 166399/375342 consumed_samples: 170393600 total_loss: 0.4169 time: 0.5416 s/iter data_time: 0.0562 s/iter total_throughput: 1890.64 samples/s lr: 5.93e-04 [09/20 05:18:12] lb.utils.events INFO: eta: 1 day, 7:42:26 iteration: 166499/375342 consumed_samples: 170496000 total_loss: 0.4124 time: 0.5416 s/iter data_time: 0.0510 s/iter total_throughput: 1890.63 samples/s lr: 5.92e-04 [09/20 05:19:07] lb.utils.events INFO: eta: 1 day, 7:41:38 iteration: 166599/375342 consumed_samples: 170598400 total_loss: 0.4249 time: 0.5416 s/iter data_time: 0.0546 s/iter total_throughput: 1890.62 samples/s lr: 5.92e-04 [09/20 05:20:01] lb.utils.events INFO: eta: 1 day, 7:41:23 iteration: 166699/375342 consumed_samples: 170700800 total_loss: 0.4268 time: 0.5416 s/iter data_time: 0.0539 s/iter total_throughput: 1890.61 samples/s lr: 5.91e-04 [09/20 05:20:56] lb.utils.events INFO: eta: 1 day, 7:41:29 iteration: 166799/375342 consumed_samples: 170803200 total_loss: 0.4168 time: 0.5416 s/iter data_time: 0.0539 s/iter total_throughput: 1890.59 samples/s lr: 5.91e-04 [09/20 05:21:51] lb.utils.events INFO: eta: 1 day, 7:40:48 iteration: 166899/375342 consumed_samples: 170905600 total_loss: 0.4164 time: 0.5416 s/iter data_time: 0.0547 s/iter total_throughput: 1890.58 samples/s lr: 5.91e-04 [09/20 05:22:46] lb.utils.events INFO: eta: 1 day, 7:40:22 iteration: 166999/375342 consumed_samples: 171008000 total_loss: 0.4207 time: 0.5416 s/iter data_time: 0.0517 s/iter total_throughput: 1890.57 samples/s lr: 5.90e-04 [09/20 05:23:40] lb.utils.events INFO: eta: 1 day, 7:39:51 iteration: 167099/375342 consumed_samples: 171110400 total_loss: 0.4114 time: 0.5416 s/iter data_time: 0.0524 s/iter total_throughput: 1890.56 samples/s lr: 5.90e-04 [09/20 05:24:35] lb.utils.events INFO: eta: 1 day, 7:39:19 iteration: 167199/375342 consumed_samples: 171212800 total_loss: 0.4031 time: 0.5416 s/iter data_time: 0.0514 s/iter total_throughput: 1890.55 samples/s lr: 5.89e-04 [09/20 05:25:30] lb.utils.events INFO: eta: 1 day, 7:38:36 iteration: 167299/375342 consumed_samples: 171315200 total_loss: 0.4088 time: 0.5416 s/iter data_time: 0.0525 s/iter total_throughput: 1890.53 samples/s lr: 5.89e-04 [09/20 05:26:25] lb.utils.events INFO: eta: 1 day, 7:37:06 iteration: 167399/375342 consumed_samples: 171417600 total_loss: 0.4195 time: 0.5416 s/iter data_time: 0.0497 s/iter total_throughput: 1890.52 samples/s lr: 5.89e-04 [09/20 05:27:19] lb.utils.events INFO: eta: 1 day, 7:36:11 iteration: 167499/375342 consumed_samples: 171520000 total_loss: 0.4155 time: 0.5417 s/iter data_time: 0.0507 s/iter total_throughput: 1890.51 samples/s lr: 5.88e-04 [09/20 05:28:14] lb.utils.events INFO: eta: 1 day, 7:35:38 iteration: 167599/375342 consumed_samples: 171622400 total_loss: 0.4134 time: 0.5417 s/iter data_time: 0.0499 s/iter total_throughput: 1890.50 samples/s lr: 5.88e-04 [09/20 05:29:09] lb.utils.events INFO: eta: 1 day, 7:34:32 iteration: 167699/375342 consumed_samples: 171724800 total_loss: 0.4182 time: 0.5417 s/iter data_time: 0.0494 s/iter total_throughput: 1890.49 samples/s lr: 5.87e-04 [09/20 05:30:03] lb.utils.events INFO: eta: 1 day, 7:32:51 iteration: 167799/375342 consumed_samples: 171827200 total_loss: 0.4181 time: 0.5417 s/iter data_time: 0.0505 s/iter total_throughput: 1890.48 samples/s lr: 5.87e-04 [09/20 05:30:58] lb.utils.events INFO: eta: 1 day, 7:31:57 iteration: 167899/375342 consumed_samples: 171929600 total_loss: 0.414 time: 0.5417 s/iter data_time: 0.0513 s/iter total_throughput: 1890.47 samples/s lr: 5.87e-04 [09/20 05:31:53] lb.utils.events INFO: eta: 1 day, 7:30:10 iteration: 167999/375342 consumed_samples: 172032000 total_loss: 0.4056 time: 0.5417 s/iter data_time: 0.0526 s/iter total_throughput: 1890.46 samples/s lr: 5.86e-04 [09/20 05:32:47] lb.utils.events INFO: eta: 1 day, 7:28:06 iteration: 168099/375342 consumed_samples: 172134400 total_loss: 0.3985 time: 0.5417 s/iter data_time: 0.0512 s/iter total_throughput: 1890.46 samples/s lr: 5.86e-04 [09/20 05:33:42] lb.utils.events INFO: eta: 1 day, 7:26:22 iteration: 168199/375342 consumed_samples: 172236800 total_loss: 0.413 time: 0.5417 s/iter data_time: 0.0512 s/iter total_throughput: 1890.45 samples/s lr: 5.85e-04 [09/20 05:34:36] lb.utils.events INFO: eta: 1 day, 7:23:56 iteration: 168299/375342 consumed_samples: 172339200 total_loss: 0.4228 time: 0.5417 s/iter data_time: 0.0518 s/iter total_throughput: 1890.44 samples/s lr: 5.85e-04 [09/20 05:35:31] lb.utils.events INFO: eta: 1 day, 7:22:04 iteration: 168399/375342 consumed_samples: 172441600 total_loss: 0.422 time: 0.5417 s/iter data_time: 0.0515 s/iter total_throughput: 1890.44 samples/s lr: 5.85e-04 [09/20 05:36:25] lb.utils.events INFO: eta: 1 day, 7:20:10 iteration: 168499/375342 consumed_samples: 172544000 total_loss: 0.4136 time: 0.5417 s/iter data_time: 0.0505 s/iter total_throughput: 1890.43 samples/s lr: 5.84e-04 [09/20 05:37:20] lb.utils.events INFO: eta: 1 day, 7:18:11 iteration: 168599/375342 consumed_samples: 172646400 total_loss: 0.4089 time: 0.5417 s/iter data_time: 0.0489 s/iter total_throughput: 1890.43 samples/s lr: 5.84e-04 [09/20 05:38:14] lb.utils.events INFO: eta: 1 day, 7:16:46 iteration: 168699/375342 consumed_samples: 172748800 total_loss: 0.4027 time: 0.5417 s/iter data_time: 0.0494 s/iter total_throughput: 1890.42 samples/s lr: 5.83e-04 [09/20 05:39:09] lb.utils.events INFO: eta: 1 day, 7:15:41 iteration: 168799/375342 consumed_samples: 172851200 total_loss: 0.4081 time: 0.5417 s/iter data_time: 0.0516 s/iter total_throughput: 1890.41 samples/s lr: 5.83e-04 [09/20 05:40:03] lb.utils.events INFO: eta: 1 day, 7:14:00 iteration: 168899/375342 consumed_samples: 172953600 total_loss: 0.415 time: 0.5417 s/iter data_time: 0.0506 s/iter total_throughput: 1890.40 samples/s lr: 5.82e-04 [09/20 05:40:58] lb.utils.events INFO: eta: 1 day, 7:12:43 iteration: 168999/375342 consumed_samples: 173056000 total_loss: 0.4106 time: 0.5417 s/iter data_time: 0.0514 s/iter total_throughput: 1890.40 samples/s lr: 5.82e-04 [09/20 05:41:53] lb.utils.events INFO: eta: 1 day, 7:11:48 iteration: 169099/375342 consumed_samples: 173158400 total_loss: 0.4103 time: 0.5417 s/iter data_time: 0.0526 s/iter total_throughput: 1890.38 samples/s lr: 5.82e-04 [09/20 05:42:47] lb.utils.events INFO: eta: 1 day, 7:11:29 iteration: 169199/375342 consumed_samples: 173260800 total_loss: 0.4192 time: 0.5417 s/iter data_time: 0.0544 s/iter total_throughput: 1890.37 samples/s lr: 5.81e-04 [09/20 05:43:42] lb.utils.events INFO: eta: 1 day, 7:11:15 iteration: 169299/375342 consumed_samples: 173363200 total_loss: 0.425 time: 0.5417 s/iter data_time: 0.0550 s/iter total_throughput: 1890.36 samples/s lr: 5.81e-04 [09/20 05:44:37] lb.utils.events INFO: eta: 1 day, 7:11:37 iteration: 169399/375342 consumed_samples: 173465600 total_loss: 0.417 time: 0.5417 s/iter data_time: 0.0557 s/iter total_throughput: 1890.35 samples/s lr: 5.80e-04 [09/20 05:45:31] lb.utils.events INFO: eta: 1 day, 7:11:35 iteration: 169499/375342 consumed_samples: 173568000 total_loss: 0.413 time: 0.5417 s/iter data_time: 0.0559 s/iter total_throughput: 1890.34 samples/s lr: 5.80e-04 [09/20 05:46:26] lb.utils.events INFO: eta: 1 day, 7:12:03 iteration: 169599/375342 consumed_samples: 173670400 total_loss: 0.4161 time: 0.5417 s/iter data_time: 0.0563 s/iter total_throughput: 1890.33 samples/s lr: 5.80e-04 [09/20 05:47:21] lb.utils.events INFO: eta: 1 day, 7:11:57 iteration: 169699/375342 consumed_samples: 173772800 total_loss: 0.4172 time: 0.5417 s/iter data_time: 0.0560 s/iter total_throughput: 1890.32 samples/s lr: 5.79e-04 [09/20 05:48:16] lb.utils.events INFO: eta: 1 day, 7:11:57 iteration: 169799/375342 consumed_samples: 173875200 total_loss: 0.4193 time: 0.5417 s/iter data_time: 0.0545 s/iter total_throughput: 1890.31 samples/s lr: 5.79e-04 [09/20 05:49:10] lb.utils.events INFO: eta: 1 day, 7:12:04 iteration: 169899/375342 consumed_samples: 173977600 total_loss: 0.4167 time: 0.5417 s/iter data_time: 0.0535 s/iter total_throughput: 1890.30 samples/s lr: 5.78e-04 [09/20 05:50:05] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0169999 [09/20 05:50:06] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 05:50:06] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 05:50:10] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0561 s/iter. Inference: 0.2458 s/iter. Eval: 0.0023 s/iter. Total: 0.3043 s/iter. ETA=0:00:11 [09/20 05:50:15] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0534 s/iter. Inference: 0.2798 s/iter. Eval: 0.0023 s/iter. Total: 0.3357 s/iter. ETA=0:00:07 [09/20 05:50:21] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0474 s/iter. Inference: 0.2808 s/iter. Eval: 0.0023 s/iter. Total: 0.3308 s/iter. ETA=0:00:01 [09/20 05:50:23] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 05:50:23] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.291360 (0.000286 s / iter per device, on 8 devices) [09/20 05:50:23] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:12 (0.000248 s / iter per device, on 8 devices) [09/20 05:50:23] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 05:50:23] lb.evaluation.utils INFO: copypaste: Acc@1=74.152 [09/20 05:50:23] lb.evaluation.utils INFO: copypaste: Acc@5=92.07 [09/20 05:50:23] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 74.15200, better than last best score 73.60600 @ iteration 164999. [09/20 05:50:23] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 05:50:23] lb.utils.events INFO: eta: 1 day, 7:11:24 iteration: 169999/375342 consumed_samples: 174080000 total_loss: 0.4182 time: 0.5417 s/iter data_time: 0.0539 s/iter total_throughput: 1890.29 samples/s lr: 5.78e-04 [09/20 05:51:18] lb.utils.events INFO: eta: 1 day, 7:10:59 iteration: 170099/375342 consumed_samples: 174182400 total_loss: 0.4139 time: 0.5417 s/iter data_time: 0.0560 s/iter total_throughput: 1890.28 samples/s lr: 5.78e-04 [09/20 05:52:13] lb.utils.events INFO: eta: 1 day, 7:10:15 iteration: 170199/375342 consumed_samples: 174284800 total_loss: 0.4072 time: 0.5417 s/iter data_time: 0.0556 s/iter total_throughput: 1890.27 samples/s lr: 5.77e-04 [09/20 05:53:08] lb.utils.events INFO: eta: 1 day, 7:09:21 iteration: 170299/375342 consumed_samples: 174387200 total_loss: 0.4112 time: 0.5417 s/iter data_time: 0.0552 s/iter total_throughput: 1890.26 samples/s lr: 5.77e-04 [09/20 05:54:02] lb.utils.events INFO: eta: 1 day, 7:08:46 iteration: 170399/375342 consumed_samples: 174489600 total_loss: 0.4125 time: 0.5417 s/iter data_time: 0.0511 s/iter total_throughput: 1890.24 samples/s lr: 5.76e-04 [09/20 05:54:57] lb.utils.events INFO: eta: 1 day, 7:08:07 iteration: 170499/375342 consumed_samples: 174592000 total_loss: 0.4216 time: 0.5417 s/iter data_time: 0.0482 s/iter total_throughput: 1890.23 samples/s lr: 5.76e-04 [09/20 05:55:52] lb.utils.events INFO: eta: 1 day, 7:07:35 iteration: 170599/375342 consumed_samples: 174694400 total_loss: 0.4203 time: 0.5417 s/iter data_time: 0.0474 s/iter total_throughput: 1890.22 samples/s lr: 5.75e-04 [09/20 05:56:46] lb.utils.events INFO: eta: 1 day, 7:06:31 iteration: 170699/375342 consumed_samples: 174796800 total_loss: 0.4174 time: 0.5417 s/iter data_time: 0.0512 s/iter total_throughput: 1890.21 samples/s lr: 5.75e-04 [09/20 05:57:41] lb.utils.events INFO: eta: 1 day, 7:05:43 iteration: 170799/375342 consumed_samples: 174899200 total_loss: 0.4197 time: 0.5417 s/iter data_time: 0.0511 s/iter total_throughput: 1890.20 samples/s lr: 5.75e-04 [09/20 05:58:36] lb.utils.events INFO: eta: 1 day, 7:04:26 iteration: 170899/375342 consumed_samples: 175001600 total_loss: 0.4239 time: 0.5417 s/iter data_time: 0.0501 s/iter total_throughput: 1890.19 samples/s lr: 5.74e-04 [09/20 05:59:31] lb.utils.events INFO: eta: 1 day, 7:03:20 iteration: 170999/375342 consumed_samples: 175104000 total_loss: 0.427 time: 0.5417 s/iter data_time: 0.0497 s/iter total_throughput: 1890.18 samples/s lr: 5.74e-04 [09/20 06:00:25] lb.utils.events INFO: eta: 1 day, 7:01:55 iteration: 171099/375342 consumed_samples: 175206400 total_loss: 0.4182 time: 0.5417 s/iter data_time: 0.0509 s/iter total_throughput: 1890.17 samples/s lr: 5.73e-04 [09/20 06:01:20] lb.utils.events INFO: eta: 1 day, 7:00:44 iteration: 171199/375342 consumed_samples: 175308800 total_loss: 0.4182 time: 0.5418 s/iter data_time: 0.0504 s/iter total_throughput: 1890.16 samples/s lr: 5.73e-04 [09/20 06:02:14] lb.utils.events INFO: eta: 1 day, 6:59:47 iteration: 171299/375342 consumed_samples: 175411200 total_loss: 0.4225 time: 0.5418 s/iter data_time: 0.0509 s/iter total_throughput: 1890.16 samples/s lr: 5.73e-04 [09/20 06:03:09] lb.utils.events INFO: eta: 1 day, 6:58:18 iteration: 171399/375342 consumed_samples: 175513600 total_loss: 0.4195 time: 0.5418 s/iter data_time: 0.0520 s/iter total_throughput: 1890.15 samples/s lr: 5.72e-04 [09/20 06:04:03] lb.utils.events INFO: eta: 1 day, 6:56:15 iteration: 171499/375342 consumed_samples: 175616000 total_loss: 0.4113 time: 0.5418 s/iter data_time: 0.0502 s/iter total_throughput: 1890.14 samples/s lr: 5.72e-04 [09/20 06:04:58] lb.utils.events INFO: eta: 1 day, 6:54:03 iteration: 171599/375342 consumed_samples: 175718400 total_loss: 0.4006 time: 0.5418 s/iter data_time: 0.0541 s/iter total_throughput: 1890.14 samples/s lr: 5.71e-04 [09/20 06:05:52] lb.utils.events INFO: eta: 1 day, 6:52:51 iteration: 171699/375342 consumed_samples: 175820800 total_loss: 0.413 time: 0.5418 s/iter data_time: 0.0527 s/iter total_throughput: 1890.13 samples/s lr: 5.71e-04 [09/20 06:06:47] lb.utils.events INFO: eta: 1 day, 6:50:57 iteration: 171799/375342 consumed_samples: 175923200 total_loss: 0.4197 time: 0.5418 s/iter data_time: 0.0521 s/iter total_throughput: 1890.12 samples/s lr: 5.71e-04 [09/20 06:07:41] lb.utils.events INFO: eta: 1 day, 6:49:21 iteration: 171899/375342 consumed_samples: 176025600 total_loss: 0.4169 time: 0.5418 s/iter data_time: 0.0520 s/iter total_throughput: 1890.12 samples/s lr: 5.70e-04 [09/20 06:08:36] lb.utils.events INFO: eta: 1 day, 6:46:53 iteration: 171999/375342 consumed_samples: 176128000 total_loss: 0.4109 time: 0.5418 s/iter data_time: 0.0520 s/iter total_throughput: 1890.11 samples/s lr: 5.70e-04 [09/20 06:09:30] lb.utils.events INFO: eta: 1 day, 6:45:22 iteration: 172099/375342 consumed_samples: 176230400 total_loss: 0.4131 time: 0.5418 s/iter data_time: 0.0502 s/iter total_throughput: 1890.11 samples/s lr: 5.69e-04 [09/20 06:10:25] lb.utils.events INFO: eta: 1 day, 6:44:05 iteration: 172199/375342 consumed_samples: 176332800 total_loss: 0.4137 time: 0.5418 s/iter data_time: 0.0476 s/iter total_throughput: 1890.10 samples/s lr: 5.69e-04 [09/20 06:11:19] lb.utils.events INFO: eta: 1 day, 6:42:53 iteration: 172299/375342 consumed_samples: 176435200 total_loss: 0.4087 time: 0.5418 s/iter data_time: 0.0501 s/iter total_throughput: 1890.09 samples/s lr: 5.69e-04 [09/20 06:12:14] lb.utils.events INFO: eta: 1 day, 6:41:48 iteration: 172399/375342 consumed_samples: 176537600 total_loss: 0.4147 time: 0.5418 s/iter data_time: 0.0505 s/iter total_throughput: 1890.09 samples/s lr: 5.68e-04 [09/20 06:13:09] lb.utils.events INFO: eta: 1 day, 6:40:59 iteration: 172499/375342 consumed_samples: 176640000 total_loss: 0.419 time: 0.5418 s/iter data_time: 0.0547 s/iter total_throughput: 1890.07 samples/s lr: 5.68e-04 [09/20 06:14:03] lb.utils.events INFO: eta: 1 day, 6:40:27 iteration: 172599/375342 consumed_samples: 176742400 total_loss: 0.4151 time: 0.5418 s/iter data_time: 0.0548 s/iter total_throughput: 1890.07 samples/s lr: 5.67e-04 [09/20 06:14:58] lb.utils.events INFO: eta: 1 day, 6:39:57 iteration: 172699/375342 consumed_samples: 176844800 total_loss: 0.421 time: 0.5418 s/iter data_time: 0.0532 s/iter total_throughput: 1890.06 samples/s lr: 5.67e-04 [09/20 06:15:53] lb.utils.events INFO: eta: 1 day, 6:39:20 iteration: 172799/375342 consumed_samples: 176947200 total_loss: 0.4158 time: 0.5418 s/iter data_time: 0.0531 s/iter total_throughput: 1890.05 samples/s lr: 5.66e-04 [09/20 06:16:47] lb.utils.events INFO: eta: 1 day, 6:38:35 iteration: 172899/375342 consumed_samples: 177049600 total_loss: 0.4077 time: 0.5418 s/iter data_time: 0.0541 s/iter total_throughput: 1890.04 samples/s lr: 5.66e-04 [09/20 06:17:42] lb.utils.events INFO: eta: 1 day, 6:39:28 iteration: 172999/375342 consumed_samples: 177152000 total_loss: 0.4093 time: 0.5418 s/iter data_time: 0.0540 s/iter total_throughput: 1890.03 samples/s lr: 5.66e-04 [09/20 06:18:37] lb.utils.events INFO: eta: 1 day, 6:39:33 iteration: 173099/375342 consumed_samples: 177254400 total_loss: 0.4159 time: 0.5418 s/iter data_time: 0.0560 s/iter total_throughput: 1890.02 samples/s lr: 5.65e-04 [09/20 06:19:31] lb.utils.events INFO: eta: 1 day, 6:38:52 iteration: 173199/375342 consumed_samples: 177356800 total_loss: 0.4158 time: 0.5418 s/iter data_time: 0.0565 s/iter total_throughput: 1890.01 samples/s lr: 5.65e-04 [09/20 06:20:26] lb.utils.events INFO: eta: 1 day, 6:38:32 iteration: 173299/375342 consumed_samples: 177459200 total_loss: 0.419 time: 0.5418 s/iter data_time: 0.0548 s/iter total_throughput: 1890.00 samples/s lr: 5.64e-04 [09/20 06:21:21] lb.utils.events INFO: eta: 1 day, 6:38:17 iteration: 173399/375342 consumed_samples: 177561600 total_loss: 0.4203 time: 0.5418 s/iter data_time: 0.0546 s/iter total_throughput: 1889.99 samples/s lr: 5.64e-04 [09/20 06:22:16] lb.utils.events INFO: eta: 1 day, 6:38:09 iteration: 173499/375342 consumed_samples: 177664000 total_loss: 0.4049 time: 0.5418 s/iter data_time: 0.0542 s/iter total_throughput: 1889.98 samples/s lr: 5.64e-04 [09/20 06:23:10] lb.utils.events INFO: eta: 1 day, 6:38:07 iteration: 173599/375342 consumed_samples: 177766400 total_loss: 0.4085 time: 0.5418 s/iter data_time: 0.0540 s/iter total_throughput: 1889.97 samples/s lr: 5.63e-04 [09/20 06:24:05] lb.utils.events INFO: eta: 1 day, 6:38:07 iteration: 173699/375342 consumed_samples: 177868800 total_loss: 0.4168 time: 0.5418 s/iter data_time: 0.0555 s/iter total_throughput: 1889.96 samples/s lr: 5.63e-04 [09/20 06:25:00] lb.utils.events INFO: eta: 1 day, 6:37:30 iteration: 173799/375342 consumed_samples: 177971200 total_loss: 0.4129 time: 0.5418 s/iter data_time: 0.0527 s/iter total_throughput: 1889.95 samples/s lr: 5.62e-04 [09/20 06:25:55] lb.utils.events INFO: eta: 1 day, 6:37:25 iteration: 173899/375342 consumed_samples: 178073600 total_loss: 0.4151 time: 0.5418 s/iter data_time: 0.0524 s/iter total_throughput: 1889.93 samples/s lr: 5.62e-04 [09/20 06:26:50] lb.utils.events INFO: eta: 1 day, 6:36:57 iteration: 173999/375342 consumed_samples: 178176000 total_loss: 0.4188 time: 0.5418 s/iter data_time: 0.0485 s/iter total_throughput: 1889.92 samples/s lr: 5.62e-04 [09/20 06:27:44] lb.utils.events INFO: eta: 1 day, 6:36:51 iteration: 174099/375342 consumed_samples: 178278400 total_loss: 0.4181 time: 0.5418 s/iter data_time: 0.0507 s/iter total_throughput: 1889.91 samples/s lr: 5.61e-04 [09/20 06:28:39] lb.utils.events INFO: eta: 1 day, 6:35:51 iteration: 174199/375342 consumed_samples: 178380800 total_loss: 0.4172 time: 0.5418 s/iter data_time: 0.0512 s/iter total_throughput: 1889.90 samples/s lr: 5.61e-04 [09/20 06:29:34] lb.utils.events INFO: eta: 1 day, 6:34:51 iteration: 174299/375342 consumed_samples: 178483200 total_loss: 0.417 time: 0.5418 s/iter data_time: 0.0505 s/iter total_throughput: 1889.89 samples/s lr: 5.60e-04 [09/20 06:30:28] lb.utils.events INFO: eta: 1 day, 6:33:17 iteration: 174399/375342 consumed_samples: 178585600 total_loss: 0.4169 time: 0.5418 s/iter data_time: 0.0502 s/iter total_throughput: 1889.88 samples/s lr: 5.60e-04 [09/20 06:31:23] lb.utils.events INFO: eta: 1 day, 6:31:36 iteration: 174499/375342 consumed_samples: 178688000 total_loss: 0.4139 time: 0.5418 s/iter data_time: 0.0508 s/iter total_throughput: 1889.87 samples/s lr: 5.59e-04 [09/20 06:32:18] lb.utils.events INFO: eta: 1 day, 6:30:25 iteration: 174599/375342 consumed_samples: 178790400 total_loss: 0.4131 time: 0.5418 s/iter data_time: 0.0516 s/iter total_throughput: 1889.87 samples/s lr: 5.59e-04 [09/20 06:33:12] lb.utils.events INFO: eta: 1 day, 6:28:49 iteration: 174699/375342 consumed_samples: 178892800 total_loss: 0.4202 time: 0.5418 s/iter data_time: 0.0506 s/iter total_throughput: 1889.86 samples/s lr: 5.59e-04 [09/20 06:34:07] lb.utils.events INFO: eta: 1 day, 6:27:29 iteration: 174799/375342 consumed_samples: 178995200 total_loss: 0.4124 time: 0.5418 s/iter data_time: 0.0534 s/iter total_throughput: 1889.85 samples/s lr: 5.58e-04 [09/20 06:35:01] lb.utils.events INFO: eta: 1 day, 6:25:57 iteration: 174899/375342 consumed_samples: 179097600 total_loss: 0.4143 time: 0.5418 s/iter data_time: 0.0521 s/iter total_throughput: 1889.84 samples/s lr: 5.58e-04 [09/20 06:35:56] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0174999 [09/20 06:35:56] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 06:35:56] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 06:36:01] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0617 s/iter. Inference: 0.2537 s/iter. Eval: 0.0033 s/iter. Total: 0.3187 s/iter. ETA=0:00:11 [09/20 06:36:06] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0739 s/iter. Inference: 0.2608 s/iter. Eval: 0.0026 s/iter. Total: 0.3375 s/iter. ETA=0:00:07 [09/20 06:36:12] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0733 s/iter. Inference: 0.2554 s/iter. Eval: 0.0026 s/iter. Total: 0.3317 s/iter. ETA=0:00:01 [09/20 06:36:14] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 06:36:14] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.249150 (0.000285 s / iter per device, on 8 devices) [09/20 06:36:14] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/20 06:36:14] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 06:36:14] lb.evaluation.utils INFO: copypaste: Acc@1=74.238 [09/20 06:36:14] lb.evaluation.utils INFO: copypaste: Acc@5=92.182 [09/20 06:36:14] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 74.23800, better than last best score 74.15200 @ iteration 169999. [09/20 06:36:14] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 06:36:14] lb.utils.events INFO: eta: 1 day, 6:24:07 iteration: 174999/375342 consumed_samples: 179200000 total_loss: 0.4188 time: 0.5418 s/iter data_time: 0.0514 s/iter total_throughput: 1889.84 samples/s lr: 5.57e-04 [09/20 06:37:09] lb.utils.events INFO: eta: 1 day, 6:21:40 iteration: 175099/375342 consumed_samples: 179302400 total_loss: 0.4191 time: 0.5418 s/iter data_time: 0.0512 s/iter total_throughput: 1889.83 samples/s lr: 5.57e-04 [09/20 06:38:03] lb.utils.events INFO: eta: 1 day, 6:19:08 iteration: 175199/375342 consumed_samples: 179404800 total_loss: 0.4198 time: 0.5418 s/iter data_time: 0.0513 s/iter total_throughput: 1889.83 samples/s lr: 5.57e-04 [09/20 06:38:58] lb.utils.events INFO: eta: 1 day, 6:17:32 iteration: 175299/375342 consumed_samples: 179507200 total_loss: 0.4167 time: 0.5419 s/iter data_time: 0.0531 s/iter total_throughput: 1889.82 samples/s lr: 5.56e-04 [09/20 06:39:52] lb.utils.events INFO: eta: 1 day, 6:15:48 iteration: 175399/375342 consumed_samples: 179609600 total_loss: 0.4129 time: 0.5419 s/iter data_time: 0.0532 s/iter total_throughput: 1889.82 samples/s lr: 5.56e-04 [09/20 06:40:46] lb.utils.events INFO: eta: 1 day, 6:14:41 iteration: 175499/375342 consumed_samples: 179712000 total_loss: 0.4181 time: 0.5419 s/iter data_time: 0.0525 s/iter total_throughput: 1889.81 samples/s lr: 5.55e-04 [09/20 06:41:41] lb.utils.events INFO: eta: 1 day, 6:13:41 iteration: 175599/375342 consumed_samples: 179814400 total_loss: 0.4179 time: 0.5419 s/iter data_time: 0.0492 s/iter total_throughput: 1889.80 samples/s lr: 5.55e-04 [09/20 06:42:36] lb.utils.events INFO: eta: 1 day, 6:12:52 iteration: 175699/375342 consumed_samples: 179916800 total_loss: 0.4129 time: 0.5419 s/iter data_time: 0.0501 s/iter total_throughput: 1889.79 samples/s lr: 5.55e-04 [09/20 06:43:30] lb.utils.events INFO: eta: 1 day, 6:12:01 iteration: 175799/375342 consumed_samples: 180019200 total_loss: 0.412 time: 0.5419 s/iter data_time: 0.0509 s/iter total_throughput: 1889.79 samples/s lr: 5.54e-04 [09/20 06:44:25] lb.utils.events INFO: eta: 1 day, 6:10:37 iteration: 175899/375342 consumed_samples: 180121600 total_loss: 0.4116 time: 0.5419 s/iter data_time: 0.0508 s/iter total_throughput: 1889.78 samples/s lr: 5.54e-04 [09/20 06:45:20] lb.utils.events INFO: eta: 1 day, 6:10:09 iteration: 175999/375342 consumed_samples: 180224000 total_loss: 0.4153 time: 0.5419 s/iter data_time: 0.0551 s/iter total_throughput: 1889.76 samples/s lr: 5.53e-04 [09/20 06:46:14] lb.utils.events INFO: eta: 1 day, 6:10:06 iteration: 176099/375342 consumed_samples: 180326400 total_loss: 0.4114 time: 0.5419 s/iter data_time: 0.0558 s/iter total_throughput: 1889.76 samples/s lr: 5.53e-04 [09/20 06:47:09] lb.utils.events INFO: eta: 1 day, 6:10:03 iteration: 176199/375342 consumed_samples: 180428800 total_loss: 0.4073 time: 0.5419 s/iter data_time: 0.0541 s/iter total_throughput: 1889.75 samples/s lr: 5.52e-04 [09/20 06:48:04] lb.utils.events INFO: eta: 1 day, 6:09:11 iteration: 176299/375342 consumed_samples: 180531200 total_loss: 0.4146 time: 0.5419 s/iter data_time: 0.0553 s/iter total_throughput: 1889.74 samples/s lr: 5.52e-04 [09/20 06:48:58] lb.utils.events INFO: eta: 1 day, 6:09:03 iteration: 176399/375342 consumed_samples: 180633600 total_loss: 0.4195 time: 0.5419 s/iter data_time: 0.0555 s/iter total_throughput: 1889.74 samples/s lr: 5.52e-04 [09/20 06:49:53] lb.utils.events INFO: eta: 1 day, 6:09:02 iteration: 176499/375342 consumed_samples: 180736000 total_loss: 0.4188 time: 0.5419 s/iter data_time: 0.0552 s/iter total_throughput: 1889.73 samples/s lr: 5.51e-04 [09/20 06:50:48] lb.utils.events INFO: eta: 1 day, 6:09:12 iteration: 176599/375342 consumed_samples: 180838400 total_loss: 0.4185 time: 0.5419 s/iter data_time: 0.0541 s/iter total_throughput: 1889.71 samples/s lr: 5.51e-04 [09/20 06:51:43] lb.utils.events INFO: eta: 1 day, 6:09:07 iteration: 176699/375342 consumed_samples: 180940800 total_loss: 0.4208 time: 0.5419 s/iter data_time: 0.0577 s/iter total_throughput: 1889.70 samples/s lr: 5.50e-04 [09/20 06:52:37] lb.utils.events INFO: eta: 1 day, 6:08:27 iteration: 176799/375342 consumed_samples: 181043200 total_loss: 0.4206 time: 0.5419 s/iter data_time: 0.0569 s/iter total_throughput: 1889.69 samples/s lr: 5.50e-04 [09/20 06:53:32] lb.utils.events INFO: eta: 1 day, 6:08:33 iteration: 176899/375342 consumed_samples: 181145600 total_loss: 0.4149 time: 0.5419 s/iter data_time: 0.0568 s/iter total_throughput: 1889.68 samples/s lr: 5.50e-04 [09/20 06:54:27] lb.utils.events INFO: eta: 1 day, 6:08:34 iteration: 176999/375342 consumed_samples: 181248000 total_loss: 0.408 time: 0.5419 s/iter data_time: 0.0559 s/iter total_throughput: 1889.67 samples/s lr: 5.49e-04 [09/20 06:55:22] lb.utils.events INFO: eta: 1 day, 6:08:00 iteration: 177099/375342 consumed_samples: 181350400 total_loss: 0.4111 time: 0.5419 s/iter data_time: 0.0550 s/iter total_throughput: 1889.66 samples/s lr: 5.49e-04 [09/20 06:56:16] lb.utils.events INFO: eta: 1 day, 6:07:22 iteration: 177199/375342 consumed_samples: 181452800 total_loss: 0.4112 time: 0.5419 s/iter data_time: 0.0538 s/iter total_throughput: 1889.65 samples/s lr: 5.48e-04 [09/20 06:57:11] lb.utils.events INFO: eta: 1 day, 6:06:40 iteration: 177299/375342 consumed_samples: 181555200 total_loss: 0.4039 time: 0.5419 s/iter data_time: 0.0508 s/iter total_throughput: 1889.64 samples/s lr: 5.48e-04 [09/20 06:58:06] lb.utils.events INFO: eta: 1 day, 6:06:04 iteration: 177399/375342 consumed_samples: 181657600 total_loss: 0.3995 time: 0.5419 s/iter data_time: 0.0494 s/iter total_throughput: 1889.63 samples/s lr: 5.48e-04 [09/20 06:59:00] lb.utils.events INFO: eta: 1 day, 6:04:59 iteration: 177499/375342 consumed_samples: 181760000 total_loss: 0.4069 time: 0.5419 s/iter data_time: 0.0484 s/iter total_throughput: 1889.62 samples/s lr: 5.47e-04 [09/20 06:59:55] lb.utils.events INFO: eta: 1 day, 6:03:37 iteration: 177599/375342 consumed_samples: 181862400 total_loss: 0.4216 time: 0.5419 s/iter data_time: 0.0494 s/iter total_throughput: 1889.61 samples/s lr: 5.47e-04 [09/20 07:00:50] lb.utils.events INFO: eta: 1 day, 6:02:40 iteration: 177699/375342 consumed_samples: 181964800 total_loss: 0.4209 time: 0.5419 s/iter data_time: 0.0520 s/iter total_throughput: 1889.60 samples/s lr: 5.46e-04 [09/20 07:01:45] lb.utils.events INFO: eta: 1 day, 6:01:15 iteration: 177799/375342 consumed_samples: 182067200 total_loss: 0.4088 time: 0.5419 s/iter data_time: 0.0501 s/iter total_throughput: 1889.59 samples/s lr: 5.46e-04 [09/20 07:02:39] lb.utils.events INFO: eta: 1 day, 5:59:29 iteration: 177899/375342 consumed_samples: 182169600 total_loss: 0.4061 time: 0.5419 s/iter data_time: 0.0507 s/iter total_throughput: 1889.58 samples/s lr: 5.45e-04 [09/20 07:03:34] lb.utils.events INFO: eta: 1 day, 5:58:19 iteration: 177999/375342 consumed_samples: 182272000 total_loss: 0.4066 time: 0.5419 s/iter data_time: 0.0510 s/iter total_throughput: 1889.57 samples/s lr: 5.45e-04 [09/20 07:04:28] lb.utils.events INFO: eta: 1 day, 5:56:50 iteration: 178099/375342 consumed_samples: 182374400 total_loss: 0.4082 time: 0.5419 s/iter data_time: 0.0510 s/iter total_throughput: 1889.57 samples/s lr: 5.45e-04 [09/20 07:05:23] lb.utils.events INFO: eta: 1 day, 5:55:52 iteration: 178199/375342 consumed_samples: 182476800 total_loss: 0.4159 time: 0.5419 s/iter data_time: 0.0510 s/iter total_throughput: 1889.56 samples/s lr: 5.44e-04 [09/20 07:06:18] lb.utils.events INFO: eta: 1 day, 5:54:27 iteration: 178299/375342 consumed_samples: 182579200 total_loss: 0.4168 time: 0.5419 s/iter data_time: 0.0513 s/iter total_throughput: 1889.55 samples/s lr: 5.44e-04 [09/20 07:07:12] lb.utils.events INFO: eta: 1 day, 5:53:11 iteration: 178399/375342 consumed_samples: 182681600 total_loss: 0.406 time: 0.5419 s/iter data_time: 0.0529 s/iter total_throughput: 1889.55 samples/s lr: 5.43e-04 [09/20 07:08:07] lb.utils.events INFO: eta: 1 day, 5:51:39 iteration: 178499/375342 consumed_samples: 182784000 total_loss: 0.4126 time: 0.5419 s/iter data_time: 0.0511 s/iter total_throughput: 1889.54 samples/s lr: 5.43e-04 [09/20 07:09:01] lb.utils.events INFO: eta: 1 day, 5:49:48 iteration: 178599/375342 consumed_samples: 182886400 total_loss: 0.4178 time: 0.5419 s/iter data_time: 0.0509 s/iter total_throughput: 1889.54 samples/s lr: 5.43e-04 [09/20 07:09:56] lb.utils.events INFO: eta: 1 day, 5:47:28 iteration: 178699/375342 consumed_samples: 182988800 total_loss: 0.4105 time: 0.5419 s/iter data_time: 0.0519 s/iter total_throughput: 1889.53 samples/s lr: 5.42e-04 [09/20 07:10:50] lb.utils.events INFO: eta: 1 day, 5:46:24 iteration: 178799/375342 consumed_samples: 183091200 total_loss: 0.4056 time: 0.5419 s/iter data_time: 0.0531 s/iter total_throughput: 1889.52 samples/s lr: 5.42e-04 [09/20 07:11:45] lb.utils.events INFO: eta: 1 day, 5:45:00 iteration: 178899/375342 consumed_samples: 183193600 total_loss: 0.41 time: 0.5419 s/iter data_time: 0.0521 s/iter total_throughput: 1889.52 samples/s lr: 5.41e-04 [09/20 07:12:39] lb.utils.events INFO: eta: 1 day, 5:43:32 iteration: 178999/375342 consumed_samples: 183296000 total_loss: 0.4092 time: 0.5419 s/iter data_time: 0.0508 s/iter total_throughput: 1889.51 samples/s lr: 5.41e-04 [09/20 07:13:34] lb.utils.events INFO: eta: 1 day, 5:42:29 iteration: 179099/375342 consumed_samples: 183398400 total_loss: 0.4101 time: 0.5419 s/iter data_time: 0.0523 s/iter total_throughput: 1889.51 samples/s lr: 5.40e-04 [09/20 07:14:28] lb.utils.events INFO: eta: 1 day, 5:41:10 iteration: 179199/375342 consumed_samples: 183500800 total_loss: 0.409 time: 0.5419 s/iter data_time: 0.0497 s/iter total_throughput: 1889.50 samples/s lr: 5.40e-04 [09/20 07:15:23] lb.utils.events INFO: eta: 1 day, 5:40:23 iteration: 179299/375342 consumed_samples: 183603200 total_loss: 0.4135 time: 0.5419 s/iter data_time: 0.0509 s/iter total_throughput: 1889.49 samples/s lr: 5.40e-04 [09/20 07:16:18] lb.utils.events INFO: eta: 1 day, 5:39:38 iteration: 179399/375342 consumed_samples: 183705600 total_loss: 0.4149 time: 0.5419 s/iter data_time: 0.0537 s/iter total_throughput: 1889.48 samples/s lr: 5.39e-04 [09/20 07:17:13] lb.utils.events INFO: eta: 1 day, 5:39:21 iteration: 179499/375342 consumed_samples: 183808000 total_loss: 0.414 time: 0.5420 s/iter data_time: 0.0541 s/iter total_throughput: 1889.47 samples/s lr: 5.39e-04 [09/20 07:18:07] lb.utils.events INFO: eta: 1 day, 5:38:59 iteration: 179599/375342 consumed_samples: 183910400 total_loss: 0.4106 time: 0.5420 s/iter data_time: 0.0539 s/iter total_throughput: 1889.46 samples/s lr: 5.38e-04 [09/20 07:19:02] lb.utils.events INFO: eta: 1 day, 5:38:43 iteration: 179699/375342 consumed_samples: 184012800 total_loss: 0.4123 time: 0.5420 s/iter data_time: 0.0536 s/iter total_throughput: 1889.45 samples/s lr: 5.38e-04 [09/20 07:19:57] lb.utils.events INFO: eta: 1 day, 5:38:06 iteration: 179799/375342 consumed_samples: 184115200 total_loss: 0.4193 time: 0.5420 s/iter data_time: 0.0545 s/iter total_throughput: 1889.44 samples/s lr: 5.38e-04 [09/20 07:20:51] lb.utils.events INFO: eta: 1 day, 5:38:47 iteration: 179899/375342 consumed_samples: 184217600 total_loss: 0.4141 time: 0.5420 s/iter data_time: 0.0565 s/iter total_throughput: 1889.43 samples/s lr: 5.37e-04 [09/20 07:21:46] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0179999 [09/20 07:21:46] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 07:21:46] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 07:21:51] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0508 s/iter. Inference: 0.2489 s/iter. Eval: 0.0023 s/iter. Total: 0.3020 s/iter. ETA=0:00:11 [09/20 07:21:56] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0732 s/iter. Inference: 0.2570 s/iter. Eval: 0.0022 s/iter. Total: 0.3327 s/iter. ETA=0:00:07 [09/20 07:22:01] lb.evaluation.evaluator INFO: Inference done 41984/50000. Dataloading: 0.0755 s/iter. Inference: 0.2554 s/iter. Eval: 0.0022 s/iter. Total: 0.3336 s/iter. ETA=0:00:02 [09/20 07:22:04] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 07:22:04] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.295110 (0.000286 s / iter per device, on 8 devices) [09/20 07:22:04] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000222 s / iter per device, on 8 devices) [09/20 07:22:04] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 07:22:04] lb.evaluation.utils INFO: copypaste: Acc@1=74.61 [09/20 07:22:04] lb.evaluation.utils INFO: copypaste: Acc@5=92.34599999999999 [09/20 07:22:04] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 74.61000, better than last best score 74.23800 @ iteration 174999. [09/20 07:22:04] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 07:22:04] lb.utils.events INFO: eta: 1 day, 5:38:47 iteration: 179999/375342 consumed_samples: 184320000 total_loss: 0.4118 time: 0.5420 s/iter data_time: 0.0541 s/iter total_throughput: 1889.42 samples/s lr: 5.37e-04 [09/20 07:22:59] lb.utils.events INFO: eta: 1 day, 5:38:34 iteration: 180099/375342 consumed_samples: 184422400 total_loss: 0.4167 time: 0.5420 s/iter data_time: 0.0563 s/iter total_throughput: 1889.41 samples/s lr: 5.36e-04 [09/20 07:23:54] lb.utils.events INFO: eta: 1 day, 5:38:05 iteration: 180199/375342 consumed_samples: 184524800 total_loss: 0.4109 time: 0.5420 s/iter data_time: 0.0558 s/iter total_throughput: 1889.40 samples/s lr: 5.36e-04 [09/20 07:24:49] lb.utils.events INFO: eta: 1 day, 5:38:01 iteration: 180299/375342 consumed_samples: 184627200 total_loss: 0.4072 time: 0.5420 s/iter data_time: 0.0552 s/iter total_throughput: 1889.39 samples/s lr: 5.36e-04 [09/20 07:25:43] lb.utils.events INFO: eta: 1 day, 5:37:40 iteration: 180399/375342 consumed_samples: 184729600 total_loss: 0.4121 time: 0.5420 s/iter data_time: 0.0558 s/iter total_throughput: 1889.38 samples/s lr: 5.35e-04 [09/20 07:26:38] lb.utils.events INFO: eta: 1 day, 5:36:55 iteration: 180499/375342 consumed_samples: 184832000 total_loss: 0.4176 time: 0.5420 s/iter data_time: 0.0556 s/iter total_throughput: 1889.37 samples/s lr: 5.35e-04 [09/20 07:27:33] lb.utils.events INFO: eta: 1 day, 5:36:35 iteration: 180599/375342 consumed_samples: 184934400 total_loss: 0.412 time: 0.5420 s/iter data_time: 0.0552 s/iter total_throughput: 1889.36 samples/s lr: 5.34e-04 [09/20 07:28:28] lb.utils.events INFO: eta: 1 day, 5:35:47 iteration: 180699/375342 consumed_samples: 185036800 total_loss: 0.4152 time: 0.5420 s/iter data_time: 0.0493 s/iter total_throughput: 1889.35 samples/s lr: 5.34e-04 [09/20 07:29:22] lb.utils.events INFO: eta: 1 day, 5:35:17 iteration: 180799/375342 consumed_samples: 185139200 total_loss: 0.4147 time: 0.5420 s/iter data_time: 0.0505 s/iter total_throughput: 1889.35 samples/s lr: 5.33e-04 [09/20 07:30:17] lb.utils.events INFO: eta: 1 day, 5:33:54 iteration: 180899/375342 consumed_samples: 185241600 total_loss: 0.4147 time: 0.5420 s/iter data_time: 0.0483 s/iter total_throughput: 1889.34 samples/s lr: 5.33e-04 [09/20 07:31:12] lb.utils.events INFO: eta: 1 day, 5:32:28 iteration: 180999/375342 consumed_samples: 185344000 total_loss: 0.4152 time: 0.5420 s/iter data_time: 0.0508 s/iter total_throughput: 1889.33 samples/s lr: 5.33e-04 [09/20 07:32:06] lb.utils.events INFO: eta: 1 day, 5:30:56 iteration: 181099/375342 consumed_samples: 185446400 total_loss: 0.4127 time: 0.5420 s/iter data_time: 0.0502 s/iter total_throughput: 1889.32 samples/s lr: 5.32e-04 [09/20 07:33:01] lb.utils.events INFO: eta: 1 day, 5:29:19 iteration: 181199/375342 consumed_samples: 185548800 total_loss: 0.4142 time: 0.5420 s/iter data_time: 0.0501 s/iter total_throughput: 1889.31 samples/s lr: 5.32e-04 [09/20 07:33:55] lb.utils.events INFO: eta: 1 day, 5:27:21 iteration: 181299/375342 consumed_samples: 185651200 total_loss: 0.4198 time: 0.5420 s/iter data_time: 0.0506 s/iter total_throughput: 1889.31 samples/s lr: 5.31e-04 [09/20 07:34:50] lb.utils.events INFO: eta: 1 day, 5:25:46 iteration: 181399/375342 consumed_samples: 185753600 total_loss: 0.4196 time: 0.5420 s/iter data_time: 0.0500 s/iter total_throughput: 1889.30 samples/s lr: 5.31e-04 [09/20 07:35:44] lb.utils.events INFO: eta: 1 day, 5:24:22 iteration: 181499/375342 consumed_samples: 185856000 total_loss: 0.4183 time: 0.5420 s/iter data_time: 0.0511 s/iter total_throughput: 1889.29 samples/s lr: 5.31e-04 [09/20 07:36:39] lb.utils.events INFO: eta: 1 day, 5:22:46 iteration: 181599/375342 consumed_samples: 185958400 total_loss: 0.4169 time: 0.5420 s/iter data_time: 0.0504 s/iter total_throughput: 1889.29 samples/s lr: 5.30e-04 [09/20 07:37:34] lb.utils.events INFO: eta: 1 day, 5:21:33 iteration: 181699/375342 consumed_samples: 186060800 total_loss: 0.4107 time: 0.5420 s/iter data_time: 0.0529 s/iter total_throughput: 1889.28 samples/s lr: 5.30e-04 [09/20 07:38:28] lb.utils.events INFO: eta: 1 day, 5:19:41 iteration: 181799/375342 consumed_samples: 186163200 total_loss: 0.4129 time: 0.5420 s/iter data_time: 0.0512 s/iter total_throughput: 1889.27 samples/s lr: 5.29e-04 [09/20 07:39:23] lb.utils.events INFO: eta: 1 day, 5:18:21 iteration: 181899/375342 consumed_samples: 186265600 total_loss: 0.4122 time: 0.5420 s/iter data_time: 0.0524 s/iter total_throughput: 1889.27 samples/s lr: 5.29e-04 [09/20 07:40:17] lb.utils.events INFO: eta: 1 day, 5:17:07 iteration: 181999/375342 consumed_samples: 186368000 total_loss: 0.411 time: 0.5420 s/iter data_time: 0.0513 s/iter total_throughput: 1889.26 samples/s lr: 5.28e-04 [09/20 07:41:12] lb.utils.events INFO: eta: 1 day, 5:15:43 iteration: 182099/375342 consumed_samples: 186470400 total_loss: 0.4054 time: 0.5420 s/iter data_time: 0.0514 s/iter total_throughput: 1889.26 samples/s lr: 5.28e-04 [09/20 07:42:06] lb.utils.events INFO: eta: 1 day, 5:14:42 iteration: 182199/375342 consumed_samples: 186572800 total_loss: 0.4073 time: 0.5420 s/iter data_time: 0.0501 s/iter total_throughput: 1889.25 samples/s lr: 5.28e-04 [09/20 07:43:01] lb.utils.events INFO: eta: 1 day, 5:13:38 iteration: 182299/375342 consumed_samples: 186675200 total_loss: 0.416 time: 0.5420 s/iter data_time: 0.0523 s/iter total_throughput: 1889.25 samples/s lr: 5.27e-04 [09/20 07:43:55] lb.utils.events INFO: eta: 1 day, 5:12:46 iteration: 182399/375342 consumed_samples: 186777600 total_loss: 0.4095 time: 0.5420 s/iter data_time: 0.0529 s/iter total_throughput: 1889.24 samples/s lr: 5.27e-04 [09/20 07:44:50] lb.utils.events INFO: eta: 1 day, 5:11:09 iteration: 182499/375342 consumed_samples: 186880000 total_loss: 0.4145 time: 0.5420 s/iter data_time: 0.0505 s/iter total_throughput: 1889.23 samples/s lr: 5.26e-04 [09/20 07:45:44] lb.utils.events INFO: eta: 1 day, 5:10:09 iteration: 182599/375342 consumed_samples: 186982400 total_loss: 0.4197 time: 0.5420 s/iter data_time: 0.0508 s/iter total_throughput: 1889.23 samples/s lr: 5.26e-04 [09/20 07:46:39] lb.utils.events INFO: eta: 1 day, 5:09:07 iteration: 182699/375342 consumed_samples: 187084800 total_loss: 0.415 time: 0.5420 s/iter data_time: 0.0507 s/iter total_throughput: 1889.22 samples/s lr: 5.26e-04 [09/20 07:47:34] lb.utils.events INFO: eta: 1 day, 5:08:14 iteration: 182799/375342 consumed_samples: 187187200 total_loss: 0.4154 time: 0.5420 s/iter data_time: 0.0517 s/iter total_throughput: 1889.21 samples/s lr: 5.25e-04 [09/20 07:48:28] lb.utils.events INFO: eta: 1 day, 5:07:43 iteration: 182899/375342 consumed_samples: 187289600 total_loss: 0.4127 time: 0.5420 s/iter data_time: 0.0522 s/iter total_throughput: 1889.20 samples/s lr: 5.25e-04 [09/20 07:49:23] lb.utils.events INFO: eta: 1 day, 5:06:58 iteration: 182999/375342 consumed_samples: 187392000 total_loss: 0.4093 time: 0.5420 s/iter data_time: 0.0537 s/iter total_throughput: 1889.19 samples/s lr: 5.24e-04 [09/20 07:50:18] lb.utils.events INFO: eta: 1 day, 5:06:22 iteration: 183099/375342 consumed_samples: 187494400 total_loss: 0.409 time: 0.5420 s/iter data_time: 0.0536 s/iter total_throughput: 1889.19 samples/s lr: 5.24e-04 [09/20 07:51:12] lb.utils.events INFO: eta: 1 day, 5:05:57 iteration: 183199/375342 consumed_samples: 187596800 total_loss: 0.4071 time: 0.5420 s/iter data_time: 0.0539 s/iter total_throughput: 1889.18 samples/s lr: 5.24e-04 [09/20 07:52:07] lb.utils.events INFO: eta: 1 day, 5:05:40 iteration: 183299/375342 consumed_samples: 187699200 total_loss: 0.4125 time: 0.5420 s/iter data_time: 0.0545 s/iter total_throughput: 1889.17 samples/s lr: 5.23e-04 [09/20 07:53:02] lb.utils.events INFO: eta: 1 day, 5:05:45 iteration: 183399/375342 consumed_samples: 187801600 total_loss: 0.4128 time: 0.5420 s/iter data_time: 0.0567 s/iter total_throughput: 1889.16 samples/s lr: 5.23e-04 [09/20 07:53:56] lb.utils.events INFO: eta: 1 day, 5:05:30 iteration: 183499/375342 consumed_samples: 187904000 total_loss: 0.4061 time: 0.5420 s/iter data_time: 0.0567 s/iter total_throughput: 1889.15 samples/s lr: 5.22e-04 [09/20 07:54:51] lb.utils.events INFO: eta: 1 day, 5:05:38 iteration: 183599/375342 consumed_samples: 188006400 total_loss: 0.408 time: 0.5420 s/iter data_time: 0.0561 s/iter total_throughput: 1889.14 samples/s lr: 5.22e-04 [09/20 07:55:46] lb.utils.events INFO: eta: 1 day, 5:05:30 iteration: 183699/375342 consumed_samples: 188108800 total_loss: 0.4096 time: 0.5420 s/iter data_time: 0.0546 s/iter total_throughput: 1889.13 samples/s lr: 5.21e-04 [09/20 07:56:41] lb.utils.events INFO: eta: 1 day, 5:05:46 iteration: 183799/375342 consumed_samples: 188211200 total_loss: 0.409 time: 0.5421 s/iter data_time: 0.0539 s/iter total_throughput: 1889.12 samples/s lr: 5.21e-04 [09/20 07:57:35] lb.utils.events INFO: eta: 1 day, 5:05:35 iteration: 183899/375342 consumed_samples: 188313600 total_loss: 0.4101 time: 0.5421 s/iter data_time: 0.0525 s/iter total_throughput: 1889.11 samples/s lr: 5.21e-04 [09/20 07:58:30] lb.utils.events INFO: eta: 1 day, 5:04:55 iteration: 183999/375342 consumed_samples: 188416000 total_loss: 0.4116 time: 0.5421 s/iter data_time: 0.0550 s/iter total_throughput: 1889.11 samples/s lr: 5.20e-04 [09/20 07:59:25] lb.utils.events INFO: eta: 1 day, 5:04:00 iteration: 184099/375342 consumed_samples: 188518400 total_loss: 0.4121 time: 0.5421 s/iter data_time: 0.0530 s/iter total_throughput: 1889.10 samples/s lr: 5.20e-04 [09/20 08:00:19] lb.utils.events INFO: eta: 1 day, 5:03:12 iteration: 184199/375342 consumed_samples: 188620800 total_loss: 0.4081 time: 0.5421 s/iter data_time: 0.0506 s/iter total_throughput: 1889.09 samples/s lr: 5.19e-04 [09/20 08:01:14] lb.utils.events INFO: eta: 1 day, 5:02:04 iteration: 184299/375342 consumed_samples: 188723200 total_loss: 0.4093 time: 0.5421 s/iter data_time: 0.0513 s/iter total_throughput: 1889.08 samples/s lr: 5.19e-04 [09/20 08:02:09] lb.utils.events INFO: eta: 1 day, 5:00:49 iteration: 184399/375342 consumed_samples: 188825600 total_loss: 0.4142 time: 0.5421 s/iter data_time: 0.0495 s/iter total_throughput: 1889.07 samples/s lr: 5.19e-04 [09/20 08:03:03] lb.utils.events INFO: eta: 1 day, 4:59:09 iteration: 184499/375342 consumed_samples: 188928000 total_loss: 0.4166 time: 0.5421 s/iter data_time: 0.0531 s/iter total_throughput: 1889.07 samples/s lr: 5.18e-04 [09/20 08:03:58] lb.utils.events INFO: eta: 1 day, 4:57:16 iteration: 184599/375342 consumed_samples: 189030400 total_loss: 0.4149 time: 0.5421 s/iter data_time: 0.0503 s/iter total_throughput: 1889.06 samples/s lr: 5.18e-04 [09/20 08:04:52] lb.utils.events INFO: eta: 1 day, 4:55:59 iteration: 184699/375342 consumed_samples: 189132800 total_loss: 0.4142 time: 0.5421 s/iter data_time: 0.0500 s/iter total_throughput: 1889.05 samples/s lr: 5.17e-04 [09/20 08:05:47] lb.utils.events INFO: eta: 1 day, 4:54:26 iteration: 184799/375342 consumed_samples: 189235200 total_loss: 0.4147 time: 0.5421 s/iter data_time: 0.0502 s/iter total_throughput: 1889.05 samples/s lr: 5.17e-04 [09/20 08:06:42] lb.utils.events INFO: eta: 1 day, 4:52:56 iteration: 184899/375342 consumed_samples: 189337600 total_loss: 0.4109 time: 0.5421 s/iter data_time: 0.0512 s/iter total_throughput: 1889.04 samples/s lr: 5.16e-04 [09/20 08:07:36] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0184999 [09/20 08:07:37] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 08:07:37] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 08:07:41] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0560 s/iter. Inference: 0.2431 s/iter. Eval: 0.0023 s/iter. Total: 0.3015 s/iter. ETA=0:00:11 [09/20 08:07:47] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0732 s/iter. Inference: 0.2569 s/iter. Eval: 0.0026 s/iter. Total: 0.3329 s/iter. ETA=0:00:07 [09/20 08:07:52] lb.evaluation.evaluator INFO: Inference done 41984/50000. Dataloading: 0.0747 s/iter. Inference: 0.2562 s/iter. Eval: 0.0025 s/iter. Total: 0.3337 s/iter. ETA=0:00:02 [09/20 08:07:54] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 08:07:54] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.377410 (0.000288 s / iter per device, on 8 devices) [09/20 08:07:54] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/20 08:07:54] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 08:07:54] lb.evaluation.utils INFO: copypaste: Acc@1=74.774 [09/20 08:07:54] lb.evaluation.utils INFO: copypaste: Acc@5=92.314 [09/20 08:07:54] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 74.77400, better than last best score 74.61000 @ iteration 179999. [09/20 08:07:54] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 08:07:54] lb.utils.events INFO: eta: 1 day, 4:52:10 iteration: 184999/375342 consumed_samples: 189440000 total_loss: 0.4107 time: 0.5421 s/iter data_time: 0.0510 s/iter total_throughput: 1889.03 samples/s lr: 5.16e-04 [09/20 08:08:49] lb.utils.events INFO: eta: 1 day, 4:50:40 iteration: 185099/375342 consumed_samples: 189542400 total_loss: 0.4083 time: 0.5421 s/iter data_time: 0.0515 s/iter total_throughput: 1889.03 samples/s lr: 5.16e-04 [09/20 08:09:44] lb.utils.events INFO: eta: 1 day, 4:49:13 iteration: 185199/375342 consumed_samples: 189644800 total_loss: 0.4086 time: 0.5421 s/iter data_time: 0.0524 s/iter total_throughput: 1889.02 samples/s lr: 5.15e-04 [09/20 08:10:38] lb.utils.events INFO: eta: 1 day, 4:48:03 iteration: 185299/375342 consumed_samples: 189747200 total_loss: 0.4112 time: 0.5421 s/iter data_time: 0.0538 s/iter total_throughput: 1889.01 samples/s lr: 5.15e-04 [09/20 08:11:33] lb.utils.events INFO: eta: 1 day, 4:46:18 iteration: 185399/375342 consumed_samples: 189849600 total_loss: 0.411 time: 0.5421 s/iter data_time: 0.0518 s/iter total_throughput: 1889.01 samples/s lr: 5.14e-04 [09/20 08:12:27] lb.utils.events INFO: eta: 1 day, 4:45:13 iteration: 185499/375342 consumed_samples: 189952000 total_loss: 0.4124 time: 0.5421 s/iter data_time: 0.0529 s/iter total_throughput: 1889.01 samples/s lr: 5.14e-04 [09/20 08:13:22] lb.utils.events INFO: eta: 1 day, 4:43:56 iteration: 185599/375342 consumed_samples: 190054400 total_loss: 0.4107 time: 0.5421 s/iter data_time: 0.0522 s/iter total_throughput: 1889.00 samples/s lr: 5.14e-04 [09/20 08:14:16] lb.utils.events INFO: eta: 1 day, 4:42:21 iteration: 185699/375342 consumed_samples: 190156800 total_loss: 0.4109 time: 0.5421 s/iter data_time: 0.0520 s/iter total_throughput: 1889.00 samples/s lr: 5.13e-04 [09/20 08:15:10] lb.utils.events INFO: eta: 1 day, 4:40:37 iteration: 185799/375342 consumed_samples: 190259200 total_loss: 0.414 time: 0.5421 s/iter data_time: 0.0513 s/iter total_throughput: 1889.00 samples/s lr: 5.13e-04 [09/20 08:16:05] lb.utils.events INFO: eta: 1 day, 4:39:16 iteration: 185899/375342 consumed_samples: 190361600 total_loss: 0.4142 time: 0.5421 s/iter data_time: 0.0515 s/iter total_throughput: 1888.99 samples/s lr: 5.12e-04 [09/20 08:16:59] lb.utils.events INFO: eta: 1 day, 4:38:11 iteration: 185999/375342 consumed_samples: 190464000 total_loss: 0.411 time: 0.5421 s/iter data_time: 0.0490 s/iter total_throughput: 1888.98 samples/s lr: 5.12e-04 [09/20 08:17:54] lb.utils.events INFO: eta: 1 day, 4:37:18 iteration: 186099/375342 consumed_samples: 190566400 total_loss: 0.4155 time: 0.5421 s/iter data_time: 0.0496 s/iter total_throughput: 1888.98 samples/s lr: 5.12e-04 [09/20 08:18:49] lb.utils.events INFO: eta: 1 day, 4:36:23 iteration: 186199/375342 consumed_samples: 190668800 total_loss: 0.4126 time: 0.5421 s/iter data_time: 0.0487 s/iter total_throughput: 1888.97 samples/s lr: 5.11e-04 [09/20 08:19:43] lb.utils.events INFO: eta: 1 day, 4:35:27 iteration: 186299/375342 consumed_samples: 190771200 total_loss: 0.4132 time: 0.5421 s/iter data_time: 0.0551 s/iter total_throughput: 1888.96 samples/s lr: 5.11e-04 [09/20 08:20:38] lb.utils.events INFO: eta: 1 day, 4:34:58 iteration: 186399/375342 consumed_samples: 190873600 total_loss: 0.4044 time: 0.5421 s/iter data_time: 0.0524 s/iter total_throughput: 1888.95 samples/s lr: 5.10e-04 [09/20 08:21:33] lb.utils.events INFO: eta: 1 day, 4:34:16 iteration: 186499/375342 consumed_samples: 190976000 total_loss: 0.4064 time: 0.5421 s/iter data_time: 0.0537 s/iter total_throughput: 1888.95 samples/s lr: 5.10e-04 [09/20 08:22:27] lb.utils.events INFO: eta: 1 day, 4:33:52 iteration: 186599/375342 consumed_samples: 191078400 total_loss: 0.4077 time: 0.5421 s/iter data_time: 0.0534 s/iter total_throughput: 1888.94 samples/s lr: 5.09e-04 [09/20 08:23:22] lb.utils.events INFO: eta: 1 day, 4:33:30 iteration: 186699/375342 consumed_samples: 191180800 total_loss: 0.4058 time: 0.5421 s/iter data_time: 0.0545 s/iter total_throughput: 1888.93 samples/s lr: 5.09e-04 [09/20 08:24:16] lb.utils.events INFO: eta: 1 day, 4:33:22 iteration: 186799/375342 consumed_samples: 191283200 total_loss: 0.415 time: 0.5421 s/iter data_time: 0.0540 s/iter total_throughput: 1888.92 samples/s lr: 5.09e-04 [09/20 08:25:11] lb.utils.events INFO: eta: 1 day, 4:33:31 iteration: 186899/375342 consumed_samples: 191385600 total_loss: 0.4058 time: 0.5421 s/iter data_time: 0.0533 s/iter total_throughput: 1888.92 samples/s lr: 5.08e-04 [09/20 08:26:06] lb.utils.events INFO: eta: 1 day, 4:33:53 iteration: 186999/375342 consumed_samples: 191488000 total_loss: 0.4038 time: 0.5421 s/iter data_time: 0.0552 s/iter total_throughput: 1888.90 samples/s lr: 5.08e-04 [09/20 08:27:01] lb.utils.events INFO: eta: 1 day, 4:33:36 iteration: 187099/375342 consumed_samples: 191590400 total_loss: 0.4046 time: 0.5421 s/iter data_time: 0.0564 s/iter total_throughput: 1888.89 samples/s lr: 5.07e-04 [09/20 08:27:56] lb.utils.events INFO: eta: 1 day, 4:33:26 iteration: 187199/375342 consumed_samples: 191692800 total_loss: 0.4112 time: 0.5421 s/iter data_time: 0.0559 s/iter total_throughput: 1888.88 samples/s lr: 5.07e-04 [09/20 08:28:50] lb.utils.events INFO: eta: 1 day, 4:33:22 iteration: 187299/375342 consumed_samples: 191795200 total_loss: 0.4094 time: 0.5421 s/iter data_time: 0.0565 s/iter total_throughput: 1888.87 samples/s lr: 5.07e-04 [09/20 08:29:45] lb.utils.events INFO: eta: 1 day, 4:33:21 iteration: 187399/375342 consumed_samples: 191897600 total_loss: 0.4033 time: 0.5421 s/iter data_time: 0.0535 s/iter total_throughput: 1888.86 samples/s lr: 5.06e-04 [09/20 08:30:40] lb.utils.events INFO: eta: 1 day, 4:33:29 iteration: 187499/375342 consumed_samples: 192000000 total_loss: 0.4015 time: 0.5421 s/iter data_time: 0.0563 s/iter total_throughput: 1888.86 samples/s lr: 5.06e-04 [09/20 08:31:35] lb.utils.events INFO: eta: 1 day, 4:32:51 iteration: 187599/375342 consumed_samples: 192102400 total_loss: 0.4155 time: 0.5421 s/iter data_time: 0.0507 s/iter total_throughput: 1888.85 samples/s lr: 5.05e-04 [09/20 08:32:29] lb.utils.events INFO: eta: 1 day, 4:32:20 iteration: 187699/375342 consumed_samples: 192204800 total_loss: 0.4217 time: 0.5421 s/iter data_time: 0.0509 s/iter total_throughput: 1888.84 samples/s lr: 5.05e-04 [09/20 08:33:24] lb.utils.events INFO: eta: 1 day, 4:30:57 iteration: 187799/375342 consumed_samples: 192307200 total_loss: 0.4201 time: 0.5421 s/iter data_time: 0.0488 s/iter total_throughput: 1888.83 samples/s lr: 5.04e-04 [09/20 08:34:19] lb.utils.events INFO: eta: 1 day, 4:29:32 iteration: 187899/375342 consumed_samples: 192409600 total_loss: 0.4158 time: 0.5421 s/iter data_time: 0.0491 s/iter total_throughput: 1888.82 samples/s lr: 5.04e-04 [09/20 08:35:13] lb.utils.events INFO: eta: 1 day, 4:28:03 iteration: 187999/375342 consumed_samples: 192512000 total_loss: 0.4054 time: 0.5421 s/iter data_time: 0.0503 s/iter total_throughput: 1888.81 samples/s lr: 5.04e-04 [09/20 08:36:08] lb.utils.events INFO: eta: 1 day, 4:26:47 iteration: 188099/375342 consumed_samples: 192614400 total_loss: 0.4058 time: 0.5421 s/iter data_time: 0.0504 s/iter total_throughput: 1888.81 samples/s lr: 5.03e-04 [09/20 08:37:03] lb.utils.events INFO: eta: 1 day, 4:25:28 iteration: 188199/375342 consumed_samples: 192716800 total_loss: 0.4096 time: 0.5421 s/iter data_time: 0.0511 s/iter total_throughput: 1888.80 samples/s lr: 5.03e-04 [09/20 08:37:57] lb.utils.events INFO: eta: 1 day, 4:24:04 iteration: 188299/375342 consumed_samples: 192819200 total_loss: 0.4096 time: 0.5421 s/iter data_time: 0.0495 s/iter total_throughput: 1888.79 samples/s lr: 5.02e-04 [09/20 08:38:52] lb.utils.events INFO: eta: 1 day, 4:22:57 iteration: 188399/375342 consumed_samples: 192921600 total_loss: 0.4152 time: 0.5421 s/iter data_time: 0.0523 s/iter total_throughput: 1888.78 samples/s lr: 5.02e-04 [09/20 08:39:47] lb.utils.events INFO: eta: 1 day, 4:21:05 iteration: 188499/375342 consumed_samples: 193024000 total_loss: 0.4162 time: 0.5422 s/iter data_time: 0.0498 s/iter total_throughput: 1888.78 samples/s lr: 5.02e-04 [09/20 08:40:41] lb.utils.events INFO: eta: 1 day, 4:19:41 iteration: 188599/375342 consumed_samples: 193126400 total_loss: 0.4086 time: 0.5422 s/iter data_time: 0.0513 s/iter total_throughput: 1888.77 samples/s lr: 5.01e-04 [09/20 08:41:36] lb.utils.events INFO: eta: 1 day, 4:18:37 iteration: 188699/375342 consumed_samples: 193228800 total_loss: 0.4043 time: 0.5422 s/iter data_time: 0.0503 s/iter total_throughput: 1888.76 samples/s lr: 5.01e-04 [09/20 08:42:30] lb.utils.events INFO: eta: 1 day, 4:16:47 iteration: 188799/375342 consumed_samples: 193331200 total_loss: 0.4042 time: 0.5422 s/iter data_time: 0.0524 s/iter total_throughput: 1888.76 samples/s lr: 5.00e-04 [09/20 08:43:25] lb.utils.events INFO: eta: 1 day, 4:15:18 iteration: 188899/375342 consumed_samples: 193433600 total_loss: 0.4111 time: 0.5422 s/iter data_time: 0.0518 s/iter total_throughput: 1888.75 samples/s lr: 5.00e-04 [09/20 08:44:19] lb.utils.events INFO: eta: 1 day, 4:13:48 iteration: 188999/375342 consumed_samples: 193536000 total_loss: 0.4192 time: 0.5422 s/iter data_time: 0.0508 s/iter total_throughput: 1888.75 samples/s lr: 4.99e-04 [09/20 08:45:14] lb.utils.events INFO: eta: 1 day, 4:12:03 iteration: 189099/375342 consumed_samples: 193638400 total_loss: 0.4184 time: 0.5422 s/iter data_time: 0.0526 s/iter total_throughput: 1888.75 samples/s lr: 4.99e-04 [09/20 08:46:08] lb.utils.events INFO: eta: 1 day, 4:10:41 iteration: 189199/375342 consumed_samples: 193740800 total_loss: 0.4085 time: 0.5422 s/iter data_time: 0.0520 s/iter total_throughput: 1888.74 samples/s lr: 4.99e-04 [09/20 08:47:02] lb.utils.events INFO: eta: 1 day, 4:09:02 iteration: 189299/375342 consumed_samples: 193843200 total_loss: 0.4151 time: 0.5422 s/iter data_time: 0.0507 s/iter total_throughput: 1888.74 samples/s lr: 4.98e-04 [09/20 08:47:57] lb.utils.events INFO: eta: 1 day, 4:07:47 iteration: 189399/375342 consumed_samples: 193945600 total_loss: 0.4147 time: 0.5422 s/iter data_time: 0.0467 s/iter total_throughput: 1888.73 samples/s lr: 4.98e-04 [09/20 08:48:51] lb.utils.events INFO: eta: 1 day, 4:06:52 iteration: 189499/375342 consumed_samples: 194048000 total_loss: 0.3982 time: 0.5422 s/iter data_time: 0.0497 s/iter total_throughput: 1888.73 samples/s lr: 4.97e-04 [09/20 08:49:46] lb.utils.events INFO: eta: 1 day, 4:05:56 iteration: 189599/375342 consumed_samples: 194150400 total_loss: 0.4035 time: 0.5422 s/iter data_time: 0.0497 s/iter total_throughput: 1888.72 samples/s lr: 4.97e-04 [09/20 08:50:41] lb.utils.events INFO: eta: 1 day, 4:04:50 iteration: 189699/375342 consumed_samples: 194252800 total_loss: 0.4063 time: 0.5422 s/iter data_time: 0.0502 s/iter total_throughput: 1888.72 samples/s lr: 4.97e-04 [09/20 08:51:36] lb.utils.events INFO: eta: 1 day, 4:04:40 iteration: 189799/375342 consumed_samples: 194355200 total_loss: 0.4079 time: 0.5422 s/iter data_time: 0.0534 s/iter total_throughput: 1888.70 samples/s lr: 4.96e-04 [09/20 08:52:31] lb.utils.events INFO: eta: 1 day, 4:04:54 iteration: 189899/375342 consumed_samples: 194457600 total_loss: 0.4113 time: 0.5422 s/iter data_time: 0.0553 s/iter total_throughput: 1888.69 samples/s lr: 4.96e-04 [09/20 08:53:25] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0189999 [09/20 08:53:26] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 08:53:26] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 08:53:31] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0693 s/iter. Inference: 0.2421 s/iter. Eval: 0.0022 s/iter. Total: 0.3136 s/iter. ETA=0:00:11 [09/20 08:53:36] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0726 s/iter. Inference: 0.2579 s/iter. Eval: 0.0022 s/iter. Total: 0.3332 s/iter. ETA=0:00:07 [09/20 08:53:41] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0713 s/iter. Inference: 0.2555 s/iter. Eval: 0.0022 s/iter. Total: 0.3296 s/iter. ETA=0:00:01 [09/20 08:53:43] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 08:53:43] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.184224 (0.000284 s / iter per device, on 8 devices) [09/20 08:53:43] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/20 08:53:43] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 08:53:43] lb.evaluation.utils INFO: copypaste: Acc@1=74.968 [09/20 08:53:43] lb.evaluation.utils INFO: copypaste: Acc@5=92.524 [09/20 08:53:43] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 74.96800, better than last best score 74.77400 @ iteration 184999. [09/20 08:53:43] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 08:53:44] lb.utils.events INFO: eta: 1 day, 4:05:38 iteration: 189999/375342 consumed_samples: 194560000 total_loss: 0.4121 time: 0.5422 s/iter data_time: 0.0524 s/iter total_throughput: 1888.68 samples/s lr: 4.95e-04 [09/20 08:54:38] lb.utils.events INFO: eta: 1 day, 4:05:39 iteration: 190099/375342 consumed_samples: 194662400 total_loss: 0.4121 time: 0.5422 s/iter data_time: 0.0560 s/iter total_throughput: 1888.67 samples/s lr: 4.95e-04 [09/20 08:55:34] lb.utils.events INFO: eta: 1 day, 4:06:18 iteration: 190199/375342 consumed_samples: 194764800 total_loss: 0.4105 time: 0.5422 s/iter data_time: 0.0556 s/iter total_throughput: 1888.65 samples/s lr: 4.95e-04 [09/20 08:56:29] lb.utils.events INFO: eta: 1 day, 4:07:36 iteration: 190299/375342 consumed_samples: 194867200 total_loss: 0.4067 time: 0.5422 s/iter data_time: 0.0504 s/iter total_throughput: 1888.64 samples/s lr: 4.94e-04 [09/20 08:57:24] lb.utils.events INFO: eta: 1 day, 4:08:38 iteration: 190399/375342 consumed_samples: 194969600 total_loss: 0.4054 time: 0.5422 s/iter data_time: 0.0504 s/iter total_throughput: 1888.62 samples/s lr: 4.94e-04 [09/20 08:58:19] lb.utils.events INFO: eta: 1 day, 4:09:32 iteration: 190499/375342 consumed_samples: 195072000 total_loss: 0.4102 time: 0.5422 s/iter data_time: 0.0491 s/iter total_throughput: 1888.61 samples/s lr: 4.93e-04 [09/20 08:59:14] lb.utils.events INFO: eta: 1 day, 4:10:37 iteration: 190599/375342 consumed_samples: 195174400 total_loss: 0.4129 time: 0.5422 s/iter data_time: 0.0538 s/iter total_throughput: 1888.59 samples/s lr: 4.93e-04 [09/20 09:00:09] lb.utils.events INFO: eta: 1 day, 4:11:58 iteration: 190699/375342 consumed_samples: 195276800 total_loss: 0.4124 time: 0.5422 s/iter data_time: 0.0498 s/iter total_throughput: 1888.58 samples/s lr: 4.92e-04 [09/20 09:01:04] lb.utils.events INFO: eta: 1 day, 4:11:58 iteration: 190799/375342 consumed_samples: 195379200 total_loss: 0.4118 time: 0.5422 s/iter data_time: 0.0531 s/iter total_throughput: 1888.56 samples/s lr: 4.92e-04 [09/20 09:01:59] lb.utils.events INFO: eta: 1 day, 4:11:56 iteration: 190899/375342 consumed_samples: 195481600 total_loss: 0.4136 time: 0.5422 s/iter data_time: 0.0534 s/iter total_throughput: 1888.55 samples/s lr: 4.92e-04 [09/20 09:02:54] lb.utils.events INFO: eta: 1 day, 4:11:40 iteration: 190999/375342 consumed_samples: 195584000 total_loss: 0.4169 time: 0.5422 s/iter data_time: 0.0480 s/iter total_throughput: 1888.54 samples/s lr: 4.91e-04 [09/20 09:03:49] lb.utils.events INFO: eta: 1 day, 4:11:46 iteration: 191099/375342 consumed_samples: 195686400 total_loss: 0.4122 time: 0.5422 s/iter data_time: 0.0455 s/iter total_throughput: 1888.52 samples/s lr: 4.91e-04 [09/20 09:04:44] lb.utils.events INFO: eta: 1 day, 4:11:10 iteration: 191199/375342 consumed_samples: 195788800 total_loss: 0.4056 time: 0.5422 s/iter data_time: 0.0496 s/iter total_throughput: 1888.51 samples/s lr: 4.90e-04 [09/20 09:05:39] lb.utils.events INFO: eta: 1 day, 4:09:40 iteration: 191299/375342 consumed_samples: 195891200 total_loss: 0.4098 time: 0.5422 s/iter data_time: 0.0504 s/iter total_throughput: 1888.49 samples/s lr: 4.90e-04 [09/20 09:06:34] lb.utils.events INFO: eta: 1 day, 4:07:53 iteration: 191399/375342 consumed_samples: 195993600 total_loss: 0.4156 time: 0.5422 s/iter data_time: 0.0512 s/iter total_throughput: 1888.48 samples/s lr: 4.90e-04 [09/20 09:07:29] lb.utils.events INFO: eta: 1 day, 4:06:24 iteration: 191499/375342 consumed_samples: 196096000 total_loss: 0.4105 time: 0.5422 s/iter data_time: 0.0511 s/iter total_throughput: 1888.47 samples/s lr: 4.89e-04 [09/20 09:08:24] lb.utils.events INFO: eta: 1 day, 4:04:33 iteration: 191599/375342 consumed_samples: 196198400 total_loss: 0.4088 time: 0.5422 s/iter data_time: 0.0511 s/iter total_throughput: 1888.46 samples/s lr: 4.89e-04 [09/20 09:09:19] lb.utils.events INFO: eta: 1 day, 4:02:25 iteration: 191699/375342 consumed_samples: 196300800 total_loss: 0.4068 time: 0.5422 s/iter data_time: 0.0515 s/iter total_throughput: 1888.45 samples/s lr: 4.88e-04 [09/20 09:10:14] lb.utils.events INFO: eta: 1 day, 4:00:57 iteration: 191799/375342 consumed_samples: 196403200 total_loss: 0.3986 time: 0.5422 s/iter data_time: 0.0521 s/iter total_throughput: 1888.44 samples/s lr: 4.88e-04 [09/20 09:11:08] lb.utils.events INFO: eta: 1 day, 3:58:55 iteration: 191899/375342 consumed_samples: 196505600 total_loss: 0.4073 time: 0.5423 s/iter data_time: 0.0520 s/iter total_throughput: 1888.42 samples/s lr: 4.87e-04 [09/20 09:12:03] lb.utils.events INFO: eta: 1 day, 3:56:56 iteration: 191999/375342 consumed_samples: 196608000 total_loss: 0.4112 time: 0.5423 s/iter data_time: 0.0529 s/iter total_throughput: 1888.42 samples/s lr: 4.87e-04 [09/20 09:12:58] lb.utils.events INFO: eta: 1 day, 3:54:22 iteration: 192099/375342 consumed_samples: 196710400 total_loss: 0.41 time: 0.5423 s/iter data_time: 0.0519 s/iter total_throughput: 1888.41 samples/s lr: 4.87e-04 [09/20 09:13:52] lb.utils.events INFO: eta: 1 day, 3:52:06 iteration: 192199/375342 consumed_samples: 196812800 total_loss: 0.4103 time: 0.5423 s/iter data_time: 0.0523 s/iter total_throughput: 1888.40 samples/s lr: 4.86e-04 [09/20 09:14:47] lb.utils.events INFO: eta: 1 day, 3:49:51 iteration: 192299/375342 consumed_samples: 196915200 total_loss: 0.4025 time: 0.5423 s/iter data_time: 0.0515 s/iter total_throughput: 1888.40 samples/s lr: 4.86e-04 [09/20 09:15:41] lb.utils.events INFO: eta: 1 day, 3:47:52 iteration: 192399/375342 consumed_samples: 197017600 total_loss: 0.4038 time: 0.5423 s/iter data_time: 0.0505 s/iter total_throughput: 1888.39 samples/s lr: 4.85e-04 [09/20 09:16:36] lb.utils.events INFO: eta: 1 day, 3:45:54 iteration: 192499/375342 consumed_samples: 197120000 total_loss: 0.4041 time: 0.5423 s/iter data_time: 0.0523 s/iter total_throughput: 1888.39 samples/s lr: 4.85e-04 [09/20 09:17:30] lb.utils.events INFO: eta: 1 day, 3:43:49 iteration: 192599/375342 consumed_samples: 197222400 total_loss: 0.4036 time: 0.5423 s/iter data_time: 0.0522 s/iter total_throughput: 1888.39 samples/s lr: 4.85e-04 [09/20 09:18:25] lb.utils.events INFO: eta: 1 day, 3:41:42 iteration: 192699/375342 consumed_samples: 197324800 total_loss: 0.4042 time: 0.5423 s/iter data_time: 0.0521 s/iter total_throughput: 1888.38 samples/s lr: 4.84e-04 [09/20 09:19:19] lb.utils.events INFO: eta: 1 day, 3:39:49 iteration: 192799/375342 consumed_samples: 197427200 total_loss: 0.41 time: 0.5423 s/iter data_time: 0.0470 s/iter total_throughput: 1888.38 samples/s lr: 4.84e-04 [09/20 09:20:14] lb.utils.events INFO: eta: 1 day, 3:37:55 iteration: 192899/375342 consumed_samples: 197529600 total_loss: 0.409 time: 0.5423 s/iter data_time: 0.0478 s/iter total_throughput: 1888.37 samples/s lr: 4.83e-04 [09/20 09:21:08] lb.utils.events INFO: eta: 1 day, 3:36:10 iteration: 192999/375342 consumed_samples: 197632000 total_loss: 0.407 time: 0.5423 s/iter data_time: 0.0503 s/iter total_throughput: 1888.37 samples/s lr: 4.83e-04 [09/20 09:22:03] lb.utils.events INFO: eta: 1 day, 3:34:56 iteration: 193099/375342 consumed_samples: 197734400 total_loss: 0.4071 time: 0.5423 s/iter data_time: 0.0493 s/iter total_throughput: 1888.36 samples/s lr: 4.83e-04 [09/20 09:22:58] lb.utils.events INFO: eta: 1 day, 3:34:14 iteration: 193199/375342 consumed_samples: 197836800 total_loss: 0.4088 time: 0.5423 s/iter data_time: 0.0549 s/iter total_throughput: 1888.35 samples/s lr: 4.82e-04 [09/20 09:23:53] lb.utils.events INFO: eta: 1 day, 3:34:14 iteration: 193299/375342 consumed_samples: 197939200 total_loss: 0.4063 time: 0.5423 s/iter data_time: 0.0543 s/iter total_throughput: 1888.34 samples/s lr: 4.82e-04 [09/20 09:24:48] lb.utils.events INFO: eta: 1 day, 3:34:16 iteration: 193399/375342 consumed_samples: 198041600 total_loss: 0.4024 time: 0.5423 s/iter data_time: 0.0557 s/iter total_throughput: 1888.33 samples/s lr: 4.81e-04 [09/20 09:25:42] lb.utils.events INFO: eta: 1 day, 3:34:42 iteration: 193499/375342 consumed_samples: 198144000 total_loss: 0.4023 time: 0.5423 s/iter data_time: 0.0523 s/iter total_throughput: 1888.32 samples/s lr: 4.81e-04 [09/20 09:26:37] lb.utils.events INFO: eta: 1 day, 3:35:42 iteration: 193599/375342 consumed_samples: 198246400 total_loss: 0.4076 time: 0.5423 s/iter data_time: 0.0535 s/iter total_throughput: 1888.30 samples/s lr: 4.80e-04 [09/20 09:27:33] lb.utils.events INFO: eta: 1 day, 3:37:14 iteration: 193699/375342 consumed_samples: 198348800 total_loss: 0.4111 time: 0.5423 s/iter data_time: 0.0516 s/iter total_throughput: 1888.29 samples/s lr: 4.80e-04 [09/20 09:28:28] lb.utils.events INFO: eta: 1 day, 3:38:38 iteration: 193799/375342 consumed_samples: 198451200 total_loss: 0.4128 time: 0.5423 s/iter data_time: 0.0547 s/iter total_throughput: 1888.27 samples/s lr: 4.80e-04 [09/20 09:29:23] lb.utils.events INFO: eta: 1 day, 3:39:56 iteration: 193899/375342 consumed_samples: 198553600 total_loss: 0.4118 time: 0.5423 s/iter data_time: 0.0539 s/iter total_throughput: 1888.26 samples/s lr: 4.79e-04 [09/20 09:30:18] lb.utils.events INFO: eta: 1 day, 3:41:05 iteration: 193999/375342 consumed_samples: 198656000 total_loss: 0.4087 time: 0.5423 s/iter data_time: 0.0539 s/iter total_throughput: 1888.24 samples/s lr: 4.79e-04 [09/20 09:31:13] lb.utils.events INFO: eta: 1 day, 3:41:27 iteration: 194099/375342 consumed_samples: 198758400 total_loss: 0.4046 time: 0.5423 s/iter data_time: 0.0542 s/iter total_throughput: 1888.23 samples/s lr: 4.78e-04 [09/20 09:32:08] lb.utils.events INFO: eta: 1 day, 3:40:39 iteration: 194199/375342 consumed_samples: 198860800 total_loss: 0.4048 time: 0.5423 s/iter data_time: 0.0537 s/iter total_throughput: 1888.22 samples/s lr: 4.78e-04 [09/20 09:33:02] lb.utils.events INFO: eta: 1 day, 3:39:16 iteration: 194299/375342 consumed_samples: 198963200 total_loss: 0.404 time: 0.5423 s/iter data_time: 0.0537 s/iter total_throughput: 1888.21 samples/s lr: 4.78e-04 [09/20 09:33:57] lb.utils.events INFO: eta: 1 day, 3:37:18 iteration: 194399/375342 consumed_samples: 199065600 total_loss: 0.4075 time: 0.5423 s/iter data_time: 0.0542 s/iter total_throughput: 1888.20 samples/s lr: 4.77e-04 [09/20 09:34:52] lb.utils.events INFO: eta: 1 day, 3:35:23 iteration: 194499/375342 consumed_samples: 199168000 total_loss: 0.408 time: 0.5423 s/iter data_time: 0.0502 s/iter total_throughput: 1888.20 samples/s lr: 4.77e-04 [09/20 09:35:47] lb.utils.events INFO: eta: 1 day, 3:33:44 iteration: 194599/375342 consumed_samples: 199270400 total_loss: 0.4022 time: 0.5423 s/iter data_time: 0.0494 s/iter total_throughput: 1888.19 samples/s lr: 4.76e-04 [09/20 09:36:41] lb.utils.events INFO: eta: 1 day, 3:31:30 iteration: 194699/375342 consumed_samples: 199372800 total_loss: 0.4011 time: 0.5423 s/iter data_time: 0.0513 s/iter total_throughput: 1888.18 samples/s lr: 4.76e-04 [09/20 09:37:36] lb.utils.events INFO: eta: 1 day, 3:28:31 iteration: 194799/375342 consumed_samples: 199475200 total_loss: 0.4033 time: 0.5423 s/iter data_time: 0.0512 s/iter total_throughput: 1888.17 samples/s lr: 4.75e-04 [09/20 09:38:31] lb.utils.events INFO: eta: 1 day, 3:25:57 iteration: 194899/375342 consumed_samples: 199577600 total_loss: 0.4066 time: 0.5423 s/iter data_time: 0.0507 s/iter total_throughput: 1888.16 samples/s lr: 4.75e-04 [09/20 09:39:25] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0194999 [09/20 09:39:26] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 09:39:26] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 09:39:30] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0654 s/iter. Inference: 0.2419 s/iter. Eval: 0.0031 s/iter. Total: 0.3105 s/iter. ETA=0:00:11 [09/20 09:39:36] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0668 s/iter. Inference: 0.2668 s/iter. Eval: 0.0024 s/iter. Total: 0.3363 s/iter. ETA=0:00:07 [09/20 09:39:41] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0675 s/iter. Inference: 0.2594 s/iter. Eval: 0.0024 s/iter. Total: 0.3297 s/iter. ETA=0:00:01 [09/20 09:39:43] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 09:39:43] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.223860 (0.000285 s / iter per device, on 8 devices) [09/20 09:39:43] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000226 s / iter per device, on 8 devices) [09/20 09:39:43] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 09:39:43] lb.evaluation.utils INFO: copypaste: Acc@1=75.278 [09/20 09:39:43] lb.evaluation.utils INFO: copypaste: Acc@5=92.61 [09/20 09:39:43] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 75.27800, better than last best score 74.96800 @ iteration 189999. [09/20 09:39:43] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 09:39:43] lb.utils.events INFO: eta: 1 day, 3:24:18 iteration: 194999/375342 consumed_samples: 199680000 total_loss: 0.4074 time: 0.5423 s/iter data_time: 0.0513 s/iter total_throughput: 1888.16 samples/s lr: 4.75e-04 [09/20 09:40:38] lb.utils.events INFO: eta: 1 day, 3:22:43 iteration: 195099/375342 consumed_samples: 199782400 total_loss: 0.4083 time: 0.5423 s/iter data_time: 0.0510 s/iter total_throughput: 1888.15 samples/s lr: 4.74e-04 [09/20 09:41:33] lb.utils.events INFO: eta: 1 day, 3:21:49 iteration: 195199/375342 consumed_samples: 199884800 total_loss: 0.4109 time: 0.5423 s/iter data_time: 0.0498 s/iter total_throughput: 1888.14 samples/s lr: 4.74e-04 [09/20 09:42:27] lb.utils.events INFO: eta: 1 day, 3:20:34 iteration: 195299/375342 consumed_samples: 199987200 total_loss: 0.4096 time: 0.5423 s/iter data_time: 0.0514 s/iter total_throughput: 1888.13 samples/s lr: 4.73e-04 [09/20 09:43:22] lb.utils.events INFO: eta: 1 day, 3:19:37 iteration: 195399/375342 consumed_samples: 200089600 total_loss: 0.4009 time: 0.5423 s/iter data_time: 0.0523 s/iter total_throughput: 1888.13 samples/s lr: 4.73e-04 [09/20 09:44:17] lb.utils.events INFO: eta: 1 day, 3:19:15 iteration: 195499/375342 consumed_samples: 200192000 total_loss: 0.4024 time: 0.5423 s/iter data_time: 0.0522 s/iter total_throughput: 1888.12 samples/s lr: 4.73e-04 [09/20 09:45:12] lb.utils.events INFO: eta: 1 day, 3:18:11 iteration: 195599/375342 consumed_samples: 200294400 total_loss: 0.4097 time: 0.5423 s/iter data_time: 0.0524 s/iter total_throughput: 1888.11 samples/s lr: 4.72e-04 [09/20 09:46:06] lb.utils.events INFO: eta: 1 day, 3:16:53 iteration: 195699/375342 consumed_samples: 200396800 total_loss: 0.4089 time: 0.5423 s/iter data_time: 0.0524 s/iter total_throughput: 1888.10 samples/s lr: 4.72e-04 [09/20 09:47:01] lb.utils.events INFO: eta: 1 day, 3:15:37 iteration: 195799/375342 consumed_samples: 200499200 total_loss: 0.4101 time: 0.5423 s/iter data_time: 0.0512 s/iter total_throughput: 1888.10 samples/s lr: 4.71e-04 [09/20 09:47:55] lb.utils.events INFO: eta: 1 day, 3:14:18 iteration: 195899/375342 consumed_samples: 200601600 total_loss: 0.4099 time: 0.5423 s/iter data_time: 0.0499 s/iter total_throughput: 1888.09 samples/s lr: 4.71e-04 [09/20 09:48:50] lb.utils.events INFO: eta: 1 day, 3:12:34 iteration: 195999/375342 consumed_samples: 200704000 total_loss: 0.4056 time: 0.5423 s/iter data_time: 0.0515 s/iter total_throughput: 1888.09 samples/s lr: 4.71e-04 [09/20 09:49:44] lb.utils.events INFO: eta: 1 day, 3:11:16 iteration: 196099/375342 consumed_samples: 200806400 total_loss: 0.4033 time: 0.5423 s/iter data_time: 0.0517 s/iter total_throughput: 1888.08 samples/s lr: 4.70e-04 [09/20 09:50:39] lb.utils.events INFO: eta: 1 day, 3:09:12 iteration: 196199/375342 consumed_samples: 200908800 total_loss: 0.4045 time: 0.5423 s/iter data_time: 0.0532 s/iter total_throughput: 1888.08 samples/s lr: 4.70e-04 [09/20 09:51:33] lb.utils.events INFO: eta: 1 day, 3:07:55 iteration: 196299/375342 consumed_samples: 201011200 total_loss: 0.407 time: 0.5424 s/iter data_time: 0.0498 s/iter total_throughput: 1888.08 samples/s lr: 4.69e-04 [09/20 09:52:28] lb.utils.events INFO: eta: 1 day, 3:06:11 iteration: 196399/375342 consumed_samples: 201113600 total_loss: 0.4073 time: 0.5424 s/iter data_time: 0.0490 s/iter total_throughput: 1888.07 samples/s lr: 4.69e-04 [09/20 09:53:22] lb.utils.events INFO: eta: 1 day, 3:04:06 iteration: 196499/375342 consumed_samples: 201216000 total_loss: 0.4099 time: 0.5424 s/iter data_time: 0.0487 s/iter total_throughput: 1888.07 samples/s lr: 4.68e-04 [09/20 09:54:17] lb.utils.events INFO: eta: 1 day, 3:02:31 iteration: 196599/375342 consumed_samples: 201318400 total_loss: 0.399 time: 0.5424 s/iter data_time: 0.0504 s/iter total_throughput: 1888.07 samples/s lr: 4.68e-04 [09/20 09:55:12] lb.utils.events INFO: eta: 1 day, 3:01:46 iteration: 196699/375342 consumed_samples: 201420800 total_loss: 0.3984 time: 0.5424 s/iter data_time: 0.0545 s/iter total_throughput: 1888.05 samples/s lr: 4.68e-04 [09/20 09:56:06] lb.utils.events INFO: eta: 1 day, 3:01:25 iteration: 196799/375342 consumed_samples: 201523200 total_loss: 0.4037 time: 0.5424 s/iter data_time: 0.0535 s/iter total_throughput: 1888.04 samples/s lr: 4.67e-04 [09/20 09:57:01] lb.utils.events INFO: eta: 1 day, 3:01:01 iteration: 196899/375342 consumed_samples: 201625600 total_loss: 0.4037 time: 0.5424 s/iter data_time: 0.0550 s/iter total_throughput: 1888.04 samples/s lr: 4.67e-04 [09/20 09:57:56] lb.utils.events INFO: eta: 1 day, 3:01:19 iteration: 196999/375342 consumed_samples: 201728000 total_loss: 0.4041 time: 0.5424 s/iter data_time: 0.0554 s/iter total_throughput: 1888.03 samples/s lr: 4.66e-04 [09/20 09:58:51] lb.utils.events INFO: eta: 1 day, 3:01:11 iteration: 197099/375342 consumed_samples: 201830400 total_loss: 0.4043 time: 0.5424 s/iter data_time: 0.0507 s/iter total_throughput: 1888.02 samples/s lr: 4.66e-04 [09/20 09:59:46] lb.utils.events INFO: eta: 1 day, 3:01:50 iteration: 197199/375342 consumed_samples: 201932800 total_loss: 0.4082 time: 0.5424 s/iter data_time: 0.0555 s/iter total_throughput: 1888.01 samples/s lr: 4.66e-04 [09/20 10:00:41] lb.utils.events INFO: eta: 1 day, 3:02:28 iteration: 197299/375342 consumed_samples: 202035200 total_loss: 0.4069 time: 0.5424 s/iter data_time: 0.0537 s/iter total_throughput: 1887.99 samples/s lr: 4.65e-04 [09/20 10:01:35] lb.utils.events INFO: eta: 1 day, 3:03:12 iteration: 197399/375342 consumed_samples: 202137600 total_loss: 0.4092 time: 0.5424 s/iter data_time: 0.0542 s/iter total_throughput: 1887.98 samples/s lr: 4.65e-04 [09/20 10:02:30] lb.utils.events INFO: eta: 1 day, 3:03:18 iteration: 197499/375342 consumed_samples: 202240000 total_loss: 0.4096 time: 0.5424 s/iter data_time: 0.0534 s/iter total_throughput: 1887.97 samples/s lr: 4.64e-04 [09/20 10:03:25] lb.utils.events INFO: eta: 1 day, 3:03:50 iteration: 197599/375342 consumed_samples: 202342400 total_loss: 0.3994 time: 0.5424 s/iter data_time: 0.0535 s/iter total_throughput: 1887.96 samples/s lr: 4.64e-04 [09/20 10:04:20] lb.utils.events INFO: eta: 1 day, 3:02:46 iteration: 197699/375342 consumed_samples: 202444800 total_loss: 0.4049 time: 0.5424 s/iter data_time: 0.0554 s/iter total_throughput: 1887.95 samples/s lr: 4.64e-04 [09/20 10:05:15] lb.utils.events INFO: eta: 1 day, 3:01:51 iteration: 197799/375342 consumed_samples: 202547200 total_loss: 0.4091 time: 0.5424 s/iter data_time: 0.0549 s/iter total_throughput: 1887.95 samples/s lr: 4.63e-04 [09/20 10:06:09] lb.utils.events INFO: eta: 1 day, 3:01:02 iteration: 197899/375342 consumed_samples: 202649600 total_loss: 0.4094 time: 0.5424 s/iter data_time: 0.0497 s/iter total_throughput: 1887.94 samples/s lr: 4.63e-04 [09/20 10:07:04] lb.utils.events INFO: eta: 1 day, 3:00:38 iteration: 197999/375342 consumed_samples: 202752000 total_loss: 0.4054 time: 0.5424 s/iter data_time: 0.0491 s/iter total_throughput: 1887.93 samples/s lr: 4.62e-04 [09/20 10:07:59] lb.utils.events INFO: eta: 1 day, 2:59:20 iteration: 198099/375342 consumed_samples: 202854400 total_loss: 0.3978 time: 0.5424 s/iter data_time: 0.0481 s/iter total_throughput: 1887.92 samples/s lr: 4.62e-04 [09/20 10:08:54] lb.utils.events INFO: eta: 1 day, 2:57:50 iteration: 198199/375342 consumed_samples: 202956800 total_loss: 0.4055 time: 0.5424 s/iter data_time: 0.0501 s/iter total_throughput: 1887.91 samples/s lr: 4.61e-04 [09/20 10:09:48] lb.utils.events INFO: eta: 1 day, 2:56:36 iteration: 198299/375342 consumed_samples: 203059200 total_loss: 0.4026 time: 0.5424 s/iter data_time: 0.0515 s/iter total_throughput: 1887.90 samples/s lr: 4.61e-04 [09/20 10:10:43] lb.utils.events INFO: eta: 1 day, 2:54:54 iteration: 198399/375342 consumed_samples: 203161600 total_loss: 0.4001 time: 0.5424 s/iter data_time: 0.0505 s/iter total_throughput: 1887.89 samples/s lr: 4.61e-04 [09/20 10:11:38] lb.utils.events INFO: eta: 1 day, 2:53:25 iteration: 198499/375342 consumed_samples: 203264000 total_loss: 0.4057 time: 0.5424 s/iter data_time: 0.0504 s/iter total_throughput: 1887.89 samples/s lr: 4.60e-04 [09/20 10:12:32] lb.utils.events INFO: eta: 1 day, 2:51:51 iteration: 198599/375342 consumed_samples: 203366400 total_loss: 0.4042 time: 0.5424 s/iter data_time: 0.0500 s/iter total_throughput: 1887.88 samples/s lr: 4.60e-04 [09/20 10:13:27] lb.utils.events INFO: eta: 1 day, 2:50:37 iteration: 198699/375342 consumed_samples: 203468800 total_loss: 0.4079 time: 0.5424 s/iter data_time: 0.0521 s/iter total_throughput: 1887.87 samples/s lr: 4.59e-04 [09/20 10:14:22] lb.utils.events INFO: eta: 1 day, 2:49:22 iteration: 198799/375342 consumed_samples: 203571200 total_loss: 0.4095 time: 0.5424 s/iter data_time: 0.0495 s/iter total_throughput: 1887.87 samples/s lr: 4.59e-04 [09/20 10:15:16] lb.utils.events INFO: eta: 1 day, 2:47:54 iteration: 198899/375342 consumed_samples: 203673600 total_loss: 0.4042 time: 0.5424 s/iter data_time: 0.0496 s/iter total_throughput: 1887.86 samples/s lr: 4.59e-04 [09/20 10:16:11] lb.utils.events INFO: eta: 1 day, 2:46:09 iteration: 198999/375342 consumed_samples: 203776000 total_loss: 0.4028 time: 0.5424 s/iter data_time: 0.0518 s/iter total_throughput: 1887.85 samples/s lr: 4.58e-04 [09/20 10:17:05] lb.utils.events INFO: eta: 1 day, 2:44:04 iteration: 199099/375342 consumed_samples: 203878400 total_loss: 0.4061 time: 0.5424 s/iter data_time: 0.0517 s/iter total_throughput: 1887.85 samples/s lr: 4.58e-04 [09/20 10:18:00] lb.utils.events INFO: eta: 1 day, 2:42:42 iteration: 199199/375342 consumed_samples: 203980800 total_loss: 0.4106 time: 0.5424 s/iter data_time: 0.0513 s/iter total_throughput: 1887.85 samples/s lr: 4.57e-04 [09/20 10:18:54] lb.utils.events INFO: eta: 1 day, 2:40:55 iteration: 199299/375342 consumed_samples: 204083200 total_loss: 0.4127 time: 0.5424 s/iter data_time: 0.0529 s/iter total_throughput: 1887.84 samples/s lr: 4.57e-04 [09/20 10:19:49] lb.utils.events INFO: eta: 1 day, 2:39:28 iteration: 199399/375342 consumed_samples: 204185600 total_loss: 0.4099 time: 0.5424 s/iter data_time: 0.0516 s/iter total_throughput: 1887.84 samples/s lr: 4.56e-04 [09/20 10:20:43] lb.utils.events INFO: eta: 1 day, 2:38:16 iteration: 199499/375342 consumed_samples: 204288000 total_loss: 0.405 time: 0.5424 s/iter data_time: 0.0520 s/iter total_throughput: 1887.84 samples/s lr: 4.56e-04 [09/20 10:21:38] lb.utils.events INFO: eta: 1 day, 2:36:42 iteration: 199599/375342 consumed_samples: 204390400 total_loss: 0.4018 time: 0.5424 s/iter data_time: 0.0509 s/iter total_throughput: 1887.83 samples/s lr: 4.56e-04 [09/20 10:22:32] lb.utils.events INFO: eta: 1 day, 2:35:01 iteration: 199699/375342 consumed_samples: 204492800 total_loss: 0.4021 time: 0.5424 s/iter data_time: 0.0516 s/iter total_throughput: 1887.83 samples/s lr: 4.55e-04 [09/20 10:23:27] lb.utils.events INFO: eta: 1 day, 2:33:59 iteration: 199799/375342 consumed_samples: 204595200 total_loss: 0.4042 time: 0.5424 s/iter data_time: 0.0487 s/iter total_throughput: 1887.83 samples/s lr: 4.55e-04 [09/20 10:24:21] lb.utils.events INFO: eta: 1 day, 2:33:06 iteration: 199899/375342 consumed_samples: 204697600 total_loss: 0.4079 time: 0.5424 s/iter data_time: 0.0517 s/iter total_throughput: 1887.82 samples/s lr: 4.54e-04 [09/20 10:25:16] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0199999 [09/20 10:25:16] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 10:25:16] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 10:25:21] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0590 s/iter. Inference: 0.2513 s/iter. Eval: 0.0025 s/iter. Total: 0.3128 s/iter. ETA=0:00:11 [09/20 10:25:26] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0733 s/iter. Inference: 0.2585 s/iter. Eval: 0.0025 s/iter. Total: 0.3346 s/iter. ETA=0:00:07 [09/20 10:25:32] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0746 s/iter. Inference: 0.2548 s/iter. Eval: 0.0026 s/iter. Total: 0.3323 s/iter. ETA=0:00:01 [09/20 10:25:33] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 10:25:33] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.245818 (0.000285 s / iter per device, on 8 devices) [09/20 10:25:33] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000222 s / iter per device, on 8 devices) [09/20 10:25:33] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 10:25:33] lb.evaluation.utils INFO: copypaste: Acc@1=75.254 [09/20 10:25:33] lb.evaluation.utils INFO: copypaste: Acc@5=92.694 [09/20 10:25:34] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 75.25400, not better than best score 75.27800 @ iteration 194999. [09/20 10:25:34] lb.utils.events INFO: eta: 1 day, 2:32:06 iteration: 199999/375342 consumed_samples: 204800000 total_loss: 0.4088 time: 0.5424 s/iter data_time: 0.0511 s/iter total_throughput: 1887.81 samples/s lr: 4.54e-04 [09/20 10:26:28] lb.utils.events INFO: eta: 1 day, 2:31:36 iteration: 200099/375342 consumed_samples: 204902400 total_loss: 0.4045 time: 0.5424 s/iter data_time: 0.0542 s/iter total_throughput: 1887.80 samples/s lr: 4.54e-04 [09/20 10:27:23] lb.utils.events INFO: eta: 1 day, 2:31:22 iteration: 200199/375342 consumed_samples: 205004800 total_loss: 0.4001 time: 0.5424 s/iter data_time: 0.0560 s/iter total_throughput: 1887.79 samples/s lr: 4.53e-04 [09/20 10:28:18] lb.utils.events INFO: eta: 1 day, 2:31:31 iteration: 200299/375342 consumed_samples: 205107200 total_loss: 0.4044 time: 0.5424 s/iter data_time: 0.0537 s/iter total_throughput: 1887.78 samples/s lr: 4.53e-04 [09/20 10:29:13] lb.utils.events INFO: eta: 1 day, 2:31:57 iteration: 200399/375342 consumed_samples: 205209600 total_loss: 0.4092 time: 0.5424 s/iter data_time: 0.0563 s/iter total_throughput: 1887.77 samples/s lr: 4.52e-04 [09/20 10:30:08] lb.utils.events INFO: eta: 1 day, 2:31:46 iteration: 200499/375342 consumed_samples: 205312000 total_loss: 0.4119 time: 0.5424 s/iter data_time: 0.0567 s/iter total_throughput: 1887.77 samples/s lr: 4.52e-04 [09/20 10:31:03] lb.utils.events INFO: eta: 1 day, 2:32:32 iteration: 200599/375342 consumed_samples: 205414400 total_loss: 0.4119 time: 0.5424 s/iter data_time: 0.0574 s/iter total_throughput: 1887.75 samples/s lr: 4.52e-04 [09/20 10:31:58] lb.utils.events INFO: eta: 1 day, 2:33:25 iteration: 200699/375342 consumed_samples: 205516800 total_loss: 0.4071 time: 0.5424 s/iter data_time: 0.0544 s/iter total_throughput: 1887.74 samples/s lr: 4.51e-04 [09/20 10:32:53] lb.utils.events INFO: eta: 1 day, 2:33:45 iteration: 200799/375342 consumed_samples: 205619200 total_loss: 0.3995 time: 0.5425 s/iter data_time: 0.0543 s/iter total_throughput: 1887.73 samples/s lr: 4.51e-04 [09/20 10:33:48] lb.utils.events INFO: eta: 1 day, 2:33:48 iteration: 200899/375342 consumed_samples: 205721600 total_loss: 0.4044 time: 0.5425 s/iter data_time: 0.0564 s/iter total_throughput: 1887.72 samples/s lr: 4.50e-04 [09/20 10:34:42] lb.utils.events INFO: eta: 1 day, 2:33:32 iteration: 200999/375342 consumed_samples: 205824000 total_loss: 0.4038 time: 0.5425 s/iter data_time: 0.0551 s/iter total_throughput: 1887.71 samples/s lr: 4.50e-04 [09/20 10:35:37] lb.utils.events INFO: eta: 1 day, 2:33:25 iteration: 201099/375342 consumed_samples: 205926400 total_loss: 0.4083 time: 0.5425 s/iter data_time: 0.0559 s/iter total_throughput: 1887.70 samples/s lr: 4.49e-04 [09/20 10:36:32] lb.utils.events INFO: eta: 1 day, 2:32:12 iteration: 201199/375342 consumed_samples: 206028800 total_loss: 0.4084 time: 0.5425 s/iter data_time: 0.0551 s/iter total_throughput: 1887.69 samples/s lr: 4.49e-04 [09/20 10:37:27] lb.utils.events INFO: eta: 1 day, 2:31:07 iteration: 201299/375342 consumed_samples: 206131200 total_loss: 0.3991 time: 0.5425 s/iter data_time: 0.0544 s/iter total_throughput: 1887.68 samples/s lr: 4.49e-04 [09/20 10:38:22] lb.utils.events INFO: eta: 1 day, 2:29:51 iteration: 201399/375342 consumed_samples: 206233600 total_loss: 0.4026 time: 0.5425 s/iter data_time: 0.0501 s/iter total_throughput: 1887.67 samples/s lr: 4.48e-04 [09/20 10:39:16] lb.utils.events INFO: eta: 1 day, 2:29:03 iteration: 201499/375342 consumed_samples: 206336000 total_loss: 0.3977 time: 0.5425 s/iter data_time: 0.0509 s/iter total_throughput: 1887.66 samples/s lr: 4.48e-04 [09/20 10:40:11] lb.utils.events INFO: eta: 1 day, 2:27:55 iteration: 201599/375342 consumed_samples: 206438400 total_loss: 0.3957 time: 0.5425 s/iter data_time: 0.0513 s/iter total_throughput: 1887.65 samples/s lr: 4.47e-04 [09/20 10:41:06] lb.utils.events INFO: eta: 1 day, 2:26:38 iteration: 201699/375342 consumed_samples: 206540800 total_loss: 0.4039 time: 0.5425 s/iter data_time: 0.0526 s/iter total_throughput: 1887.64 samples/s lr: 4.47e-04 [09/20 10:42:01] lb.utils.events INFO: eta: 1 day, 2:25:11 iteration: 201799/375342 consumed_samples: 206643200 total_loss: 0.4014 time: 0.5425 s/iter data_time: 0.0507 s/iter total_throughput: 1887.63 samples/s lr: 4.47e-04 [09/20 10:42:56] lb.utils.events INFO: eta: 1 day, 2:23:26 iteration: 201899/375342 consumed_samples: 206745600 total_loss: 0.4046 time: 0.5425 s/iter data_time: 0.0504 s/iter total_throughput: 1887.62 samples/s lr: 4.46e-04 [09/20 10:43:50] lb.utils.events INFO: eta: 1 day, 2:22:17 iteration: 201999/375342 consumed_samples: 206848000 total_loss: 0.4055 time: 0.5425 s/iter data_time: 0.0515 s/iter total_throughput: 1887.62 samples/s lr: 4.46e-04 [09/20 10:44:45] lb.utils.events INFO: eta: 1 day, 2:21:36 iteration: 202099/375342 consumed_samples: 206950400 total_loss: 0.4033 time: 0.5425 s/iter data_time: 0.0520 s/iter total_throughput: 1887.61 samples/s lr: 4.45e-04 [09/20 10:45:40] lb.utils.events INFO: eta: 1 day, 2:20:38 iteration: 202199/375342 consumed_samples: 207052800 total_loss: 0.4097 time: 0.5425 s/iter data_time: 0.0525 s/iter total_throughput: 1887.60 samples/s lr: 4.45e-04 [09/20 10:46:35] lb.utils.events INFO: eta: 1 day, 2:19:50 iteration: 202299/375342 consumed_samples: 207155200 total_loss: 0.41 time: 0.5425 s/iter data_time: 0.0515 s/iter total_throughput: 1887.59 samples/s lr: 4.45e-04 [09/20 10:47:29] lb.utils.events INFO: eta: 1 day, 2:18:36 iteration: 202399/375342 consumed_samples: 207257600 total_loss: 0.3995 time: 0.5425 s/iter data_time: 0.0510 s/iter total_throughput: 1887.58 samples/s lr: 4.44e-04 [09/20 10:48:24] lb.utils.events INFO: eta: 1 day, 2:17:02 iteration: 202499/375342 consumed_samples: 207360000 total_loss: 0.401 time: 0.5425 s/iter data_time: 0.0509 s/iter total_throughput: 1887.58 samples/s lr: 4.44e-04 [09/20 10:49:19] lb.utils.events INFO: eta: 1 day, 2:15:15 iteration: 202599/375342 consumed_samples: 207462400 total_loss: 0.3999 time: 0.5425 s/iter data_time: 0.0516 s/iter total_throughput: 1887.57 samples/s lr: 4.43e-04 [09/20 10:50:13] lb.utils.events INFO: eta: 1 day, 2:13:13 iteration: 202699/375342 consumed_samples: 207564800 total_loss: 0.3995 time: 0.5425 s/iter data_time: 0.0509 s/iter total_throughput: 1887.57 samples/s lr: 4.43e-04 [09/20 10:51:08] lb.utils.events INFO: eta: 1 day, 2:11:40 iteration: 202799/375342 consumed_samples: 207667200 total_loss: 0.4088 time: 0.5425 s/iter data_time: 0.0510 s/iter total_throughput: 1887.56 samples/s lr: 4.42e-04 [09/20 10:52:02] lb.utils.events INFO: eta: 1 day, 2:10:22 iteration: 202899/375342 consumed_samples: 207769600 total_loss: 0.4097 time: 0.5425 s/iter data_time: 0.0514 s/iter total_throughput: 1887.56 samples/s lr: 4.42e-04 [09/20 10:52:57] lb.utils.events INFO: eta: 1 day, 2:09:12 iteration: 202999/375342 consumed_samples: 207872000 total_loss: 0.4076 time: 0.5425 s/iter data_time: 0.0530 s/iter total_throughput: 1887.55 samples/s lr: 4.42e-04 [09/20 10:53:51] lb.utils.events INFO: eta: 1 day, 2:07:19 iteration: 203099/375342 consumed_samples: 207974400 total_loss: 0.4047 time: 0.5425 s/iter data_time: 0.0521 s/iter total_throughput: 1887.55 samples/s lr: 4.41e-04 [09/20 10:54:46] lb.utils.events INFO: eta: 1 day, 2:06:11 iteration: 203199/375342 consumed_samples: 208076800 total_loss: 0.4006 time: 0.5425 s/iter data_time: 0.0485 s/iter total_throughput: 1887.54 samples/s lr: 4.41e-04 [09/20 10:55:40] lb.utils.events INFO: eta: 1 day, 2:04:36 iteration: 203299/375342 consumed_samples: 208179200 total_loss: 0.4062 time: 0.5425 s/iter data_time: 0.0487 s/iter total_throughput: 1887.54 samples/s lr: 4.40e-04 [09/20 10:56:35] lb.utils.events INFO: eta: 1 day, 2:03:10 iteration: 203399/375342 consumed_samples: 208281600 total_loss: 0.4047 time: 0.5425 s/iter data_time: 0.0494 s/iter total_throughput: 1887.53 samples/s lr: 4.40e-04 [09/20 10:57:30] lb.utils.events INFO: eta: 1 day, 2:01:55 iteration: 203499/375342 consumed_samples: 208384000 total_loss: 0.4048 time: 0.5425 s/iter data_time: 0.0513 s/iter total_throughput: 1887.53 samples/s lr: 4.40e-04 [09/20 10:58:25] lb.utils.events INFO: eta: 1 day, 2:01:25 iteration: 203599/375342 consumed_samples: 208486400 total_loss: 0.4042 time: 0.5425 s/iter data_time: 0.0555 s/iter total_throughput: 1887.52 samples/s lr: 4.39e-04 [09/20 10:59:19] lb.utils.events INFO: eta: 1 day, 2:01:24 iteration: 203699/375342 consumed_samples: 208588800 total_loss: 0.4024 time: 0.5425 s/iter data_time: 0.0535 s/iter total_throughput: 1887.51 samples/s lr: 4.39e-04 [09/20 11:00:14] lb.utils.events INFO: eta: 1 day, 2:00:59 iteration: 203799/375342 consumed_samples: 208691200 total_loss: 0.3974 time: 0.5425 s/iter data_time: 0.0555 s/iter total_throughput: 1887.50 samples/s lr: 4.38e-04 [09/20 11:01:09] lb.utils.events INFO: eta: 1 day, 2:00:24 iteration: 203899/375342 consumed_samples: 208793600 total_loss: 0.408 time: 0.5425 s/iter data_time: 0.0548 s/iter total_throughput: 1887.49 samples/s lr: 4.38e-04 [09/20 11:02:04] lb.utils.events INFO: eta: 1 day, 2:00:45 iteration: 203999/375342 consumed_samples: 208896000 total_loss: 0.4014 time: 0.5425 s/iter data_time: 0.0519 s/iter total_throughput: 1887.48 samples/s lr: 4.38e-04 [09/20 11:02:59] lb.utils.events INFO: eta: 1 day, 2:01:15 iteration: 204099/375342 consumed_samples: 208998400 total_loss: 0.3983 time: 0.5425 s/iter data_time: 0.0551 s/iter total_throughput: 1887.47 samples/s lr: 4.37e-04 [09/20 11:03:54] lb.utils.events INFO: eta: 1 day, 2:01:44 iteration: 204199/375342 consumed_samples: 209100800 total_loss: 0.4056 time: 0.5425 s/iter data_time: 0.0557 s/iter total_throughput: 1887.46 samples/s lr: 4.37e-04 [09/20 11:04:49] lb.utils.events INFO: eta: 1 day, 2:01:48 iteration: 204299/375342 consumed_samples: 209203200 total_loss: 0.4065 time: 0.5425 s/iter data_time: 0.0552 s/iter total_throughput: 1887.45 samples/s lr: 4.36e-04 [09/20 11:05:43] lb.utils.events INFO: eta: 1 day, 2:01:36 iteration: 204399/375342 consumed_samples: 209305600 total_loss: 0.4035 time: 0.5425 s/iter data_time: 0.0525 s/iter total_throughput: 1887.44 samples/s lr: 4.36e-04 [09/20 11:06:38] lb.utils.events INFO: eta: 1 day, 2:01:27 iteration: 204499/375342 consumed_samples: 209408000 total_loss: 0.3982 time: 0.5425 s/iter data_time: 0.0547 s/iter total_throughput: 1887.43 samples/s lr: 4.36e-04 [09/20 11:07:33] lb.utils.events INFO: eta: 1 day, 2:00:36 iteration: 204599/375342 consumed_samples: 209510400 total_loss: 0.3952 time: 0.5425 s/iter data_time: 0.0550 s/iter total_throughput: 1887.42 samples/s lr: 4.35e-04 [09/20 11:08:28] lb.utils.events INFO: eta: 1 day, 1:59:34 iteration: 204699/375342 consumed_samples: 209612800 total_loss: 0.3932 time: 0.5425 s/iter data_time: 0.0547 s/iter total_throughput: 1887.41 samples/s lr: 4.35e-04 [09/20 11:09:23] lb.utils.events INFO: eta: 1 day, 1:58:32 iteration: 204799/375342 consumed_samples: 209715200 total_loss: 0.3926 time: 0.5425 s/iter data_time: 0.0487 s/iter total_throughput: 1887.40 samples/s lr: 4.34e-04 [09/20 11:10:17] lb.utils.events INFO: eta: 1 day, 1:57:48 iteration: 204899/375342 consumed_samples: 209817600 total_loss: 0.3963 time: 0.5425 s/iter data_time: 0.0511 s/iter total_throughput: 1887.39 samples/s lr: 4.34e-04 [09/20 11:11:12] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0204999 [09/20 11:11:13] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 11:11:13] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 11:11:17] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0508 s/iter. Inference: 0.2478 s/iter. Eval: 0.0023 s/iter. Total: 0.3009 s/iter. ETA=0:00:11 [09/20 11:11:23] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0677 s/iter. Inference: 0.2644 s/iter. Eval: 0.0025 s/iter. Total: 0.3350 s/iter. ETA=0:00:07 [09/20 11:11:28] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0714 s/iter. Inference: 0.2572 s/iter. Eval: 0.0024 s/iter. Total: 0.3313 s/iter. ETA=0:00:01 [09/20 11:11:30] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 11:11:30] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.360908 (0.000287 s / iter per device, on 8 devices) [09/20 11:11:30] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/20 11:11:30] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 11:11:30] lb.evaluation.utils INFO: copypaste: Acc@1=75.69 [09/20 11:11:30] lb.evaluation.utils INFO: copypaste: Acc@5=92.96799999999999 [09/20 11:11:30] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 75.69000, better than last best score 75.27800 @ iteration 194999. [09/20 11:11:30] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 11:11:31] lb.utils.events INFO: eta: 1 day, 1:56:45 iteration: 204999/375342 consumed_samples: 209920000 total_loss: 0.4018 time: 0.5425 s/iter data_time: 0.0501 s/iter total_throughput: 1887.39 samples/s lr: 4.33e-04 [09/20 11:12:25] lb.utils.events INFO: eta: 1 day, 1:55:10 iteration: 205099/375342 consumed_samples: 210022400 total_loss: 0.4041 time: 0.5426 s/iter data_time: 0.0493 s/iter total_throughput: 1887.38 samples/s lr: 4.33e-04 [09/20 11:13:20] lb.utils.events INFO: eta: 1 day, 1:53:39 iteration: 205199/375342 consumed_samples: 210124800 total_loss: 0.4006 time: 0.5426 s/iter data_time: 0.0499 s/iter total_throughput: 1887.37 samples/s lr: 4.33e-04 [09/20 11:14:15] lb.utils.events INFO: eta: 1 day, 1:52:16 iteration: 205299/375342 consumed_samples: 210227200 total_loss: 0.392 time: 0.5426 s/iter data_time: 0.0497 s/iter total_throughput: 1887.36 samples/s lr: 4.32e-04 [09/20 11:15:09] lb.utils.events INFO: eta: 1 day, 1:51:17 iteration: 205399/375342 consumed_samples: 210329600 total_loss: 0.3946 time: 0.5426 s/iter data_time: 0.0514 s/iter total_throughput: 1887.36 samples/s lr: 4.32e-04 [09/20 11:16:04] lb.utils.events INFO: eta: 1 day, 1:50:07 iteration: 205499/375342 consumed_samples: 210432000 total_loss: 0.3967 time: 0.5426 s/iter data_time: 0.0496 s/iter total_throughput: 1887.35 samples/s lr: 4.31e-04 [09/20 11:16:59] lb.utils.events INFO: eta: 1 day, 1:49:26 iteration: 205599/375342 consumed_samples: 210534400 total_loss: 0.3935 time: 0.5426 s/iter data_time: 0.0516 s/iter total_throughput: 1887.34 samples/s lr: 4.31e-04 [09/20 11:17:54] lb.utils.events INFO: eta: 1 day, 1:48:31 iteration: 205699/375342 consumed_samples: 210636800 total_loss: 0.3984 time: 0.5426 s/iter data_time: 0.0522 s/iter total_throughput: 1887.33 samples/s lr: 4.31e-04 [09/20 11:18:49] lb.utils.events INFO: eta: 1 day, 1:47:11 iteration: 205799/375342 consumed_samples: 210739200 total_loss: 0.4091 time: 0.5426 s/iter data_time: 0.0518 s/iter total_throughput: 1887.32 samples/s lr: 4.30e-04 [09/20 11:19:43] lb.utils.events INFO: eta: 1 day, 1:46:08 iteration: 205899/375342 consumed_samples: 210841600 total_loss: 0.403 time: 0.5426 s/iter data_time: 0.0519 s/iter total_throughput: 1887.32 samples/s lr: 4.30e-04 [09/20 11:20:38] lb.utils.events INFO: eta: 1 day, 1:45:00 iteration: 205999/375342 consumed_samples: 210944000 total_loss: 0.4023 time: 0.5426 s/iter data_time: 0.0529 s/iter total_throughput: 1887.31 samples/s lr: 4.29e-04 [09/20 11:21:33] lb.utils.events INFO: eta: 1 day, 1:43:48 iteration: 206099/375342 consumed_samples: 211046400 total_loss: 0.4019 time: 0.5426 s/iter data_time: 0.0523 s/iter total_throughput: 1887.30 samples/s lr: 4.29e-04 [09/20 11:22:27] lb.utils.events INFO: eta: 1 day, 1:42:41 iteration: 206199/375342 consumed_samples: 211148800 total_loss: 0.4047 time: 0.5426 s/iter data_time: 0.0523 s/iter total_throughput: 1887.30 samples/s lr: 4.29e-04 [09/20 11:23:22] lb.utils.events INFO: eta: 1 day, 1:41:26 iteration: 206299/375342 consumed_samples: 211251200 total_loss: 0.4057 time: 0.5426 s/iter data_time: 0.0531 s/iter total_throughput: 1887.29 samples/s lr: 4.28e-04 [09/20 11:24:16] lb.utils.events INFO: eta: 1 day, 1:40:04 iteration: 206399/375342 consumed_samples: 211353600 total_loss: 0.3992 time: 0.5426 s/iter data_time: 0.0520 s/iter total_throughput: 1887.29 samples/s lr: 4.28e-04 [09/20 11:25:11] lb.utils.events INFO: eta: 1 day, 1:38:33 iteration: 206499/375342 consumed_samples: 211456000 total_loss: 0.4018 time: 0.5426 s/iter data_time: 0.0533 s/iter total_throughput: 1887.28 samples/s lr: 4.27e-04 [09/20 11:26:05] lb.utils.events INFO: eta: 1 day, 1:36:19 iteration: 206599/375342 consumed_samples: 211558400 total_loss: 0.4044 time: 0.5426 s/iter data_time: 0.0486 s/iter total_throughput: 1887.28 samples/s lr: 4.27e-04 [09/20 11:27:00] lb.utils.events INFO: eta: 1 day, 1:34:45 iteration: 206699/375342 consumed_samples: 211660800 total_loss: 0.4057 time: 0.5426 s/iter data_time: 0.0487 s/iter total_throughput: 1887.27 samples/s lr: 4.26e-04 [09/20 11:27:55] lb.utils.events INFO: eta: 1 day, 1:33:02 iteration: 206799/375342 consumed_samples: 211763200 total_loss: 0.4063 time: 0.5426 s/iter data_time: 0.0482 s/iter total_throughput: 1887.27 samples/s lr: 4.26e-04 [09/20 11:28:49] lb.utils.events INFO: eta: 1 day, 1:31:27 iteration: 206899/375342 consumed_samples: 211865600 total_loss: 0.3981 time: 0.5426 s/iter data_time: 0.0496 s/iter total_throughput: 1887.26 samples/s lr: 4.26e-04 [09/20 11:29:44] lb.utils.events INFO: eta: 1 day, 1:30:24 iteration: 206999/375342 consumed_samples: 211968000 total_loss: 0.3964 time: 0.5426 s/iter data_time: 0.0558 s/iter total_throughput: 1887.25 samples/s lr: 4.25e-04 [09/20 11:30:39] lb.utils.events INFO: eta: 1 day, 1:29:58 iteration: 207099/375342 consumed_samples: 212070400 total_loss: 0.3977 time: 0.5426 s/iter data_time: 0.0516 s/iter total_throughput: 1887.24 samples/s lr: 4.25e-04 [09/20 11:31:34] lb.utils.events INFO: eta: 1 day, 1:29:17 iteration: 207199/375342 consumed_samples: 212172800 total_loss: 0.4036 time: 0.5426 s/iter data_time: 0.0554 s/iter total_throughput: 1887.23 samples/s lr: 4.24e-04 [09/20 11:32:28] lb.utils.events INFO: eta: 1 day, 1:28:42 iteration: 207299/375342 consumed_samples: 212275200 total_loss: 0.4054 time: 0.5426 s/iter data_time: 0.0544 s/iter total_throughput: 1887.23 samples/s lr: 4.24e-04 [09/20 11:33:23] lb.utils.events INFO: eta: 1 day, 1:28:24 iteration: 207399/375342 consumed_samples: 212377600 total_loss: 0.4015 time: 0.5426 s/iter data_time: 0.0545 s/iter total_throughput: 1887.22 samples/s lr: 4.24e-04 [09/20 11:34:18] lb.utils.events INFO: eta: 1 day, 1:28:31 iteration: 207499/375342 consumed_samples: 212480000 total_loss: 0.4008 time: 0.5426 s/iter data_time: 0.0564 s/iter total_throughput: 1887.21 samples/s lr: 4.23e-04 [09/20 11:35:13] lb.utils.events INFO: eta: 1 day, 1:28:55 iteration: 207599/375342 consumed_samples: 212582400 total_loss: 0.4087 time: 0.5426 s/iter data_time: 0.0539 s/iter total_throughput: 1887.20 samples/s lr: 4.23e-04 [09/20 11:36:08] lb.utils.events INFO: eta: 1 day, 1:29:11 iteration: 207699/375342 consumed_samples: 212684800 total_loss: 0.4103 time: 0.5426 s/iter data_time: 0.0548 s/iter total_throughput: 1887.19 samples/s lr: 4.22e-04 [09/20 11:37:03] lb.utils.events INFO: eta: 1 day, 1:28:58 iteration: 207799/375342 consumed_samples: 212787200 total_loss: 0.3999 time: 0.5426 s/iter data_time: 0.0545 s/iter total_throughput: 1887.18 samples/s lr: 4.22e-04 [09/20 11:37:57] lb.utils.events INFO: eta: 1 day, 1:28:41 iteration: 207899/375342 consumed_samples: 212889600 total_loss: 0.4001 time: 0.5426 s/iter data_time: 0.0558 s/iter total_throughput: 1887.17 samples/s lr: 4.22e-04 [09/20 11:38:52] lb.utils.events INFO: eta: 1 day, 1:28:03 iteration: 207999/375342 consumed_samples: 212992000 total_loss: 0.4069 time: 0.5426 s/iter data_time: 0.0561 s/iter total_throughput: 1887.17 samples/s lr: 4.21e-04 [09/20 11:39:47] lb.utils.events INFO: eta: 1 day, 1:27:10 iteration: 208099/375342 consumed_samples: 213094400 total_loss: 0.3997 time: 0.5426 s/iter data_time: 0.0557 s/iter total_throughput: 1887.16 samples/s lr: 4.21e-04 [09/20 11:40:42] lb.utils.events INFO: eta: 1 day, 1:26:22 iteration: 208199/375342 consumed_samples: 213196800 total_loss: 0.3882 time: 0.5426 s/iter data_time: 0.0536 s/iter total_throughput: 1887.15 samples/s lr: 4.20e-04 [09/20 11:41:36] lb.utils.events INFO: eta: 1 day, 1:25:38 iteration: 208299/375342 consumed_samples: 213299200 total_loss: 0.399 time: 0.5426 s/iter data_time: 0.0501 s/iter total_throughput: 1887.14 samples/s lr: 4.20e-04 [09/20 11:42:31] lb.utils.events INFO: eta: 1 day, 1:24:43 iteration: 208399/375342 consumed_samples: 213401600 total_loss: 0.4009 time: 0.5426 s/iter data_time: 0.0498 s/iter total_throughput: 1887.13 samples/s lr: 4.20e-04 [09/20 11:43:26] lb.utils.events INFO: eta: 1 day, 1:23:45 iteration: 208499/375342 consumed_samples: 213504000 total_loss: 0.4055 time: 0.5426 s/iter data_time: 0.0502 s/iter total_throughput: 1887.12 samples/s lr: 4.19e-04 [09/20 11:44:21] lb.utils.events INFO: eta: 1 day, 1:22:25 iteration: 208599/375342 consumed_samples: 213606400 total_loss: 0.4049 time: 0.5426 s/iter data_time: 0.0505 s/iter total_throughput: 1887.12 samples/s lr: 4.19e-04 [09/20 11:45:15] lb.utils.events INFO: eta: 1 day, 1:20:50 iteration: 208699/375342 consumed_samples: 213708800 total_loss: 0.3993 time: 0.5426 s/iter data_time: 0.0488 s/iter total_throughput: 1887.11 samples/s lr: 4.18e-04 [09/20 11:46:10] lb.utils.events INFO: eta: 1 day, 1:19:24 iteration: 208799/375342 consumed_samples: 213811200 total_loss: 0.4006 time: 0.5426 s/iter data_time: 0.0497 s/iter total_throughput: 1887.10 samples/s lr: 4.18e-04 [09/20 11:47:05] lb.utils.events INFO: eta: 1 day, 1:18:01 iteration: 208899/375342 consumed_samples: 213913600 total_loss: 0.3968 time: 0.5426 s/iter data_time: 0.0519 s/iter total_throughput: 1887.10 samples/s lr: 4.18e-04 [09/20 11:48:00] lb.utils.events INFO: eta: 1 day, 1:17:16 iteration: 208999/375342 consumed_samples: 214016000 total_loss: 0.3949 time: 0.5426 s/iter data_time: 0.0504 s/iter total_throughput: 1887.09 samples/s lr: 4.17e-04 [09/20 11:48:54] lb.utils.events INFO: eta: 1 day, 1:15:38 iteration: 209099/375342 consumed_samples: 214118400 total_loss: 0.389 time: 0.5426 s/iter data_time: 0.0518 s/iter total_throughput: 1887.08 samples/s lr: 4.17e-04 [09/20 11:49:49] lb.utils.events INFO: eta: 1 day, 1:14:36 iteration: 209199/375342 consumed_samples: 214220800 total_loss: 0.3983 time: 0.5426 s/iter data_time: 0.0524 s/iter total_throughput: 1887.08 samples/s lr: 4.16e-04 [09/20 11:50:44] lb.utils.events INFO: eta: 1 day, 1:13:31 iteration: 209299/375342 consumed_samples: 214323200 total_loss: 0.3991 time: 0.5426 s/iter data_time: 0.0513 s/iter total_throughput: 1887.07 samples/s lr: 4.16e-04 [09/20 11:51:38] lb.utils.events INFO: eta: 1 day, 1:12:04 iteration: 209399/375342 consumed_samples: 214425600 total_loss: 0.3954 time: 0.5426 s/iter data_time: 0.0510 s/iter total_throughput: 1887.07 samples/s lr: 4.15e-04 [09/20 11:52:33] lb.utils.events INFO: eta: 1 day, 1:10:42 iteration: 209499/375342 consumed_samples: 214528000 total_loss: 0.4007 time: 0.5426 s/iter data_time: 0.0524 s/iter total_throughput: 1887.06 samples/s lr: 4.15e-04 [09/20 11:53:27] lb.utils.events INFO: eta: 1 day, 1:09:07 iteration: 209599/375342 consumed_samples: 214630400 total_loss: 0.4039 time: 0.5426 s/iter data_time: 0.0518 s/iter total_throughput: 1887.06 samples/s lr: 4.15e-04 [09/20 11:54:22] lb.utils.events INFO: eta: 1 day, 1:07:59 iteration: 209699/375342 consumed_samples: 214732800 total_loss: 0.3989 time: 0.5426 s/iter data_time: 0.0511 s/iter total_throughput: 1887.05 samples/s lr: 4.14e-04 [09/20 11:55:16] lb.utils.events INFO: eta: 1 day, 1:06:48 iteration: 209799/375342 consumed_samples: 214835200 total_loss: 0.4049 time: 0.5426 s/iter data_time: 0.0503 s/iter total_throughput: 1887.05 samples/s lr: 4.14e-04 [09/20 11:56:11] lb.utils.events INFO: eta: 1 day, 1:05:20 iteration: 209899/375342 consumed_samples: 214937600 total_loss: 0.4014 time: 0.5426 s/iter data_time: 0.0523 s/iter total_throughput: 1887.05 samples/s lr: 4.13e-04 [09/20 11:57:05] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0209999 [09/20 11:57:06] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 11:57:06] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 11:57:11] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0523 s/iter. Inference: 0.2595 s/iter. Eval: 0.0027 s/iter. Total: 0.3145 s/iter. ETA=0:00:11 [09/20 11:57:16] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0801 s/iter. Inference: 0.2552 s/iter. Eval: 0.0027 s/iter. Total: 0.3382 s/iter. ETA=0:00:07 [09/20 11:57:21] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0783 s/iter. Inference: 0.2514 s/iter. Eval: 0.0027 s/iter. Total: 0.3326 s/iter. ETA=0:00:01 [09/20 11:57:23] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 11:57:23] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.313219 (0.000286 s / iter per device, on 8 devices) [09/20 11:57:23] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000220 s / iter per device, on 8 devices) [09/20 11:57:23] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 11:57:23] lb.evaluation.utils INFO: copypaste: Acc@1=75.574 [09/20 11:57:23] lb.evaluation.utils INFO: copypaste: Acc@5=92.868 [09/20 11:57:23] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 75.57400, not better than best score 75.69000 @ iteration 204999. [09/20 11:57:23] lb.utils.events INFO: eta: 1 day, 1:03:33 iteration: 209999/375342 consumed_samples: 215040000 total_loss: 0.3922 time: 0.5426 s/iter data_time: 0.0517 s/iter total_throughput: 1887.04 samples/s lr: 4.13e-04 [09/20 11:58:17] lb.utils.events INFO: eta: 1 day, 1:02:05 iteration: 210099/375342 consumed_samples: 215142400 total_loss: 0.3965 time: 0.5426 s/iter data_time: 0.0487 s/iter total_throughput: 1887.04 samples/s lr: 4.13e-04 [09/20 11:59:12] lb.utils.events INFO: eta: 1 day, 1:00:24 iteration: 210199/375342 consumed_samples: 215244800 total_loss: 0.4017 time: 0.5426 s/iter data_time: 0.0485 s/iter total_throughput: 1887.04 samples/s lr: 4.12e-04 [09/20 12:00:06] lb.utils.events INFO: eta: 1 day, 0:59:04 iteration: 210299/375342 consumed_samples: 215347200 total_loss: 0.3922 time: 0.5427 s/iter data_time: 0.0479 s/iter total_throughput: 1887.04 samples/s lr: 4.12e-04 [09/20 12:01:01] lb.utils.events INFO: eta: 1 day, 0:58:07 iteration: 210399/375342 consumed_samples: 215449600 total_loss: 0.391 time: 0.5427 s/iter data_time: 0.0504 s/iter total_throughput: 1887.03 samples/s lr: 4.11e-04 [09/20 12:01:56] lb.utils.events INFO: eta: 1 day, 0:57:25 iteration: 210499/375342 consumed_samples: 215552000 total_loss: 0.4028 time: 0.5427 s/iter data_time: 0.0546 s/iter total_throughput: 1887.02 samples/s lr: 4.11e-04 [09/20 12:02:51] lb.utils.events INFO: eta: 1 day, 0:56:55 iteration: 210599/375342 consumed_samples: 215654400 total_loss: 0.4062 time: 0.5427 s/iter data_time: 0.0539 s/iter total_throughput: 1887.01 samples/s lr: 4.11e-04 [09/20 12:03:45] lb.utils.events INFO: eta: 1 day, 0:56:06 iteration: 210699/375342 consumed_samples: 215756800 total_loss: 0.4011 time: 0.5427 s/iter data_time: 0.0536 s/iter total_throughput: 1887.01 samples/s lr: 4.10e-04 [09/20 12:04:40] lb.utils.events INFO: eta: 1 day, 0:55:54 iteration: 210799/375342 consumed_samples: 215859200 total_loss: 0.3964 time: 0.5427 s/iter data_time: 0.0546 s/iter total_throughput: 1887.00 samples/s lr: 4.10e-04 [09/20 12:05:35] lb.utils.events INFO: eta: 1 day, 0:55:46 iteration: 210899/375342 consumed_samples: 215961600 total_loss: 0.4027 time: 0.5427 s/iter data_time: 0.0540 s/iter total_throughput: 1886.99 samples/s lr: 4.09e-04 [09/20 12:06:30] lb.utils.events INFO: eta: 1 day, 0:55:51 iteration: 210999/375342 consumed_samples: 216064000 total_loss: 0.4077 time: 0.5427 s/iter data_time: 0.0528 s/iter total_throughput: 1886.98 samples/s lr: 4.09e-04 [09/20 12:07:25] lb.utils.events INFO: eta: 1 day, 0:56:11 iteration: 211099/375342 consumed_samples: 216166400 total_loss: 0.4081 time: 0.5427 s/iter data_time: 0.0564 s/iter total_throughput: 1886.97 samples/s lr: 4.09e-04 [09/20 12:08:20] lb.utils.events INFO: eta: 1 day, 0:55:58 iteration: 211199/375342 consumed_samples: 216268800 total_loss: 0.4065 time: 0.5427 s/iter data_time: 0.0535 s/iter total_throughput: 1886.96 samples/s lr: 4.08e-04 [09/20 12:09:14] lb.utils.events INFO: eta: 1 day, 0:56:16 iteration: 211299/375342 consumed_samples: 216371200 total_loss: 0.4001 time: 0.5427 s/iter data_time: 0.0558 s/iter total_throughput: 1886.95 samples/s lr: 4.08e-04 [09/20 12:10:09] lb.utils.events INFO: eta: 1 day, 0:56:10 iteration: 211399/375342 consumed_samples: 216473600 total_loss: 0.3945 time: 0.5427 s/iter data_time: 0.0563 s/iter total_throughput: 1886.95 samples/s lr: 4.07e-04 [09/20 12:11:04] lb.utils.events INFO: eta: 1 day, 0:55:33 iteration: 211499/375342 consumed_samples: 216576000 total_loss: 0.3879 time: 0.5427 s/iter data_time: 0.0562 s/iter total_throughput: 1886.94 samples/s lr: 4.07e-04 [09/20 12:11:59] lb.utils.events INFO: eta: 1 day, 0:55:18 iteration: 211599/375342 consumed_samples: 216678400 total_loss: 0.3952 time: 0.5427 s/iter data_time: 0.0549 s/iter total_throughput: 1886.93 samples/s lr: 4.07e-04 [09/20 12:12:53] lb.utils.events INFO: eta: 1 day, 0:54:20 iteration: 211699/375342 consumed_samples: 216780800 total_loss: 0.3957 time: 0.5427 s/iter data_time: 0.0497 s/iter total_throughput: 1886.92 samples/s lr: 4.06e-04 [09/20 12:13:48] lb.utils.events INFO: eta: 1 day, 0:53:40 iteration: 211799/375342 consumed_samples: 216883200 total_loss: 0.3961 time: 0.5427 s/iter data_time: 0.0493 s/iter total_throughput: 1886.91 samples/s lr: 4.06e-04 [09/20 12:14:43] lb.utils.events INFO: eta: 1 day, 0:52:36 iteration: 211899/375342 consumed_samples: 216985600 total_loss: 0.3994 time: 0.5427 s/iter data_time: 0.0480 s/iter total_throughput: 1886.90 samples/s lr: 4.05e-04 [09/20 12:15:38] lb.utils.events INFO: eta: 1 day, 0:51:18 iteration: 211999/375342 consumed_samples: 217088000 total_loss: 0.3967 time: 0.5427 s/iter data_time: 0.0502 s/iter total_throughput: 1886.90 samples/s lr: 4.05e-04 [09/20 12:16:33] lb.utils.events INFO: eta: 1 day, 0:50:01 iteration: 212099/375342 consumed_samples: 217190400 total_loss: 0.394 time: 0.5427 s/iter data_time: 0.0508 s/iter total_throughput: 1886.89 samples/s lr: 4.04e-04 [09/20 12:17:27] lb.utils.events INFO: eta: 1 day, 0:48:44 iteration: 212199/375342 consumed_samples: 217292800 total_loss: 0.3941 time: 0.5427 s/iter data_time: 0.0510 s/iter total_throughput: 1886.88 samples/s lr: 4.04e-04 [09/20 12:18:22] lb.utils.events INFO: eta: 1 day, 0:47:22 iteration: 212299/375342 consumed_samples: 217395200 total_loss: 0.3943 time: 0.5427 s/iter data_time: 0.0500 s/iter total_throughput: 1886.88 samples/s lr: 4.04e-04 [09/20 12:19:17] lb.utils.events INFO: eta: 1 day, 0:46:01 iteration: 212399/375342 consumed_samples: 217497600 total_loss: 0.3973 time: 0.5427 s/iter data_time: 0.0503 s/iter total_throughput: 1886.87 samples/s lr: 4.03e-04 [09/20 12:20:11] lb.utils.events INFO: eta: 1 day, 0:44:53 iteration: 212499/375342 consumed_samples: 217600000 total_loss: 0.3982 time: 0.5427 s/iter data_time: 0.0501 s/iter total_throughput: 1886.87 samples/s lr: 4.03e-04 [09/20 12:21:06] lb.utils.events INFO: eta: 1 day, 0:43:47 iteration: 212599/375342 consumed_samples: 217702400 total_loss: 0.3984 time: 0.5427 s/iter data_time: 0.0509 s/iter total_throughput: 1886.86 samples/s lr: 4.02e-04 [09/20 12:22:00] lb.utils.events INFO: eta: 1 day, 0:42:52 iteration: 212699/375342 consumed_samples: 217804800 total_loss: 0.3991 time: 0.5427 s/iter data_time: 0.0515 s/iter total_throughput: 1886.86 samples/s lr: 4.02e-04 [09/20 12:22:55] lb.utils.events INFO: eta: 1 day, 0:41:01 iteration: 212799/375342 consumed_samples: 217907200 total_loss: 0.3955 time: 0.5427 s/iter data_time: 0.0518 s/iter total_throughput: 1886.85 samples/s lr: 4.02e-04 [09/20 12:23:50] lb.utils.events INFO: eta: 1 day, 0:39:12 iteration: 212899/375342 consumed_samples: 218009600 total_loss: 0.3953 time: 0.5427 s/iter data_time: 0.0501 s/iter total_throughput: 1886.85 samples/s lr: 4.01e-04 [09/20 12:24:44] lb.utils.events INFO: eta: 1 day, 0:37:33 iteration: 212999/375342 consumed_samples: 218112000 total_loss: 0.3972 time: 0.5427 s/iter data_time: 0.0524 s/iter total_throughput: 1886.84 samples/s lr: 4.01e-04 [09/20 12:25:39] lb.utils.events INFO: eta: 1 day, 0:36:19 iteration: 213099/375342 consumed_samples: 218214400 total_loss: 0.3955 time: 0.5427 s/iter data_time: 0.0507 s/iter total_throughput: 1886.84 samples/s lr: 4.00e-04 [09/20 12:26:33] lb.utils.events INFO: eta: 1 day, 0:34:47 iteration: 213199/375342 consumed_samples: 218316800 total_loss: 0.4009 time: 0.5427 s/iter data_time: 0.0524 s/iter total_throughput: 1886.83 samples/s lr: 4.00e-04 [09/20 12:27:28] lb.utils.events INFO: eta: 1 day, 0:33:45 iteration: 213299/375342 consumed_samples: 218419200 total_loss: 0.3979 time: 0.5427 s/iter data_time: 0.0505 s/iter total_throughput: 1886.83 samples/s lr: 4.00e-04 [09/20 12:28:22] lb.utils.events INFO: eta: 1 day, 0:32:44 iteration: 213399/375342 consumed_samples: 218521600 total_loss: 0.3949 time: 0.5427 s/iter data_time: 0.0518 s/iter total_throughput: 1886.83 samples/s lr: 3.99e-04 [09/20 12:29:17] lb.utils.events INFO: eta: 1 day, 0:31:11 iteration: 213499/375342 consumed_samples: 218624000 total_loss: 0.3927 time: 0.5427 s/iter data_time: 0.0497 s/iter total_throughput: 1886.82 samples/s lr: 3.99e-04 [09/20 12:30:12] lb.utils.events INFO: eta: 1 day, 0:30:15 iteration: 213599/375342 consumed_samples: 218726400 total_loss: 0.399 time: 0.5427 s/iter data_time: 0.0523 s/iter total_throughput: 1886.82 samples/s lr: 3.98e-04 [09/20 12:31:06] lb.utils.events INFO: eta: 1 day, 0:29:21 iteration: 213699/375342 consumed_samples: 218828800 total_loss: 0.4068 time: 0.5427 s/iter data_time: 0.0511 s/iter total_throughput: 1886.81 samples/s lr: 3.98e-04 [09/20 12:32:01] lb.utils.events INFO: eta: 1 day, 0:28:47 iteration: 213799/375342 consumed_samples: 218931200 total_loss: 0.4068 time: 0.5427 s/iter data_time: 0.0505 s/iter total_throughput: 1886.80 samples/s lr: 3.98e-04 [09/20 12:32:56] lb.utils.events INFO: eta: 1 day, 0:28:18 iteration: 213899/375342 consumed_samples: 219033600 total_loss: 0.4 time: 0.5427 s/iter data_time: 0.0578 s/iter total_throughput: 1886.79 samples/s lr: 3.97e-04 [09/20 12:33:51] lb.utils.events INFO: eta: 1 day, 0:28:23 iteration: 213999/375342 consumed_samples: 219136000 total_loss: 0.3963 time: 0.5427 s/iter data_time: 0.0558 s/iter total_throughput: 1886.78 samples/s lr: 3.97e-04 [09/20 12:34:46] lb.utils.events INFO: eta: 1 day, 0:28:04 iteration: 214099/375342 consumed_samples: 219238400 total_loss: 0.3979 time: 0.5427 s/iter data_time: 0.0542 s/iter total_throughput: 1886.77 samples/s lr: 3.96e-04 [09/20 12:35:41] lb.utils.events INFO: eta: 1 day, 0:27:39 iteration: 214199/375342 consumed_samples: 219340800 total_loss: 0.3997 time: 0.5427 s/iter data_time: 0.0550 s/iter total_throughput: 1886.76 samples/s lr: 3.96e-04 [09/20 12:36:35] lb.utils.events INFO: eta: 1 day, 0:27:26 iteration: 214299/375342 consumed_samples: 219443200 total_loss: 0.4048 time: 0.5427 s/iter data_time: 0.0527 s/iter total_throughput: 1886.75 samples/s lr: 3.96e-04 [09/20 12:37:30] lb.utils.events INFO: eta: 1 day, 0:27:06 iteration: 214399/375342 consumed_samples: 219545600 total_loss: 0.4039 time: 0.5427 s/iter data_time: 0.0522 s/iter total_throughput: 1886.75 samples/s lr: 3.95e-04 [09/20 12:38:25] lb.utils.events INFO: eta: 1 day, 0:27:48 iteration: 214499/375342 consumed_samples: 219648000 total_loss: 0.3969 time: 0.5427 s/iter data_time: 0.0539 s/iter total_throughput: 1886.73 samples/s lr: 3.95e-04 [09/20 12:39:20] lb.utils.events INFO: eta: 1 day, 0:27:16 iteration: 214599/375342 consumed_samples: 219750400 total_loss: 0.3935 time: 0.5427 s/iter data_time: 0.0544 s/iter total_throughput: 1886.73 samples/s lr: 3.94e-04 [09/20 12:40:15] lb.utils.events INFO: eta: 1 day, 0:26:45 iteration: 214699/375342 consumed_samples: 219852800 total_loss: 0.3984 time: 0.5427 s/iter data_time: 0.0566 s/iter total_throughput: 1886.72 samples/s lr: 3.94e-04 [09/20 12:41:10] lb.utils.events INFO: eta: 1 day, 0:26:28 iteration: 214799/375342 consumed_samples: 219955200 total_loss: 0.408 time: 0.5427 s/iter data_time: 0.0549 s/iter total_throughput: 1886.71 samples/s lr: 3.94e-04 [09/20 12:42:05] lb.utils.events INFO: eta: 1 day, 0:25:21 iteration: 214899/375342 consumed_samples: 220057600 total_loss: 0.4059 time: 0.5427 s/iter data_time: 0.0524 s/iter total_throughput: 1886.70 samples/s lr: 3.93e-04 [09/20 12:42:59] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0214999 [09/20 12:43:00] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 12:43:00] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 12:43:05] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0593 s/iter. Inference: 0.2544 s/iter. Eval: 0.0035 s/iter. Total: 0.3173 s/iter. ETA=0:00:11 [09/20 12:43:10] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0786 s/iter. Inference: 0.2555 s/iter. Eval: 0.0026 s/iter. Total: 0.3372 s/iter. ETA=0:00:07 [09/20 12:43:15] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0761 s/iter. Inference: 0.2539 s/iter. Eval: 0.0025 s/iter. Total: 0.3330 s/iter. ETA=0:00:01 [09/20 12:43:17] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 12:43:17] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.286328 (0.000286 s / iter per device, on 8 devices) [09/20 12:43:17] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000222 s / iter per device, on 8 devices) [09/20 12:43:17] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 12:43:17] lb.evaluation.utils INFO: copypaste: Acc@1=75.986 [09/20 12:43:17] lb.evaluation.utils INFO: copypaste: Acc@5=93.024 [09/20 12:43:17] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 75.98600, better than last best score 75.69000 @ iteration 204999. [09/20 12:43:17] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 12:43:18] lb.utils.events INFO: eta: 1 day, 0:23:46 iteration: 214999/375342 consumed_samples: 220160000 total_loss: 0.3967 time: 0.5427 s/iter data_time: 0.0539 s/iter total_throughput: 1886.69 samples/s lr: 3.93e-04 [09/20 12:44:13] lb.utils.events INFO: eta: 1 day, 0:23:05 iteration: 215099/375342 consumed_samples: 220262400 total_loss: 0.4015 time: 0.5428 s/iter data_time: 0.0542 s/iter total_throughput: 1886.68 samples/s lr: 3.92e-04 [09/20 12:45:07] lb.utils.events INFO: eta: 1 day, 0:22:47 iteration: 215199/375342 consumed_samples: 220364800 total_loss: 0.4052 time: 0.5428 s/iter data_time: 0.0478 s/iter total_throughput: 1886.68 samples/s lr: 3.92e-04 [09/20 12:46:02] lb.utils.events INFO: eta: 1 day, 0:21:54 iteration: 215299/375342 consumed_samples: 220467200 total_loss: 0.3987 time: 0.5428 s/iter data_time: 0.0485 s/iter total_throughput: 1886.67 samples/s lr: 3.92e-04 [09/20 12:46:57] lb.utils.events INFO: eta: 1 day, 0:20:57 iteration: 215399/375342 consumed_samples: 220569600 total_loss: 0.3964 time: 0.5428 s/iter data_time: 0.0492 s/iter total_throughput: 1886.66 samples/s lr: 3.91e-04 [09/20 12:47:52] lb.utils.events INFO: eta: 1 day, 0:19:25 iteration: 215499/375342 consumed_samples: 220672000 total_loss: 0.3962 time: 0.5428 s/iter data_time: 0.0518 s/iter total_throughput: 1886.65 samples/s lr: 3.91e-04 [09/20 12:48:47] lb.utils.events INFO: eta: 1 day, 0:18:13 iteration: 215599/375342 consumed_samples: 220774400 total_loss: 0.3972 time: 0.5428 s/iter data_time: 0.0504 s/iter total_throughput: 1886.65 samples/s lr: 3.90e-04 [09/20 12:49:41] lb.utils.events INFO: eta: 1 day, 0:17:01 iteration: 215699/375342 consumed_samples: 220876800 total_loss: 0.399 time: 0.5428 s/iter data_time: 0.0490 s/iter total_throughput: 1886.64 samples/s lr: 3.90e-04 [09/20 12:50:36] lb.utils.events INFO: eta: 1 day, 0:15:48 iteration: 215799/375342 consumed_samples: 220979200 total_loss: 0.4009 time: 0.5428 s/iter data_time: 0.0489 s/iter total_throughput: 1886.63 samples/s lr: 3.90e-04 [09/20 12:51:31] lb.utils.events INFO: eta: 1 day, 0:14:46 iteration: 215899/375342 consumed_samples: 221081600 total_loss: 0.4052 time: 0.5428 s/iter data_time: 0.0507 s/iter total_throughput: 1886.63 samples/s lr: 3.89e-04 [09/20 12:52:25] lb.utils.events INFO: eta: 1 day, 0:13:27 iteration: 215999/375342 consumed_samples: 221184000 total_loss: 0.4003 time: 0.5428 s/iter data_time: 0.0518 s/iter total_throughput: 1886.62 samples/s lr: 3.89e-04 [09/20 12:53:20] lb.utils.events INFO: eta: 1 day, 0:12:03 iteration: 216099/375342 consumed_samples: 221286400 total_loss: 0.4032 time: 0.5428 s/iter data_time: 0.0517 s/iter total_throughput: 1886.62 samples/s lr: 3.88e-04 [09/20 12:54:15] lb.utils.events INFO: eta: 1 day, 0:10:34 iteration: 216199/375342 consumed_samples: 221388800 total_loss: 0.4045 time: 0.5428 s/iter data_time: 0.0506 s/iter total_throughput: 1886.61 samples/s lr: 3.88e-04 [09/20 12:55:09] lb.utils.events INFO: eta: 1 day, 0:09:16 iteration: 216299/375342 consumed_samples: 221491200 total_loss: 0.4009 time: 0.5428 s/iter data_time: 0.0522 s/iter total_throughput: 1886.60 samples/s lr: 3.88e-04 [09/20 12:56:04] lb.utils.events INFO: eta: 1 day, 0:07:59 iteration: 216399/375342 consumed_samples: 221593600 total_loss: 0.3961 time: 0.5428 s/iter data_time: 0.0523 s/iter total_throughput: 1886.60 samples/s lr: 3.87e-04 [09/20 12:56:58] lb.utils.events INFO: eta: 1 day, 0:06:32 iteration: 216499/375342 consumed_samples: 221696000 total_loss: 0.3906 time: 0.5428 s/iter data_time: 0.0523 s/iter total_throughput: 1886.59 samples/s lr: 3.87e-04 [09/20 12:57:53] lb.utils.events INFO: eta: 1 day, 0:05:25 iteration: 216599/375342 consumed_samples: 221798400 total_loss: 0.3945 time: 0.5428 s/iter data_time: 0.0525 s/iter total_throughput: 1886.59 samples/s lr: 3.86e-04 [09/20 12:58:48] lb.utils.events INFO: eta: 1 day, 0:03:52 iteration: 216699/375342 consumed_samples: 221900800 total_loss: 0.3962 time: 0.5428 s/iter data_time: 0.0514 s/iter total_throughput: 1886.59 samples/s lr: 3.86e-04 [09/20 12:59:42] lb.utils.events INFO: eta: 1 day, 0:02:15 iteration: 216799/375342 consumed_samples: 222003200 total_loss: 0.3999 time: 0.5428 s/iter data_time: 0.0516 s/iter total_throughput: 1886.58 samples/s lr: 3.86e-04 [09/20 13:00:37] lb.utils.events INFO: eta: 1 day, 0:00:48 iteration: 216899/375342 consumed_samples: 222105600 total_loss: 0.4054 time: 0.5428 s/iter data_time: 0.0514 s/iter total_throughput: 1886.58 samples/s lr: 3.85e-04 [09/20 13:01:31] lb.utils.events INFO: eta: 23:59:50 iteration: 216999/375342 consumed_samples: 222208000 total_loss: 0.3978 time: 0.5428 s/iter data_time: 0.0499 s/iter total_throughput: 1886.58 samples/s lr: 3.85e-04 [09/20 13:02:26] lb.utils.events INFO: eta: 23:58:47 iteration: 217099/375342 consumed_samples: 222310400 total_loss: 0.3948 time: 0.5428 s/iter data_time: 0.0500 s/iter total_throughput: 1886.57 samples/s lr: 3.84e-04 [09/20 13:03:20] lb.utils.events INFO: eta: 23:57:34 iteration: 217199/375342 consumed_samples: 222412800 total_loss: 0.3944 time: 0.5428 s/iter data_time: 0.0488 s/iter total_throughput: 1886.57 samples/s lr: 3.84e-04 [09/20 13:04:15] lb.utils.events INFO: eta: 23:56:30 iteration: 217299/375342 consumed_samples: 222515200 total_loss: 0.3997 time: 0.5428 s/iter data_time: 0.0524 s/iter total_throughput: 1886.56 samples/s lr: 3.84e-04 [09/20 13:05:10] lb.utils.events INFO: eta: 23:55:42 iteration: 217399/375342 consumed_samples: 222617600 total_loss: 0.4013 time: 0.5428 s/iter data_time: 0.0528 s/iter total_throughput: 1886.55 samples/s lr: 3.83e-04 [09/20 13:06:05] lb.utils.events INFO: eta: 23:55:25 iteration: 217499/375342 consumed_samples: 222720000 total_loss: 0.3981 time: 0.5428 s/iter data_time: 0.0527 s/iter total_throughput: 1886.54 samples/s lr: 3.83e-04 [09/20 13:06:59] lb.utils.events INFO: eta: 23:54:39 iteration: 217599/375342 consumed_samples: 222822400 total_loss: 0.3989 time: 0.5428 s/iter data_time: 0.0548 s/iter total_throughput: 1886.54 samples/s lr: 3.82e-04 [09/20 13:07:54] lb.utils.events INFO: eta: 23:53:58 iteration: 217699/375342 consumed_samples: 222924800 total_loss: 0.3962 time: 0.5428 s/iter data_time: 0.0549 s/iter total_throughput: 1886.53 samples/s lr: 3.82e-04 [09/20 13:08:49] lb.utils.events INFO: eta: 23:54:36 iteration: 217799/375342 consumed_samples: 223027200 total_loss: 0.3962 time: 0.5428 s/iter data_time: 0.0574 s/iter total_throughput: 1886.52 samples/s lr: 3.81e-04 [09/20 13:09:44] lb.utils.events INFO: eta: 23:54:23 iteration: 217899/375342 consumed_samples: 223129600 total_loss: 0.399 time: 0.5428 s/iter data_time: 0.0501 s/iter total_throughput: 1886.51 samples/s lr: 3.81e-04 [09/20 13:10:39] lb.utils.events INFO: eta: 23:54:32 iteration: 217999/375342 consumed_samples: 223232000 total_loss: 0.3996 time: 0.5428 s/iter data_time: 0.0570 s/iter total_throughput: 1886.50 samples/s lr: 3.81e-04 [09/20 13:11:33] lb.utils.events INFO: eta: 23:54:11 iteration: 218099/375342 consumed_samples: 223334400 total_loss: 0.4003 time: 0.5428 s/iter data_time: 0.0546 s/iter total_throughput: 1886.50 samples/s lr: 3.80e-04 [09/20 13:12:28] lb.utils.events INFO: eta: 23:53:52 iteration: 218199/375342 consumed_samples: 223436800 total_loss: 0.4031 time: 0.5428 s/iter data_time: 0.0544 s/iter total_throughput: 1886.49 samples/s lr: 3.80e-04 [09/20 13:13:23] lb.utils.events INFO: eta: 23:53:23 iteration: 218299/375342 consumed_samples: 223539200 total_loss: 0.4017 time: 0.5428 s/iter data_time: 0.0540 s/iter total_throughput: 1886.48 samples/s lr: 3.79e-04 [09/20 13:14:18] lb.utils.events INFO: eta: 23:52:31 iteration: 218399/375342 consumed_samples: 223641600 total_loss: 0.3929 time: 0.5428 s/iter data_time: 0.0554 s/iter total_throughput: 1886.47 samples/s lr: 3.79e-04 [09/20 13:15:13] lb.utils.events INFO: eta: 23:51:20 iteration: 218499/375342 consumed_samples: 223744000 total_loss: 0.3961 time: 0.5428 s/iter data_time: 0.0541 s/iter total_throughput: 1886.47 samples/s lr: 3.79e-04 [09/20 13:16:07] lb.utils.events INFO: eta: 23:51:20 iteration: 218599/375342 consumed_samples: 223846400 total_loss: 0.3985 time: 0.5428 s/iter data_time: 0.0493 s/iter total_throughput: 1886.46 samples/s lr: 3.78e-04 [09/20 13:17:02] lb.utils.events INFO: eta: 23:50:56 iteration: 218699/375342 consumed_samples: 223948800 total_loss: 0.4004 time: 0.5428 s/iter data_time: 0.0479 s/iter total_throughput: 1886.45 samples/s lr: 3.78e-04 [09/20 13:17:57] lb.utils.events INFO: eta: 23:49:54 iteration: 218799/375342 consumed_samples: 224051200 total_loss: 0.4049 time: 0.5428 s/iter data_time: 0.0504 s/iter total_throughput: 1886.44 samples/s lr: 3.77e-04 [09/20 13:18:52] lb.utils.events INFO: eta: 23:48:52 iteration: 218899/375342 consumed_samples: 224153600 total_loss: 0.3954 time: 0.5428 s/iter data_time: 0.0496 s/iter total_throughput: 1886.43 samples/s lr: 3.77e-04 [09/20 13:19:47] lb.utils.events INFO: eta: 23:47:06 iteration: 218999/375342 consumed_samples: 224256000 total_loss: 0.3946 time: 0.5428 s/iter data_time: 0.0495 s/iter total_throughput: 1886.43 samples/s lr: 3.77e-04 [09/20 13:20:41] lb.utils.events INFO: eta: 23:46:02 iteration: 219099/375342 consumed_samples: 224358400 total_loss: 0.3951 time: 0.5428 s/iter data_time: 0.0509 s/iter total_throughput: 1886.42 samples/s lr: 3.76e-04 [09/20 13:21:36] lb.utils.events INFO: eta: 23:44:32 iteration: 219199/375342 consumed_samples: 224460800 total_loss: 0.3941 time: 0.5428 s/iter data_time: 0.0506 s/iter total_throughput: 1886.42 samples/s lr: 3.76e-04 [09/20 13:22:30] lb.utils.events INFO: eta: 23:43:08 iteration: 219299/375342 consumed_samples: 224563200 total_loss: 0.3943 time: 0.5428 s/iter data_time: 0.0503 s/iter total_throughput: 1886.41 samples/s lr: 3.75e-04 [09/20 13:23:25] lb.utils.events INFO: eta: 23:42:07 iteration: 219399/375342 consumed_samples: 224665600 total_loss: 0.3925 time: 0.5428 s/iter data_time: 0.0513 s/iter total_throughput: 1886.41 samples/s lr: 3.75e-04 [09/20 13:24:20] lb.utils.events INFO: eta: 23:41:17 iteration: 219499/375342 consumed_samples: 224768000 total_loss: 0.3935 time: 0.5428 s/iter data_time: 0.0496 s/iter total_throughput: 1886.40 samples/s lr: 3.75e-04 [09/20 13:25:14] lb.utils.events INFO: eta: 23:40:00 iteration: 219599/375342 consumed_samples: 224870400 total_loss: 0.3982 time: 0.5428 s/iter data_time: 0.0500 s/iter total_throughput: 1886.39 samples/s lr: 3.74e-04 [09/20 13:26:09] lb.utils.events INFO: eta: 23:38:14 iteration: 219699/375342 consumed_samples: 224972800 total_loss: 0.3957 time: 0.5428 s/iter data_time: 0.0506 s/iter total_throughput: 1886.39 samples/s lr: 3.74e-04 [09/20 13:27:04] lb.utils.events INFO: eta: 23:36:44 iteration: 219799/375342 consumed_samples: 225075200 total_loss: 0.3981 time: 0.5428 s/iter data_time: 0.0518 s/iter total_throughput: 1886.39 samples/s lr: 3.73e-04 [09/20 13:27:58] lb.utils.events INFO: eta: 23:34:57 iteration: 219899/375342 consumed_samples: 225177600 total_loss: 0.3985 time: 0.5428 s/iter data_time: 0.0521 s/iter total_throughput: 1886.38 samples/s lr: 3.73e-04 [09/20 13:28:53] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0219999 [09/20 13:28:53] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 13:28:53] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 13:28:58] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0513 s/iter. Inference: 0.2453 s/iter. Eval: 0.0023 s/iter. Total: 0.2989 s/iter. ETA=0:00:11 [09/20 13:29:03] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0796 s/iter. Inference: 0.2502 s/iter. Eval: 0.0023 s/iter. Total: 0.3322 s/iter. ETA=0:00:07 [09/20 13:29:08] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0772 s/iter. Inference: 0.2511 s/iter. Eval: 0.0023 s/iter. Total: 0.3309 s/iter. ETA=0:00:01 [09/20 13:29:10] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 13:29:10] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.204975 (0.000284 s / iter per device, on 8 devices) [09/20 13:29:10] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000220 s / iter per device, on 8 devices) [09/20 13:29:10] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 13:29:10] lb.evaluation.utils INFO: copypaste: Acc@1=76.03 [09/20 13:29:10] lb.evaluation.utils INFO: copypaste: Acc@5=93.146 [09/20 13:29:10] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 76.03000, better than last best score 75.98600 @ iteration 214999. [09/20 13:29:10] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 13:29:11] lb.utils.events INFO: eta: 23:33:48 iteration: 219999/375342 consumed_samples: 225280000 total_loss: 0.3962 time: 0.5428 s/iter data_time: 0.0516 s/iter total_throughput: 1886.38 samples/s lr: 3.73e-04 [09/20 13:30:05] lb.utils.events INFO: eta: 23:32:39 iteration: 220099/375342 consumed_samples: 225382400 total_loss: 0.3976 time: 0.5428 s/iter data_time: 0.0528 s/iter total_throughput: 1886.38 samples/s lr: 3.72e-04 [09/20 13:31:00] lb.utils.events INFO: eta: 23:31:07 iteration: 220199/375342 consumed_samples: 225484800 total_loss: 0.3942 time: 0.5428 s/iter data_time: 0.0508 s/iter total_throughput: 1886.37 samples/s lr: 3.72e-04 [09/20 13:31:54] lb.utils.events INFO: eta: 23:29:52 iteration: 220299/375342 consumed_samples: 225587200 total_loss: 0.3937 time: 0.5428 s/iter data_time: 0.0527 s/iter total_throughput: 1886.37 samples/s lr: 3.71e-04 [09/20 13:32:49] lb.utils.events INFO: eta: 23:28:07 iteration: 220399/375342 consumed_samples: 225689600 total_loss: 0.3993 time: 0.5428 s/iter data_time: 0.0489 s/iter total_throughput: 1886.37 samples/s lr: 3.71e-04 [09/20 13:33:43] lb.utils.events INFO: eta: 23:26:44 iteration: 220499/375342 consumed_samples: 225792000 total_loss: 0.401 time: 0.5428 s/iter data_time: 0.0494 s/iter total_throughput: 1886.36 samples/s lr: 3.71e-04 [09/20 13:34:38] lb.utils.events INFO: eta: 23:25:20 iteration: 220599/375342 consumed_samples: 225894400 total_loss: 0.3972 time: 0.5428 s/iter data_time: 0.0489 s/iter total_throughput: 1886.36 samples/s lr: 3.70e-04 [09/20 13:35:33] lb.utils.events INFO: eta: 23:24:13 iteration: 220699/375342 consumed_samples: 225996800 total_loss: 0.3964 time: 0.5428 s/iter data_time: 0.0511 s/iter total_throughput: 1886.36 samples/s lr: 3.70e-04 [09/20 13:36:28] lb.utils.events INFO: eta: 23:23:22 iteration: 220799/375342 consumed_samples: 226099200 total_loss: 0.3926 time: 0.5428 s/iter data_time: 0.0543 s/iter total_throughput: 1886.35 samples/s lr: 3.69e-04 [09/20 13:37:22] lb.utils.events INFO: eta: 23:23:12 iteration: 220899/375342 consumed_samples: 226201600 total_loss: 0.3902 time: 0.5429 s/iter data_time: 0.0551 s/iter total_throughput: 1886.34 samples/s lr: 3.69e-04 [09/20 13:38:17] lb.utils.events INFO: eta: 23:23:08 iteration: 220999/375342 consumed_samples: 226304000 total_loss: 0.3944 time: 0.5429 s/iter data_time: 0.0546 s/iter total_throughput: 1886.33 samples/s lr: 3.69e-04 [09/20 13:39:12] lb.utils.events INFO: eta: 23:23:18 iteration: 221099/375342 consumed_samples: 226406400 total_loss: 0.4009 time: 0.5429 s/iter data_time: 0.0548 s/iter total_throughput: 1886.33 samples/s lr: 3.68e-04 [09/20 13:40:07] lb.utils.events INFO: eta: 23:23:16 iteration: 221199/375342 consumed_samples: 226508800 total_loss: 0.4003 time: 0.5429 s/iter data_time: 0.0546 s/iter total_throughput: 1886.32 samples/s lr: 3.68e-04 [09/20 13:41:01] lb.utils.events INFO: eta: 23:23:49 iteration: 221299/375342 consumed_samples: 226611200 total_loss: 0.3999 time: 0.5429 s/iter data_time: 0.0520 s/iter total_throughput: 1886.31 samples/s lr: 3.68e-04 [09/20 13:41:56] lb.utils.events INFO: eta: 23:24:13 iteration: 221399/375342 consumed_samples: 226713600 total_loss: 0.4012 time: 0.5429 s/iter data_time: 0.0553 s/iter total_throughput: 1886.30 samples/s lr: 3.67e-04 [09/20 13:42:51] lb.utils.events INFO: eta: 23:23:58 iteration: 221499/375342 consumed_samples: 226816000 total_loss: 0.4016 time: 0.5429 s/iter data_time: 0.0575 s/iter total_throughput: 1886.29 samples/s lr: 3.67e-04 [09/20 13:43:46] lb.utils.events INFO: eta: 23:24:17 iteration: 221599/375342 consumed_samples: 226918400 total_loss: 0.4007 time: 0.5429 s/iter data_time: 0.0578 s/iter total_throughput: 1886.28 samples/s lr: 3.66e-04 [09/20 13:44:41] lb.utils.events INFO: eta: 23:24:12 iteration: 221699/375342 consumed_samples: 227020800 total_loss: 0.3928 time: 0.5429 s/iter data_time: 0.0537 s/iter total_throughput: 1886.27 samples/s lr: 3.66e-04 [09/20 13:45:36] lb.utils.events INFO: eta: 23:23:19 iteration: 221799/375342 consumed_samples: 227123200 total_loss: 0.3791 time: 0.5429 s/iter data_time: 0.0552 s/iter total_throughput: 1886.26 samples/s lr: 3.66e-04 [09/20 13:46:31] lb.utils.events INFO: eta: 23:22:32 iteration: 221899/375342 consumed_samples: 227225600 total_loss: 0.3899 time: 0.5429 s/iter data_time: 0.0554 s/iter total_throughput: 1886.26 samples/s lr: 3.65e-04 [09/20 13:47:26] lb.utils.events INFO: eta: 23:21:44 iteration: 221999/375342 consumed_samples: 227328000 total_loss: 0.3932 time: 0.5429 s/iter data_time: 0.0542 s/iter total_throughput: 1886.25 samples/s lr: 3.65e-04 [09/20 13:48:20] lb.utils.events INFO: eta: 23:20:51 iteration: 222099/375342 consumed_samples: 227430400 total_loss: 0.3942 time: 0.5429 s/iter data_time: 0.0507 s/iter total_throughput: 1886.24 samples/s lr: 3.64e-04 [09/20 13:49:15] lb.utils.events INFO: eta: 23:19:46 iteration: 222199/375342 consumed_samples: 227532800 total_loss: 0.3983 time: 0.5429 s/iter data_time: 0.0495 s/iter total_throughput: 1886.23 samples/s lr: 3.64e-04 [09/20 13:50:10] lb.utils.events INFO: eta: 23:18:24 iteration: 222299/375342 consumed_samples: 227635200 total_loss: 0.3991 time: 0.5429 s/iter data_time: 0.0510 s/iter total_throughput: 1886.23 samples/s lr: 3.64e-04 [09/20 13:51:05] lb.utils.events INFO: eta: 23:16:49 iteration: 222399/375342 consumed_samples: 227737600 total_loss: 0.3951 time: 0.5429 s/iter data_time: 0.0498 s/iter total_throughput: 1886.22 samples/s lr: 3.63e-04 [09/20 13:51:59] lb.utils.events INFO: eta: 23:15:42 iteration: 222499/375342 consumed_samples: 227840000 total_loss: 0.395 time: 0.5429 s/iter data_time: 0.0504 s/iter total_throughput: 1886.21 samples/s lr: 3.63e-04 [09/20 13:52:54] lb.utils.events INFO: eta: 23:14:15 iteration: 222599/375342 consumed_samples: 227942400 total_loss: 0.3922 time: 0.5429 s/iter data_time: 0.0528 s/iter total_throughput: 1886.21 samples/s lr: 3.62e-04 [09/20 13:53:49] lb.utils.events INFO: eta: 23:12:46 iteration: 222699/375342 consumed_samples: 228044800 total_loss: 0.3912 time: 0.5429 s/iter data_time: 0.0497 s/iter total_throughput: 1886.20 samples/s lr: 3.62e-04 [09/20 13:54:44] lb.utils.events INFO: eta: 23:11:44 iteration: 222799/375342 consumed_samples: 228147200 total_loss: 0.3974 time: 0.5429 s/iter data_time: 0.0505 s/iter total_throughput: 1886.19 samples/s lr: 3.62e-04 [09/20 13:55:38] lb.utils.events INFO: eta: 23:10:36 iteration: 222899/375342 consumed_samples: 228249600 total_loss: 0.3987 time: 0.5429 s/iter data_time: 0.0512 s/iter total_throughput: 1886.19 samples/s lr: 3.61e-04 [09/20 13:56:33] lb.utils.events INFO: eta: 23:09:30 iteration: 222999/375342 consumed_samples: 228352000 total_loss: 0.3927 time: 0.5429 s/iter data_time: 0.0511 s/iter total_throughput: 1886.18 samples/s lr: 3.61e-04 [09/20 13:57:28] lb.utils.events INFO: eta: 23:08:11 iteration: 223099/375342 consumed_samples: 228454400 total_loss: 0.3893 time: 0.5429 s/iter data_time: 0.0520 s/iter total_throughput: 1886.18 samples/s lr: 3.60e-04 [09/20 13:58:22] lb.utils.events INFO: eta: 23:07:14 iteration: 223199/375342 consumed_samples: 228556800 total_loss: 0.3938 time: 0.5429 s/iter data_time: 0.0534 s/iter total_throughput: 1886.17 samples/s lr: 3.60e-04 [09/20 13:59:17] lb.utils.events INFO: eta: 23:05:46 iteration: 223299/375342 consumed_samples: 228659200 total_loss: 0.3959 time: 0.5429 s/iter data_time: 0.0519 s/iter total_throughput: 1886.17 samples/s lr: 3.60e-04 [09/20 14:00:11] lb.utils.events INFO: eta: 23:04:21 iteration: 223399/375342 consumed_samples: 228761600 total_loss: 0.3933 time: 0.5429 s/iter data_time: 0.0513 s/iter total_throughput: 1886.16 samples/s lr: 3.59e-04 [09/20 14:01:06] lb.utils.events INFO: eta: 23:02:48 iteration: 223499/375342 consumed_samples: 228864000 total_loss: 0.3893 time: 0.5429 s/iter data_time: 0.0512 s/iter total_throughput: 1886.16 samples/s lr: 3.59e-04 [09/20 14:02:00] lb.utils.events INFO: eta: 23:00:43 iteration: 223599/375342 consumed_samples: 228966400 total_loss: 0.3912 time: 0.5429 s/iter data_time: 0.0528 s/iter total_throughput: 1886.16 samples/s lr: 3.58e-04 [09/20 14:02:55] lb.utils.events INFO: eta: 22:59:32 iteration: 223699/375342 consumed_samples: 229068800 total_loss: 0.3949 time: 0.5429 s/iter data_time: 0.0527 s/iter total_throughput: 1886.16 samples/s lr: 3.58e-04 [09/20 14:03:49] lb.utils.events INFO: eta: 22:57:25 iteration: 223799/375342 consumed_samples: 229171200 total_loss: 0.3974 time: 0.5429 s/iter data_time: 0.0518 s/iter total_throughput: 1886.15 samples/s lr: 3.58e-04 [09/20 14:04:44] lb.utils.events INFO: eta: 22:55:57 iteration: 223899/375342 consumed_samples: 229273600 total_loss: 0.3966 time: 0.5429 s/iter data_time: 0.0492 s/iter total_throughput: 1886.15 samples/s lr: 3.57e-04 [09/20 14:05:38] lb.utils.events INFO: eta: 22:54:27 iteration: 223999/375342 consumed_samples: 229376000 total_loss: 0.3935 time: 0.5429 s/iter data_time: 0.0499 s/iter total_throughput: 1886.15 samples/s lr: 3.57e-04 [09/20 14:06:33] lb.utils.events INFO: eta: 22:53:15 iteration: 224099/375342 consumed_samples: 229478400 total_loss: 0.4016 time: 0.5429 s/iter data_time: 0.0494 s/iter total_throughput: 1886.14 samples/s lr: 3.56e-04 [09/20 14:07:27] lb.utils.events INFO: eta: 22:52:12 iteration: 224199/375342 consumed_samples: 229580800 total_loss: 0.3961 time: 0.5429 s/iter data_time: 0.0490 s/iter total_throughput: 1886.14 samples/s lr: 3.56e-04 [09/20 14:08:22] lb.utils.events INFO: eta: 22:51:20 iteration: 224299/375342 consumed_samples: 229683200 total_loss: 0.3893 time: 0.5429 s/iter data_time: 0.0546 s/iter total_throughput: 1886.13 samples/s lr: 3.56e-04 [09/20 14:09:17] lb.utils.events INFO: eta: 22:50:55 iteration: 224399/375342 consumed_samples: 229785600 total_loss: 0.3962 time: 0.5429 s/iter data_time: 0.0535 s/iter total_throughput: 1886.12 samples/s lr: 3.55e-04 [09/20 14:10:12] lb.utils.events INFO: eta: 22:50:19 iteration: 224499/375342 consumed_samples: 229888000 total_loss: 0.3949 time: 0.5429 s/iter data_time: 0.0551 s/iter total_throughput: 1886.12 samples/s lr: 3.55e-04 [09/20 14:11:07] lb.utils.events INFO: eta: 22:50:40 iteration: 224599/375342 consumed_samples: 229990400 total_loss: 0.3949 time: 0.5429 s/iter data_time: 0.0557 s/iter total_throughput: 1886.11 samples/s lr: 3.54e-04 [09/20 14:12:01] lb.utils.events INFO: eta: 22:50:43 iteration: 224699/375342 consumed_samples: 230092800 total_loss: 0.3977 time: 0.5429 s/iter data_time: 0.0535 s/iter total_throughput: 1886.10 samples/s lr: 3.54e-04 [09/20 14:12:56] lb.utils.events INFO: eta: 22:51:25 iteration: 224799/375342 consumed_samples: 230195200 total_loss: 0.3999 time: 0.5429 s/iter data_time: 0.0532 s/iter total_throughput: 1886.09 samples/s lr: 3.54e-04 [09/20 14:13:51] lb.utils.events INFO: eta: 22:51:38 iteration: 224899/375342 consumed_samples: 230297600 total_loss: 0.3947 time: 0.5429 s/iter data_time: 0.0558 s/iter total_throughput: 1886.09 samples/s lr: 3.53e-04 [09/20 14:14:46] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0224999 [09/20 14:14:47] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 14:14:47] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 14:14:51] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0493 s/iter. Inference: 0.2410 s/iter. Eval: 0.0024 s/iter. Total: 0.2927 s/iter. ETA=0:00:10 [09/20 14:14:56] lb.evaluation.evaluator INFO: Inference done 25600/50000. Dataloading: 0.0643 s/iter. Inference: 0.2734 s/iter. Eval: 0.0022 s/iter. Total: 0.3401 s/iter. ETA=0:00:07 [09/20 14:15:01] lb.evaluation.evaluator INFO: Inference done 41984/50000. Dataloading: 0.0656 s/iter. Inference: 0.2637 s/iter. Eval: 0.0023 s/iter. Total: 0.3319 s/iter. ETA=0:00:02 [09/20 14:15:04] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 14:15:04] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.274313 (0.000286 s / iter per device, on 8 devices) [09/20 14:15:04] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000230 s / iter per device, on 8 devices) [09/20 14:15:04] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 14:15:04] lb.evaluation.utils INFO: copypaste: Acc@1=75.96000000000001 [09/20 14:15:04] lb.evaluation.utils INFO: copypaste: Acc@5=93.084 [09/20 14:15:04] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 75.96000, not better than best score 76.03000 @ iteration 219999. [09/20 14:15:04] lb.utils.events INFO: eta: 22:51:25 iteration: 224999/375342 consumed_samples: 230400000 total_loss: 0.3907 time: 0.5429 s/iter data_time: 0.0547 s/iter total_throughput: 1886.08 samples/s lr: 3.53e-04 [09/20 14:15:59] lb.utils.events INFO: eta: 22:51:19 iteration: 225099/375342 consumed_samples: 230502400 total_loss: 0.3901 time: 0.5429 s/iter data_time: 0.0536 s/iter total_throughput: 1886.07 samples/s lr: 3.52e-04 [09/20 14:16:53] lb.utils.events INFO: eta: 22:50:41 iteration: 225199/375342 consumed_samples: 230604800 total_loss: 0.3984 time: 0.5429 s/iter data_time: 0.0550 s/iter total_throughput: 1886.06 samples/s lr: 3.52e-04 [09/20 14:17:48] lb.utils.events INFO: eta: 22:49:52 iteration: 225299/375342 consumed_samples: 230707200 total_loss: 0.4013 time: 0.5429 s/iter data_time: 0.0549 s/iter total_throughput: 1886.06 samples/s lr: 3.52e-04 [09/20 14:18:43] lb.utils.events INFO: eta: 22:48:57 iteration: 225399/375342 consumed_samples: 230809600 total_loss: 0.3982 time: 0.5429 s/iter data_time: 0.0526 s/iter total_throughput: 1886.05 samples/s lr: 3.51e-04 [09/20 14:19:38] lb.utils.events INFO: eta: 22:48:06 iteration: 225499/375342 consumed_samples: 230912000 total_loss: 0.397 time: 0.5429 s/iter data_time: 0.0478 s/iter total_throughput: 1886.04 samples/s lr: 3.51e-04 [09/20 14:20:32] lb.utils.events INFO: eta: 22:47:26 iteration: 225599/375342 consumed_samples: 231014400 total_loss: 0.3923 time: 0.5429 s/iter data_time: 0.0486 s/iter total_throughput: 1886.03 samples/s lr: 3.50e-04 [09/20 14:21:27] lb.utils.events INFO: eta: 22:46:32 iteration: 225699/375342 consumed_samples: 231116800 total_loss: 0.3962 time: 0.5429 s/iter data_time: 0.0500 s/iter total_throughput: 1886.03 samples/s lr: 3.50e-04 [09/20 14:22:22] lb.utils.events INFO: eta: 22:45:28 iteration: 225799/375342 consumed_samples: 231219200 total_loss: 0.3875 time: 0.5429 s/iter data_time: 0.0600 s/iter total_throughput: 1886.02 samples/s lr: 3.50e-04 [09/20 14:23:17] lb.utils.events INFO: eta: 22:44:11 iteration: 225899/375342 consumed_samples: 231321600 total_loss: 0.3833 time: 0.5429 s/iter data_time: 0.0543 s/iter total_throughput: 1886.01 samples/s lr: 3.49e-04 [09/20 14:24:12] lb.utils.events INFO: eta: 22:42:42 iteration: 225999/375342 consumed_samples: 231424000 total_loss: 0.3923 time: 0.5429 s/iter data_time: 0.0538 s/iter total_throughput: 1886.00 samples/s lr: 3.49e-04 [09/20 14:25:06] lb.utils.events INFO: eta: 22:41:18 iteration: 226099/375342 consumed_samples: 231526400 total_loss: 0.3943 time: 0.5429 s/iter data_time: 0.0545 s/iter total_throughput: 1886.00 samples/s lr: 3.49e-04 [09/20 14:26:01] lb.utils.events INFO: eta: 22:40:20 iteration: 226199/375342 consumed_samples: 231628800 total_loss: 0.3899 time: 0.5430 s/iter data_time: 0.0549 s/iter total_throughput: 1885.99 samples/s lr: 3.48e-04 [09/20 14:26:56] lb.utils.events INFO: eta: 22:39:53 iteration: 226299/375342 consumed_samples: 231731200 total_loss: 0.3902 time: 0.5430 s/iter data_time: 0.0539 s/iter total_throughput: 1885.98 samples/s lr: 3.48e-04 [09/20 14:27:51] lb.utils.events INFO: eta: 22:39:19 iteration: 226399/375342 consumed_samples: 231833600 total_loss: 0.3933 time: 0.5430 s/iter data_time: 0.0533 s/iter total_throughput: 1885.97 samples/s lr: 3.47e-04 [09/20 14:28:46] lb.utils.events INFO: eta: 22:39:02 iteration: 226499/375342 consumed_samples: 231936000 total_loss: 0.3918 time: 0.5430 s/iter data_time: 0.0528 s/iter total_throughput: 1885.96 samples/s lr: 3.47e-04 [09/20 14:29:41] lb.utils.events INFO: eta: 22:38:33 iteration: 226599/375342 consumed_samples: 232038400 total_loss: 0.3889 time: 0.5430 s/iter data_time: 0.0543 s/iter total_throughput: 1885.95 samples/s lr: 3.47e-04 [09/20 14:30:36] lb.utils.events INFO: eta: 22:38:09 iteration: 226699/375342 consumed_samples: 232140800 total_loss: 0.3947 time: 0.5430 s/iter data_time: 0.0485 s/iter total_throughput: 1885.94 samples/s lr: 3.46e-04 [09/20 14:31:31] lb.utils.events INFO: eta: 22:37:42 iteration: 226799/375342 consumed_samples: 232243200 total_loss: 0.3928 time: 0.5430 s/iter data_time: 0.0508 s/iter total_throughput: 1885.93 samples/s lr: 3.46e-04 [09/20 14:32:26] lb.utils.events INFO: eta: 22:37:28 iteration: 226899/375342 consumed_samples: 232345600 total_loss: 0.3873 time: 0.5430 s/iter data_time: 0.0494 s/iter total_throughput: 1885.92 samples/s lr: 3.45e-04 [09/20 14:33:21] lb.utils.events INFO: eta: 22:37:46 iteration: 226999/375342 consumed_samples: 232448000 total_loss: 0.389 time: 0.5430 s/iter data_time: 0.0536 s/iter total_throughput: 1885.90 samples/s lr: 3.45e-04 [09/20 14:34:17] lb.utils.events INFO: eta: 22:38:06 iteration: 227099/375342 consumed_samples: 232550400 total_loss: 0.3903 time: 0.5430 s/iter data_time: 0.0527 s/iter total_throughput: 1885.89 samples/s lr: 3.45e-04 [09/20 14:35:12] lb.utils.events INFO: eta: 22:37:56 iteration: 227199/375342 consumed_samples: 232652800 total_loss: 0.387 time: 0.5430 s/iter data_time: 0.0530 s/iter total_throughput: 1885.88 samples/s lr: 3.44e-04 [09/20 14:36:07] lb.utils.events INFO: eta: 22:37:15 iteration: 227299/375342 consumed_samples: 232755200 total_loss: 0.387 time: 0.5430 s/iter data_time: 0.0521 s/iter total_throughput: 1885.87 samples/s lr: 3.44e-04 [09/20 14:37:02] lb.utils.events INFO: eta: 22:37:01 iteration: 227399/375342 consumed_samples: 232857600 total_loss: 0.3924 time: 0.5430 s/iter data_time: 0.0537 s/iter total_throughput: 1885.86 samples/s lr: 3.43e-04 [09/20 14:37:57] lb.utils.events INFO: eta: 22:36:13 iteration: 227499/375342 consumed_samples: 232960000 total_loss: 0.3939 time: 0.5430 s/iter data_time: 0.0495 s/iter total_throughput: 1885.85 samples/s lr: 3.43e-04 [09/20 14:38:52] lb.utils.events INFO: eta: 22:35:51 iteration: 227599/375342 consumed_samples: 233062400 total_loss: 0.3981 time: 0.5430 s/iter data_time: 0.0493 s/iter total_throughput: 1885.83 samples/s lr: 3.43e-04 [09/20 14:39:48] lb.utils.events INFO: eta: 22:34:45 iteration: 227699/375342 consumed_samples: 233164800 total_loss: 0.4009 time: 0.5430 s/iter data_time: 0.0520 s/iter total_throughput: 1885.82 samples/s lr: 3.42e-04 [09/20 14:40:42] lb.utils.events INFO: eta: 22:33:25 iteration: 227799/375342 consumed_samples: 233267200 total_loss: 0.4022 time: 0.5430 s/iter data_time: 0.0528 s/iter total_throughput: 1885.81 samples/s lr: 3.42e-04 [09/20 14:41:37] lb.utils.events INFO: eta: 22:32:09 iteration: 227899/375342 consumed_samples: 233369600 total_loss: 0.3927 time: 0.5430 s/iter data_time: 0.0542 s/iter total_throughput: 1885.80 samples/s lr: 3.41e-04 [09/20 14:42:32] lb.utils.events INFO: eta: 22:30:50 iteration: 227999/375342 consumed_samples: 233472000 total_loss: 0.3904 time: 0.5430 s/iter data_time: 0.0550 s/iter total_throughput: 1885.79 samples/s lr: 3.41e-04 [09/20 14:43:27] lb.utils.events INFO: eta: 22:29:09 iteration: 228099/375342 consumed_samples: 233574400 total_loss: 0.392 time: 0.5430 s/iter data_time: 0.0540 s/iter total_throughput: 1885.78 samples/s lr: 3.41e-04 [09/20 14:44:22] lb.utils.events INFO: eta: 22:27:39 iteration: 228199/375342 consumed_samples: 233676800 total_loss: 0.3965 time: 0.5430 s/iter data_time: 0.0543 s/iter total_throughput: 1885.78 samples/s lr: 3.40e-04 [09/20 14:45:17] lb.utils.events INFO: eta: 22:25:54 iteration: 228299/375342 consumed_samples: 233779200 total_loss: 0.3926 time: 0.5430 s/iter data_time: 0.0529 s/iter total_throughput: 1885.77 samples/s lr: 3.40e-04 [09/20 14:46:12] lb.utils.events INFO: eta: 22:24:23 iteration: 228399/375342 consumed_samples: 233881600 total_loss: 0.3892 time: 0.5430 s/iter data_time: 0.0553 s/iter total_throughput: 1885.76 samples/s lr: 3.40e-04 [09/20 14:47:06] lb.utils.events INFO: eta: 22:22:30 iteration: 228499/375342 consumed_samples: 233984000 total_loss: 0.3857 time: 0.5430 s/iter data_time: 0.0573 s/iter total_throughput: 1885.75 samples/s lr: 3.39e-04 [09/20 14:48:01] lb.utils.events INFO: eta: 22:20:45 iteration: 228599/375342 consumed_samples: 234086400 total_loss: 0.3887 time: 0.5430 s/iter data_time: 0.0553 s/iter total_throughput: 1885.75 samples/s lr: 3.39e-04 [09/20 14:48:56] lb.utils.events INFO: eta: 22:19:31 iteration: 228699/375342 consumed_samples: 234188800 total_loss: 0.3917 time: 0.5430 s/iter data_time: 0.0531 s/iter total_throughput: 1885.74 samples/s lr: 3.38e-04 [09/20 14:49:51] lb.utils.events INFO: eta: 22:18:40 iteration: 228799/375342 consumed_samples: 234291200 total_loss: 0.3899 time: 0.5430 s/iter data_time: 0.0550 s/iter total_throughput: 1885.73 samples/s lr: 3.38e-04 [09/20 14:50:46] lb.utils.events INFO: eta: 22:17:42 iteration: 228899/375342 consumed_samples: 234393600 total_loss: 0.387 time: 0.5430 s/iter data_time: 0.0533 s/iter total_throughput: 1885.72 samples/s lr: 3.38e-04 [09/20 14:51:40] lb.utils.events INFO: eta: 22:16:46 iteration: 228999/375342 consumed_samples: 234496000 total_loss: 0.3854 time: 0.5430 s/iter data_time: 0.0513 s/iter total_throughput: 1885.72 samples/s lr: 3.37e-04 [09/20 14:52:35] lb.utils.events INFO: eta: 22:15:51 iteration: 229099/375342 consumed_samples: 234598400 total_loss: 0.389 time: 0.5430 s/iter data_time: 0.0505 s/iter total_throughput: 1885.71 samples/s lr: 3.37e-04 [09/20 14:53:30] lb.utils.events INFO: eta: 22:14:41 iteration: 229199/375342 consumed_samples: 234700800 total_loss: 0.3863 time: 0.5430 s/iter data_time: 0.0544 s/iter total_throughput: 1885.70 samples/s lr: 3.36e-04 [09/20 14:54:25] lb.utils.events INFO: eta: 22:13:35 iteration: 229299/375342 consumed_samples: 234803200 total_loss: 0.3833 time: 0.5430 s/iter data_time: 0.0565 s/iter total_throughput: 1885.69 samples/s lr: 3.36e-04 [09/20 14:55:20] lb.utils.events INFO: eta: 22:12:32 iteration: 229399/375342 consumed_samples: 234905600 total_loss: 0.3879 time: 0.5430 s/iter data_time: 0.0539 s/iter total_throughput: 1885.69 samples/s lr: 3.36e-04 [09/20 14:56:14] lb.utils.events INFO: eta: 22:11:40 iteration: 229499/375342 consumed_samples: 235008000 total_loss: 0.393 time: 0.5430 s/iter data_time: 0.0541 s/iter total_throughput: 1885.68 samples/s lr: 3.35e-04 [09/20 14:57:09] lb.utils.events INFO: eta: 22:10:48 iteration: 229599/375342 consumed_samples: 235110400 total_loss: 0.3904 time: 0.5430 s/iter data_time: 0.0522 s/iter total_throughput: 1885.67 samples/s lr: 3.35e-04 [09/20 14:58:04] lb.utils.events INFO: eta: 22:10:25 iteration: 229699/375342 consumed_samples: 235212800 total_loss: 0.3924 time: 0.5430 s/iter data_time: 0.0534 s/iter total_throughput: 1885.66 samples/s lr: 3.34e-04 [09/20 14:58:59] lb.utils.events INFO: eta: 22:10:17 iteration: 229799/375342 consumed_samples: 235315200 total_loss: 0.3922 time: 0.5430 s/iter data_time: 0.0530 s/iter total_throughput: 1885.65 samples/s lr: 3.34e-04 [09/20 14:59:54] lb.utils.events INFO: eta: 22:09:57 iteration: 229899/375342 consumed_samples: 235417600 total_loss: 0.3877 time: 0.5431 s/iter data_time: 0.0532 s/iter total_throughput: 1885.64 samples/s lr: 3.34e-04 [09/20 15:00:49] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0229999 [09/20 15:00:50] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 15:00:50] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 15:00:55] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0671 s/iter. Inference: 0.2456 s/iter. Eval: 0.0024 s/iter. Total: 0.3151 s/iter. ETA=0:00:11 [09/20 15:01:00] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0725 s/iter. Inference: 0.2622 s/iter. Eval: 0.0024 s/iter. Total: 0.3373 s/iter. ETA=0:00:07 [09/20 15:01:05] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0702 s/iter. Inference: 0.2586 s/iter. Eval: 0.0025 s/iter. Total: 0.3317 s/iter. ETA=0:00:01 [09/20 15:01:07] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 15:01:07] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.272740 (0.000285 s / iter per device, on 8 devices) [09/20 15:01:07] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000225 s / iter per device, on 8 devices) [09/20 15:01:07] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 15:01:07] lb.evaluation.utils INFO: copypaste: Acc@1=76.652 [09/20 15:01:07] lb.evaluation.utils INFO: copypaste: Acc@5=93.382 [09/20 15:01:07] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 76.65200, better than last best score 76.03000 @ iteration 219999. [09/20 15:01:07] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 15:01:08] lb.utils.events INFO: eta: 22:09:26 iteration: 229999/375342 consumed_samples: 235520000 total_loss: 0.3842 time: 0.5431 s/iter data_time: 0.0536 s/iter total_throughput: 1885.63 samples/s lr: 3.33e-04 [09/20 15:02:03] lb.utils.events INFO: eta: 22:08:44 iteration: 230099/375342 consumed_samples: 235622400 total_loss: 0.3947 time: 0.5431 s/iter data_time: 0.0544 s/iter total_throughput: 1885.62 samples/s lr: 3.33e-04 [09/20 15:02:58] lb.utils.events INFO: eta: 22:08:37 iteration: 230199/375342 consumed_samples: 235724800 total_loss: 0.3982 time: 0.5431 s/iter data_time: 0.0522 s/iter total_throughput: 1885.61 samples/s lr: 3.32e-04 [09/20 15:03:53] lb.utils.events INFO: eta: 22:08:35 iteration: 230299/375342 consumed_samples: 235827200 total_loss: 0.4005 time: 0.5431 s/iter data_time: 0.0530 s/iter total_throughput: 1885.59 samples/s lr: 3.32e-04 [09/20 15:04:48] lb.utils.events INFO: eta: 22:08:38 iteration: 230399/375342 consumed_samples: 235929600 total_loss: 0.3994 time: 0.5431 s/iter data_time: 0.0537 s/iter total_throughput: 1885.58 samples/s lr: 3.32e-04 [09/20 15:05:43] lb.utils.events INFO: eta: 22:08:20 iteration: 230499/375342 consumed_samples: 236032000 total_loss: 0.3968 time: 0.5431 s/iter data_time: 0.0542 s/iter total_throughput: 1885.57 samples/s lr: 3.31e-04 [09/20 15:06:39] lb.utils.events INFO: eta: 22:08:10 iteration: 230599/375342 consumed_samples: 236134400 total_loss: 0.3966 time: 0.5431 s/iter data_time: 0.0524 s/iter total_throughput: 1885.56 samples/s lr: 3.31e-04 [09/20 15:07:34] lb.utils.events INFO: eta: 22:07:39 iteration: 230699/375342 consumed_samples: 236236800 total_loss: 0.3962 time: 0.5431 s/iter data_time: 0.0528 s/iter total_throughput: 1885.55 samples/s lr: 3.31e-04 [09/20 15:08:29] lb.utils.events INFO: eta: 22:07:15 iteration: 230799/375342 consumed_samples: 236339200 total_loss: 0.3958 time: 0.5431 s/iter data_time: 0.0533 s/iter total_throughput: 1885.53 samples/s lr: 3.30e-04 [09/20 15:09:24] lb.utils.events INFO: eta: 22:06:53 iteration: 230899/375342 consumed_samples: 236441600 total_loss: 0.3887 time: 0.5431 s/iter data_time: 0.0524 s/iter total_throughput: 1885.52 samples/s lr: 3.30e-04 [09/20 15:10:19] lb.utils.events INFO: eta: 22:06:10 iteration: 230999/375342 consumed_samples: 236544000 total_loss: 0.391 time: 0.5431 s/iter data_time: 0.0520 s/iter total_throughput: 1885.51 samples/s lr: 3.29e-04 [09/20 15:11:14] lb.utils.events INFO: eta: 22:04:56 iteration: 231099/375342 consumed_samples: 236646400 total_loss: 0.3945 time: 0.5431 s/iter data_time: 0.0542 s/iter total_throughput: 1885.50 samples/s lr: 3.29e-04 [09/20 15:12:09] lb.utils.events INFO: eta: 22:02:49 iteration: 231199/375342 consumed_samples: 236748800 total_loss: 0.3899 time: 0.5431 s/iter data_time: 0.0534 s/iter total_throughput: 1885.48 samples/s lr: 3.29e-04 [09/20 15:13:04] lb.utils.events INFO: eta: 22:01:19 iteration: 231299/375342 consumed_samples: 236851200 total_loss: 0.3906 time: 0.5431 s/iter data_time: 0.0509 s/iter total_throughput: 1885.48 samples/s lr: 3.28e-04 [09/20 15:13:59] lb.utils.events INFO: eta: 21:59:59 iteration: 231399/375342 consumed_samples: 236953600 total_loss: 0.3967 time: 0.5431 s/iter data_time: 0.0556 s/iter total_throughput: 1885.47 samples/s lr: 3.28e-04 [09/20 15:14:54] lb.utils.events INFO: eta: 21:58:36 iteration: 231499/375342 consumed_samples: 237056000 total_loss: 0.3904 time: 0.5431 s/iter data_time: 0.0535 s/iter total_throughput: 1885.46 samples/s lr: 3.27e-04 [09/20 15:15:49] lb.utils.events INFO: eta: 21:57:06 iteration: 231599/375342 consumed_samples: 237158400 total_loss: 0.3876 time: 0.5431 s/iter data_time: 0.0505 s/iter total_throughput: 1885.45 samples/s lr: 3.27e-04 [09/20 15:16:44] lb.utils.events INFO: eta: 21:55:31 iteration: 231699/375342 consumed_samples: 237260800 total_loss: 0.388 time: 0.5431 s/iter data_time: 0.0537 s/iter total_throughput: 1885.44 samples/s lr: 3.27e-04 [09/20 15:17:39] lb.utils.events INFO: eta: 21:54:05 iteration: 231799/375342 consumed_samples: 237363200 total_loss: 0.3905 time: 0.5431 s/iter data_time: 0.0519 s/iter total_throughput: 1885.43 samples/s lr: 3.26e-04 [09/20 15:18:34] lb.utils.events INFO: eta: 21:52:06 iteration: 231899/375342 consumed_samples: 237465600 total_loss: 0.3902 time: 0.5431 s/iter data_time: 0.0522 s/iter total_throughput: 1885.43 samples/s lr: 3.26e-04 [09/20 15:19:29] lb.utils.events INFO: eta: 21:50:37 iteration: 231999/375342 consumed_samples: 237568000 total_loss: 0.3816 time: 0.5431 s/iter data_time: 0.0546 s/iter total_throughput: 1885.42 samples/s lr: 3.26e-04 [09/20 15:20:24] lb.utils.events INFO: eta: 21:49:34 iteration: 232099/375342 consumed_samples: 237670400 total_loss: 0.3812 time: 0.5431 s/iter data_time: 0.0518 s/iter total_throughput: 1885.41 samples/s lr: 3.25e-04 [09/20 15:21:18] lb.utils.events INFO: eta: 21:48:39 iteration: 232199/375342 consumed_samples: 237772800 total_loss: 0.3886 time: 0.5431 s/iter data_time: 0.0534 s/iter total_throughput: 1885.40 samples/s lr: 3.25e-04 [09/20 15:22:13] lb.utils.events INFO: eta: 21:47:58 iteration: 232299/375342 consumed_samples: 237875200 total_loss: 0.3913 time: 0.5431 s/iter data_time: 0.0531 s/iter total_throughput: 1885.39 samples/s lr: 3.24e-04 [09/20 15:23:08] lb.utils.events INFO: eta: 21:46:58 iteration: 232399/375342 consumed_samples: 237977600 total_loss: 0.3969 time: 0.5431 s/iter data_time: 0.0520 s/iter total_throughput: 1885.39 samples/s lr: 3.24e-04 [09/20 15:24:03] lb.utils.events INFO: eta: 21:45:56 iteration: 232499/375342 consumed_samples: 238080000 total_loss: 0.3916 time: 0.5431 s/iter data_time: 0.0535 s/iter total_throughput: 1885.38 samples/s lr: 3.24e-04 [09/20 15:24:58] lb.utils.events INFO: eta: 21:45:00 iteration: 232599/375342 consumed_samples: 238182400 total_loss: 0.3896 time: 0.5431 s/iter data_time: 0.0527 s/iter total_throughput: 1885.37 samples/s lr: 3.23e-04 [09/20 15:25:53] lb.utils.events INFO: eta: 21:44:33 iteration: 232699/375342 consumed_samples: 238284800 total_loss: 0.3911 time: 0.5431 s/iter data_time: 0.0549 s/iter total_throughput: 1885.36 samples/s lr: 3.23e-04 [09/20 15:26:48] lb.utils.events INFO: eta: 21:44:07 iteration: 232799/375342 consumed_samples: 238387200 total_loss: 0.3896 time: 0.5431 s/iter data_time: 0.0497 s/iter total_throughput: 1885.35 samples/s lr: 3.22e-04 [09/20 15:27:43] lb.utils.events INFO: eta: 21:44:03 iteration: 232899/375342 consumed_samples: 238489600 total_loss: 0.3904 time: 0.5431 s/iter data_time: 0.0506 s/iter total_throughput: 1885.34 samples/s lr: 3.22e-04 [09/20 15:28:39] lb.utils.events INFO: eta: 21:44:15 iteration: 232999/375342 consumed_samples: 238592000 total_loss: 0.3899 time: 0.5431 s/iter data_time: 0.0530 s/iter total_throughput: 1885.32 samples/s lr: 3.22e-04 [09/20 15:29:34] lb.utils.events INFO: eta: 21:45:13 iteration: 233099/375342 consumed_samples: 238694400 total_loss: 0.3882 time: 0.5431 s/iter data_time: 0.0530 s/iter total_throughput: 1885.30 samples/s lr: 3.21e-04 [09/20 15:30:30] lb.utils.events INFO: eta: 21:45:27 iteration: 233199/375342 consumed_samples: 238796800 total_loss: 0.3877 time: 0.5432 s/iter data_time: 0.0537 s/iter total_throughput: 1885.29 samples/s lr: 3.21e-04 [09/20 15:31:25] lb.utils.events INFO: eta: 21:45:16 iteration: 233299/375342 consumed_samples: 238899200 total_loss: 0.388 time: 0.5432 s/iter data_time: 0.0531 s/iter total_throughput: 1885.27 samples/s lr: 3.21e-04 [09/20 15:32:20] lb.utils.events INFO: eta: 21:45:22 iteration: 233399/375342 consumed_samples: 239001600 total_loss: 0.3927 time: 0.5432 s/iter data_time: 0.0539 s/iter total_throughput: 1885.26 samples/s lr: 3.20e-04 [09/20 15:33:16] lb.utils.events INFO: eta: 21:45:39 iteration: 233499/375342 consumed_samples: 239104000 total_loss: 0.3973 time: 0.5432 s/iter data_time: 0.0534 s/iter total_throughput: 1885.24 samples/s lr: 3.20e-04 [09/20 15:34:11] lb.utils.events INFO: eta: 21:45:52 iteration: 233599/375342 consumed_samples: 239206400 total_loss: 0.3927 time: 0.5432 s/iter data_time: 0.0518 s/iter total_throughput: 1885.23 samples/s lr: 3.19e-04 [09/20 15:35:06] lb.utils.events INFO: eta: 21:44:58 iteration: 233699/375342 consumed_samples: 239308800 total_loss: 0.3888 time: 0.5432 s/iter data_time: 0.0527 s/iter total_throughput: 1885.22 samples/s lr: 3.19e-04 [09/20 15:36:01] lb.utils.events INFO: eta: 21:44:01 iteration: 233799/375342 consumed_samples: 239411200 total_loss: 0.391 time: 0.5432 s/iter data_time: 0.0529 s/iter total_throughput: 1885.21 samples/s lr: 3.19e-04 [09/20 15:36:56] lb.utils.events INFO: eta: 21:42:27 iteration: 233899/375342 consumed_samples: 239513600 total_loss: 0.3907 time: 0.5432 s/iter data_time: 0.0531 s/iter total_throughput: 1885.19 samples/s lr: 3.18e-04 [09/20 15:37:51] lb.utils.events INFO: eta: 21:40:41 iteration: 233999/375342 consumed_samples: 239616000 total_loss: 0.3914 time: 0.5432 s/iter data_time: 0.0555 s/iter total_throughput: 1885.18 samples/s lr: 3.18e-04 [09/20 15:38:46] lb.utils.events INFO: eta: 21:38:15 iteration: 234099/375342 consumed_samples: 239718400 total_loss: 0.3927 time: 0.5432 s/iter data_time: 0.0517 s/iter total_throughput: 1885.17 samples/s lr: 3.17e-04 [09/20 15:39:41] lb.utils.events INFO: eta: 21:36:19 iteration: 234199/375342 consumed_samples: 239820800 total_loss: 0.3811 time: 0.5432 s/iter data_time: 0.0530 s/iter total_throughput: 1885.17 samples/s lr: 3.17e-04 [09/20 15:40:36] lb.utils.events INFO: eta: 21:34:19 iteration: 234299/375342 consumed_samples: 239923200 total_loss: 0.3818 time: 0.5432 s/iter data_time: 0.0539 s/iter total_throughput: 1885.16 samples/s lr: 3.17e-04 [09/20 15:41:31] lb.utils.events INFO: eta: 21:32:31 iteration: 234399/375342 consumed_samples: 240025600 total_loss: 0.3877 time: 0.5432 s/iter data_time: 0.0536 s/iter total_throughput: 1885.15 samples/s lr: 3.16e-04 [09/20 15:42:25] lb.utils.events INFO: eta: 21:29:51 iteration: 234499/375342 consumed_samples: 240128000 total_loss: 0.3857 time: 0.5432 s/iter data_time: 0.0525 s/iter total_throughput: 1885.15 samples/s lr: 3.16e-04 [09/20 15:43:21] lb.utils.events INFO: eta: 21:27:41 iteration: 234599/375342 consumed_samples: 240230400 total_loss: 0.3902 time: 0.5432 s/iter data_time: 0.0543 s/iter total_throughput: 1885.14 samples/s lr: 3.16e-04 [09/20 15:44:15] lb.utils.events INFO: eta: 21:26:18 iteration: 234699/375342 consumed_samples: 240332800 total_loss: 0.3908 time: 0.5432 s/iter data_time: 0.0549 s/iter total_throughput: 1885.13 samples/s lr: 3.15e-04 [09/20 15:45:10] lb.utils.events INFO: eta: 21:24:30 iteration: 234799/375342 consumed_samples: 240435200 total_loss: 0.3894 time: 0.5432 s/iter data_time: 0.0537 s/iter total_throughput: 1885.12 samples/s lr: 3.15e-04 [09/20 15:46:05] lb.utils.events INFO: eta: 21:23:03 iteration: 234899/375342 consumed_samples: 240537600 total_loss: 0.3844 time: 0.5432 s/iter data_time: 0.0544 s/iter total_throughput: 1885.11 samples/s lr: 3.14e-04 [09/20 15:47:00] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0234999 [09/20 15:47:01] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 15:47:01] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 15:47:05] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0637 s/iter. Inference: 0.2477 s/iter. Eval: 0.0023 s/iter. Total: 0.3137 s/iter. ETA=0:00:11 [09/20 15:47:11] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0664 s/iter. Inference: 0.2686 s/iter. Eval: 0.0022 s/iter. Total: 0.3376 s/iter. ETA=0:00:07 [09/20 15:47:16] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0676 s/iter. Inference: 0.2622 s/iter. Eval: 0.0026 s/iter. Total: 0.3328 s/iter. ETA=0:00:01 [09/20 15:47:18] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 15:47:18] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.290655 (0.000286 s / iter per device, on 8 devices) [09/20 15:47:18] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000228 s / iter per device, on 8 devices) [09/20 15:47:18] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 15:47:18] lb.evaluation.utils INFO: copypaste: Acc@1=76.73400000000001 [09/20 15:47:18] lb.evaluation.utils INFO: copypaste: Acc@5=93.516 [09/20 15:47:18] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 76.73400, better than last best score 76.65200 @ iteration 229999. [09/20 15:47:18] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 15:47:18] lb.utils.events INFO: eta: 21:21:45 iteration: 234999/375342 consumed_samples: 240640000 total_loss: 0.3816 time: 0.5432 s/iter data_time: 0.0538 s/iter total_throughput: 1885.11 samples/s lr: 3.14e-04 [09/20 15:48:13] lb.utils.events INFO: eta: 21:20:47 iteration: 235099/375342 consumed_samples: 240742400 total_loss: 0.3857 time: 0.5432 s/iter data_time: 0.0543 s/iter total_throughput: 1885.10 samples/s lr: 3.14e-04 [09/20 15:49:08] lb.utils.events INFO: eta: 21:19:55 iteration: 235199/375342 consumed_samples: 240844800 total_loss: 0.3872 time: 0.5432 s/iter data_time: 0.0551 s/iter total_throughput: 1885.09 samples/s lr: 3.13e-04 [09/20 15:50:03] lb.utils.events INFO: eta: 21:19:18 iteration: 235299/375342 consumed_samples: 240947200 total_loss: 0.3923 time: 0.5432 s/iter data_time: 0.0558 s/iter total_throughput: 1885.08 samples/s lr: 3.13e-04 [09/20 15:50:58] lb.utils.events INFO: eta: 21:19:13 iteration: 235399/375342 consumed_samples: 241049600 total_loss: 0.3943 time: 0.5432 s/iter data_time: 0.0505 s/iter total_throughput: 1885.07 samples/s lr: 3.12e-04 [09/20 15:51:53] lb.utils.events INFO: eta: 21:19:22 iteration: 235499/375342 consumed_samples: 241152000 total_loss: 0.3937 time: 0.5432 s/iter data_time: 0.0527 s/iter total_throughput: 1885.06 samples/s lr: 3.12e-04 [09/20 15:52:48] lb.utils.events INFO: eta: 21:19:42 iteration: 235599/375342 consumed_samples: 241254400 total_loss: 0.3913 time: 0.5432 s/iter data_time: 0.0527 s/iter total_throughput: 1885.05 samples/s lr: 3.12e-04 [09/20 15:53:44] lb.utils.events INFO: eta: 21:19:05 iteration: 235699/375342 consumed_samples: 241356800 total_loss: 0.3973 time: 0.5432 s/iter data_time: 0.0575 s/iter total_throughput: 1885.04 samples/s lr: 3.11e-04 [09/20 15:54:39] lb.utils.events INFO: eta: 21:18:47 iteration: 235799/375342 consumed_samples: 241459200 total_loss: 0.3954 time: 0.5432 s/iter data_time: 0.0499 s/iter total_throughput: 1885.03 samples/s lr: 3.11e-04 [09/20 15:55:34] lb.utils.events INFO: eta: 21:18:34 iteration: 235899/375342 consumed_samples: 241561600 total_loss: 0.3889 time: 0.5432 s/iter data_time: 0.0548 s/iter total_throughput: 1885.02 samples/s lr: 3.11e-04 [09/20 15:56:29] lb.utils.events INFO: eta: 21:18:26 iteration: 235999/375342 consumed_samples: 241664000 total_loss: 0.3904 time: 0.5432 s/iter data_time: 0.0562 s/iter total_throughput: 1885.01 samples/s lr: 3.10e-04 [09/20 15:57:24] lb.utils.events INFO: eta: 21:18:15 iteration: 236099/375342 consumed_samples: 241766400 total_loss: 0.3902 time: 0.5432 s/iter data_time: 0.0468 s/iter total_throughput: 1885.00 samples/s lr: 3.10e-04 [09/20 15:58:19] lb.utils.events INFO: eta: 21:17:45 iteration: 236199/375342 consumed_samples: 241868800 total_loss: 0.3904 time: 0.5432 s/iter data_time: 0.0529 s/iter total_throughput: 1884.99 samples/s lr: 3.09e-04 [09/20 15:59:14] lb.utils.events INFO: eta: 21:17:00 iteration: 236299/375342 consumed_samples: 241971200 total_loss: 0.393 time: 0.5432 s/iter data_time: 0.0545 s/iter total_throughput: 1884.97 samples/s lr: 3.09e-04 [09/20 16:00:09] lb.utils.events INFO: eta: 21:16:19 iteration: 236399/375342 consumed_samples: 242073600 total_loss: 0.395 time: 0.5432 s/iter data_time: 0.0533 s/iter total_throughput: 1884.96 samples/s lr: 3.09e-04 [09/20 16:01:04] lb.utils.events INFO: eta: 21:15:34 iteration: 236499/375342 consumed_samples: 242176000 total_loss: 0.3921 time: 0.5433 s/iter data_time: 0.0522 s/iter total_throughput: 1884.95 samples/s lr: 3.08e-04 [09/20 16:01:59] lb.utils.events INFO: eta: 21:14:26 iteration: 236599/375342 consumed_samples: 242278400 total_loss: 0.391 time: 0.5433 s/iter data_time: 0.0506 s/iter total_throughput: 1884.94 samples/s lr: 3.08e-04 [09/20 16:02:54] lb.utils.events INFO: eta: 21:13:28 iteration: 236699/375342 consumed_samples: 242380800 total_loss: 0.3882 time: 0.5433 s/iter data_time: 0.0518 s/iter total_throughput: 1884.93 samples/s lr: 3.08e-04 [09/20 16:03:50] lb.utils.events INFO: eta: 21:12:36 iteration: 236799/375342 consumed_samples: 242483200 total_loss: 0.3835 time: 0.5433 s/iter data_time: 0.0498 s/iter total_throughput: 1884.92 samples/s lr: 3.07e-04 [09/20 16:04:45] lb.utils.events INFO: eta: 21:11:40 iteration: 236899/375342 consumed_samples: 242585600 total_loss: 0.3872 time: 0.5433 s/iter data_time: 0.0514 s/iter total_throughput: 1884.91 samples/s lr: 3.07e-04 [09/20 16:05:40] lb.utils.events INFO: eta: 21:10:06 iteration: 236999/375342 consumed_samples: 242688000 total_loss: 0.3883 time: 0.5433 s/iter data_time: 0.0499 s/iter total_throughput: 1884.90 samples/s lr: 3.06e-04 [09/20 16:06:34] lb.utils.events INFO: eta: 21:07:58 iteration: 237099/375342 consumed_samples: 242790400 total_loss: 0.3887 time: 0.5433 s/iter data_time: 0.0510 s/iter total_throughput: 1884.89 samples/s lr: 3.06e-04 [09/20 16:07:29] lb.utils.events INFO: eta: 21:06:20 iteration: 237199/375342 consumed_samples: 242892800 total_loss: 0.3949 time: 0.5433 s/iter data_time: 0.0532 s/iter total_throughput: 1884.89 samples/s lr: 3.06e-04 [09/20 16:08:24] lb.utils.events INFO: eta: 21:04:36 iteration: 237299/375342 consumed_samples: 242995200 total_loss: 0.392 time: 0.5433 s/iter data_time: 0.0521 s/iter total_throughput: 1884.88 samples/s lr: 3.05e-04 [09/20 16:09:19] lb.utils.events INFO: eta: 21:02:24 iteration: 237399/375342 consumed_samples: 243097600 total_loss: 0.3908 time: 0.5433 s/iter data_time: 0.0515 s/iter total_throughput: 1884.88 samples/s lr: 3.05e-04 [09/20 16:10:13] lb.utils.events INFO: eta: 21:00:19 iteration: 237499/375342 consumed_samples: 243200000 total_loss: 0.3868 time: 0.5433 s/iter data_time: 0.0518 s/iter total_throughput: 1884.87 samples/s lr: 3.04e-04 [09/20 16:11:08] lb.utils.events INFO: eta: 20:58:39 iteration: 237599/375342 consumed_samples: 243302400 total_loss: 0.387 time: 0.5433 s/iter data_time: 0.0506 s/iter total_throughput: 1884.87 samples/s lr: 3.04e-04 [09/20 16:12:02] lb.utils.events INFO: eta: 20:56:23 iteration: 237699/375342 consumed_samples: 243404800 total_loss: 0.3919 time: 0.5433 s/iter data_time: 0.0532 s/iter total_throughput: 1884.87 samples/s lr: 3.04e-04 [09/20 16:12:57] lb.utils.events INFO: eta: 20:54:03 iteration: 237799/375342 consumed_samples: 243507200 total_loss: 0.3911 time: 0.5433 s/iter data_time: 0.0511 s/iter total_throughput: 1884.86 samples/s lr: 3.03e-04 [09/20 16:13:51] lb.utils.events INFO: eta: 20:52:01 iteration: 237899/375342 consumed_samples: 243609600 total_loss: 0.3861 time: 0.5433 s/iter data_time: 0.0510 s/iter total_throughput: 1884.86 samples/s lr: 3.03e-04 [09/20 16:14:46] lb.utils.events INFO: eta: 20:50:06 iteration: 237999/375342 consumed_samples: 243712000 total_loss: 0.3858 time: 0.5433 s/iter data_time: 0.0515 s/iter total_throughput: 1884.86 samples/s lr: 3.03e-04 [09/20 16:15:41] lb.utils.events INFO: eta: 20:49:17 iteration: 238099/375342 consumed_samples: 243814400 total_loss: 0.3845 time: 0.5433 s/iter data_time: 0.0537 s/iter total_throughput: 1884.85 samples/s lr: 3.02e-04 [09/20 16:16:36] lb.utils.events INFO: eta: 20:48:22 iteration: 238199/375342 consumed_samples: 243916800 total_loss: 0.3834 time: 0.5433 s/iter data_time: 0.0541 s/iter total_throughput: 1884.84 samples/s lr: 3.02e-04 [09/20 16:17:31] lb.utils.events INFO: eta: 20:47:27 iteration: 238299/375342 consumed_samples: 244019200 total_loss: 0.3864 time: 0.5433 s/iter data_time: 0.0549 s/iter total_throughput: 1884.83 samples/s lr: 3.01e-04 [09/20 16:18:26] lb.utils.events INFO: eta: 20:46:38 iteration: 238399/375342 consumed_samples: 244121600 total_loss: 0.3925 time: 0.5433 s/iter data_time: 0.0530 s/iter total_throughput: 1884.83 samples/s lr: 3.01e-04 [09/20 16:19:20] lb.utils.events INFO: eta: 20:46:03 iteration: 238499/375342 consumed_samples: 244224000 total_loss: 0.3912 time: 0.5433 s/iter data_time: 0.0543 s/iter total_throughput: 1884.82 samples/s lr: 3.01e-04 [09/20 16:20:15] lb.utils.events INFO: eta: 20:45:42 iteration: 238599/375342 consumed_samples: 244326400 total_loss: 0.3872 time: 0.5433 s/iter data_time: 0.0532 s/iter total_throughput: 1884.81 samples/s lr: 3.00e-04 [09/20 16:21:10] lb.utils.events INFO: eta: 20:45:48 iteration: 238699/375342 consumed_samples: 244428800 total_loss: 0.3871 time: 0.5433 s/iter data_time: 0.0562 s/iter total_throughput: 1884.80 samples/s lr: 3.00e-04 [09/20 16:22:05] lb.utils.events INFO: eta: 20:46:02 iteration: 238799/375342 consumed_samples: 244531200 total_loss: 0.3896 time: 0.5433 s/iter data_time: 0.0520 s/iter total_throughput: 1884.80 samples/s lr: 3.00e-04 [09/20 16:23:00] lb.utils.events INFO: eta: 20:46:13 iteration: 238899/375342 consumed_samples: 244633600 total_loss: 0.3916 time: 0.5433 s/iter data_time: 0.0511 s/iter total_throughput: 1884.79 samples/s lr: 2.99e-04 [09/20 16:23:55] lb.utils.events INFO: eta: 20:46:56 iteration: 238999/375342 consumed_samples: 244736000 total_loss: 0.3881 time: 0.5433 s/iter data_time: 0.0527 s/iter total_throughput: 1884.78 samples/s lr: 2.99e-04 [09/20 16:24:50] lb.utils.events INFO: eta: 20:46:16 iteration: 239099/375342 consumed_samples: 244838400 total_loss: 0.3843 time: 0.5433 s/iter data_time: 0.0548 s/iter total_throughput: 1884.77 samples/s lr: 2.98e-04 [09/20 16:25:45] lb.utils.events INFO: eta: 20:45:31 iteration: 239199/375342 consumed_samples: 244940800 total_loss: 0.3894 time: 0.5433 s/iter data_time: 0.0504 s/iter total_throughput: 1884.76 samples/s lr: 2.98e-04 [09/20 16:26:40] lb.utils.events INFO: eta: 20:44:44 iteration: 239299/375342 consumed_samples: 245043200 total_loss: 0.3873 time: 0.5433 s/iter data_time: 0.0548 s/iter total_throughput: 1884.75 samples/s lr: 2.98e-04 [09/20 16:27:35] lb.utils.events INFO: eta: 20:44:02 iteration: 239399/375342 consumed_samples: 245145600 total_loss: 0.386 time: 0.5433 s/iter data_time: 0.0507 s/iter total_throughput: 1884.75 samples/s lr: 2.97e-04 [09/20 16:28:30] lb.utils.events INFO: eta: 20:43:23 iteration: 239499/375342 consumed_samples: 245248000 total_loss: 0.3879 time: 0.5433 s/iter data_time: 0.0499 s/iter total_throughput: 1884.74 samples/s lr: 2.97e-04 [09/20 16:29:25] lb.utils.events INFO: eta: 20:43:00 iteration: 239599/375342 consumed_samples: 245350400 total_loss: 0.3879 time: 0.5433 s/iter data_time: 0.0499 s/iter total_throughput: 1884.73 samples/s lr: 2.97e-04 [09/20 16:30:20] lb.utils.events INFO: eta: 20:42:27 iteration: 239699/375342 consumed_samples: 245452800 total_loss: 0.3894 time: 0.5433 s/iter data_time: 0.0523 s/iter total_throughput: 1884.72 samples/s lr: 2.96e-04 [09/20 16:31:15] lb.utils.events INFO: eta: 20:41:50 iteration: 239799/375342 consumed_samples: 245555200 total_loss: 0.3906 time: 0.5433 s/iter data_time: 0.0523 s/iter total_throughput: 1884.71 samples/s lr: 2.96e-04 [09/20 16:32:10] lb.utils.events INFO: eta: 20:40:53 iteration: 239899/375342 consumed_samples: 245657600 total_loss: 0.3897 time: 0.5433 s/iter data_time: 0.0518 s/iter total_throughput: 1884.70 samples/s lr: 2.95e-04 [09/20 16:33:05] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0239999 [09/20 16:33:05] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 16:33:05] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 16:33:10] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0574 s/iter. Inference: 0.2498 s/iter. Eval: 0.0024 s/iter. Total: 0.3096 s/iter. ETA=0:00:11 [09/20 16:33:15] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0757 s/iter. Inference: 0.2560 s/iter. Eval: 0.0023 s/iter. Total: 0.3343 s/iter. ETA=0:00:07 [09/20 16:33:20] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0739 s/iter. Inference: 0.2533 s/iter. Eval: 0.0026 s/iter. Total: 0.3301 s/iter. ETA=0:00:01 [09/20 16:33:22] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 16:33:22] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.256245 (0.000285 s / iter per device, on 8 devices) [09/20 16:33:22] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000221 s / iter per device, on 8 devices) [09/20 16:33:22] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 16:33:22] lb.evaluation.utils INFO: copypaste: Acc@1=77.068 [09/20 16:33:22] lb.evaluation.utils INFO: copypaste: Acc@5=93.658 [09/20 16:33:22] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 77.06800, better than last best score 76.73400 @ iteration 234999. [09/20 16:33:22] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 16:33:23] lb.utils.events INFO: eta: 20:39:57 iteration: 239999/375342 consumed_samples: 245760000 total_loss: 0.3848 time: 0.5433 s/iter data_time: 0.0515 s/iter total_throughput: 1884.69 samples/s lr: 2.95e-04 [09/20 16:34:18] lb.utils.events INFO: eta: 20:38:49 iteration: 240099/375342 consumed_samples: 245862400 total_loss: 0.3885 time: 0.5433 s/iter data_time: 0.0510 s/iter total_throughput: 1884.68 samples/s lr: 2.95e-04 [09/20 16:35:13] lb.utils.events INFO: eta: 20:37:55 iteration: 240199/375342 consumed_samples: 245964800 total_loss: 0.3902 time: 0.5433 s/iter data_time: 0.0512 s/iter total_throughput: 1884.67 samples/s lr: 2.94e-04 [09/20 16:36:08] lb.utils.events INFO: eta: 20:36:58 iteration: 240299/375342 consumed_samples: 246067200 total_loss: 0.391 time: 0.5433 s/iter data_time: 0.0508 s/iter total_throughput: 1884.67 samples/s lr: 2.94e-04 [09/20 16:37:03] lb.utils.events INFO: eta: 20:35:50 iteration: 240399/375342 consumed_samples: 246169600 total_loss: 0.3923 time: 0.5433 s/iter data_time: 0.0517 s/iter total_throughput: 1884.66 samples/s lr: 2.94e-04 [09/20 16:37:57] lb.utils.events INFO: eta: 20:34:45 iteration: 240499/375342 consumed_samples: 246272000 total_loss: 0.3887 time: 0.5433 s/iter data_time: 0.0526 s/iter total_throughput: 1884.65 samples/s lr: 2.93e-04 [09/20 16:38:52] lb.utils.events INFO: eta: 20:33:15 iteration: 240599/375342 consumed_samples: 246374400 total_loss: 0.3893 time: 0.5433 s/iter data_time: 0.0530 s/iter total_throughput: 1884.64 samples/s lr: 2.93e-04 [09/20 16:39:47] lb.utils.events INFO: eta: 20:31:49 iteration: 240699/375342 consumed_samples: 246476800 total_loss: 0.387 time: 0.5433 s/iter data_time: 0.0537 s/iter total_throughput: 1884.64 samples/s lr: 2.92e-04 [09/20 16:40:42] lb.utils.events INFO: eta: 20:30:06 iteration: 240799/375342 consumed_samples: 246579200 total_loss: 0.3828 time: 0.5433 s/iter data_time: 0.0519 s/iter total_throughput: 1884.63 samples/s lr: 2.92e-04 [09/20 16:41:36] lb.utils.events INFO: eta: 20:28:20 iteration: 240899/375342 consumed_samples: 246681600 total_loss: 0.3872 time: 0.5433 s/iter data_time: 0.0519 s/iter total_throughput: 1884.63 samples/s lr: 2.92e-04 [09/20 16:42:31] lb.utils.events INFO: eta: 20:26:18 iteration: 240999/375342 consumed_samples: 246784000 total_loss: 0.3903 time: 0.5433 s/iter data_time: 0.0543 s/iter total_throughput: 1884.63 samples/s lr: 2.91e-04 [09/20 16:43:26] lb.utils.events INFO: eta: 20:24:35 iteration: 241099/375342 consumed_samples: 246886400 total_loss: 0.3883 time: 0.5433 s/iter data_time: 0.0494 s/iter total_throughput: 1884.62 samples/s lr: 2.91e-04 [09/20 16:44:20] lb.utils.events INFO: eta: 20:23:15 iteration: 241199/375342 consumed_samples: 246988800 total_loss: 0.3811 time: 0.5433 s/iter data_time: 0.0473 s/iter total_throughput: 1884.62 samples/s lr: 2.91e-04 [09/20 16:45:15] lb.utils.events INFO: eta: 20:21:33 iteration: 241299/375342 consumed_samples: 247091200 total_loss: 0.3883 time: 0.5433 s/iter data_time: 0.0491 s/iter total_throughput: 1884.62 samples/s lr: 2.90e-04 [09/20 16:46:09] lb.utils.events INFO: eta: 20:20:03 iteration: 241399/375342 consumed_samples: 247193600 total_loss: 0.3893 time: 0.5433 s/iter data_time: 0.0487 s/iter total_throughput: 1884.61 samples/s lr: 2.90e-04 [09/20 16:47:05] lb.utils.events INFO: eta: 20:18:42 iteration: 241499/375342 consumed_samples: 247296000 total_loss: 0.3781 time: 0.5434 s/iter data_time: 0.0535 s/iter total_throughput: 1884.60 samples/s lr: 2.89e-04 [09/20 16:47:59] lb.utils.events INFO: eta: 20:17:48 iteration: 241599/375342 consumed_samples: 247398400 total_loss: 0.3834 time: 0.5434 s/iter data_time: 0.0554 s/iter total_throughput: 1884.59 samples/s lr: 2.89e-04 [09/20 16:48:54] lb.utils.events INFO: eta: 20:16:58 iteration: 241699/375342 consumed_samples: 247500800 total_loss: 0.3888 time: 0.5434 s/iter data_time: 0.0567 s/iter total_throughput: 1884.59 samples/s lr: 2.89e-04 [09/20 16:49:49] lb.utils.events INFO: eta: 20:16:22 iteration: 241799/375342 consumed_samples: 247603200 total_loss: 0.3853 time: 0.5434 s/iter data_time: 0.0545 s/iter total_throughput: 1884.58 samples/s lr: 2.88e-04 [09/20 16:50:44] lb.utils.events INFO: eta: 20:15:37 iteration: 241899/375342 consumed_samples: 247705600 total_loss: 0.3862 time: 0.5434 s/iter data_time: 0.0545 s/iter total_throughput: 1884.57 samples/s lr: 2.88e-04 [09/20 16:51:39] lb.utils.events INFO: eta: 20:15:20 iteration: 241999/375342 consumed_samples: 247808000 total_loss: 0.3899 time: 0.5434 s/iter data_time: 0.0520 s/iter total_throughput: 1884.57 samples/s lr: 2.88e-04 [09/20 16:52:34] lb.utils.events INFO: eta: 20:15:05 iteration: 242099/375342 consumed_samples: 247910400 total_loss: 0.3884 time: 0.5434 s/iter data_time: 0.0541 s/iter total_throughput: 1884.56 samples/s lr: 2.87e-04 [09/20 16:53:29] lb.utils.events INFO: eta: 20:14:48 iteration: 242199/375342 consumed_samples: 248012800 total_loss: 0.3924 time: 0.5434 s/iter data_time: 0.0560 s/iter total_throughput: 1884.55 samples/s lr: 2.87e-04 [09/20 16:54:24] lb.utils.events INFO: eta: 20:15:15 iteration: 242299/375342 consumed_samples: 248115200 total_loss: 0.3927 time: 0.5434 s/iter data_time: 0.0493 s/iter total_throughput: 1884.54 samples/s lr: 2.86e-04 [09/20 16:55:19] lb.utils.events INFO: eta: 20:15:22 iteration: 242399/375342 consumed_samples: 248217600 total_loss: 0.3962 time: 0.5434 s/iter data_time: 0.0574 s/iter total_throughput: 1884.53 samples/s lr: 2.86e-04 [09/20 16:56:14] lb.utils.events INFO: eta: 20:15:04 iteration: 242499/375342 consumed_samples: 248320000 total_loss: 0.3945 time: 0.5434 s/iter data_time: 0.0542 s/iter total_throughput: 1884.52 samples/s lr: 2.86e-04 [09/20 16:57:09] lb.utils.events INFO: eta: 20:14:19 iteration: 242599/375342 consumed_samples: 248422400 total_loss: 0.3908 time: 0.5434 s/iter data_time: 0.0533 s/iter total_throughput: 1884.52 samples/s lr: 2.85e-04 [09/20 16:58:04] lb.utils.events INFO: eta: 20:13:43 iteration: 242699/375342 consumed_samples: 248524800 total_loss: 0.391 time: 0.5434 s/iter data_time: 0.0541 s/iter total_throughput: 1884.51 samples/s lr: 2.85e-04 [09/20 16:58:58] lb.utils.events INFO: eta: 20:13:11 iteration: 242799/375342 consumed_samples: 248627200 total_loss: 0.3907 time: 0.5434 s/iter data_time: 0.0508 s/iter total_throughput: 1884.50 samples/s lr: 2.85e-04 [09/20 16:59:53] lb.utils.events INFO: eta: 20:13:27 iteration: 242899/375342 consumed_samples: 248729600 total_loss: 0.387 time: 0.5434 s/iter data_time: 0.0511 s/iter total_throughput: 1884.49 samples/s lr: 2.84e-04 [09/20 17:00:49] lb.utils.events INFO: eta: 20:13:06 iteration: 242999/375342 consumed_samples: 248832000 total_loss: 0.3844 time: 0.5434 s/iter data_time: 0.0510 s/iter total_throughput: 1884.48 samples/s lr: 2.84e-04 [09/20 17:01:44] lb.utils.events INFO: eta: 20:12:23 iteration: 243099/375342 consumed_samples: 248934400 total_loss: 0.3798 time: 0.5434 s/iter data_time: 0.0496 s/iter total_throughput: 1884.47 samples/s lr: 2.84e-04 [09/20 17:02:39] lb.utils.events INFO: eta: 20:11:30 iteration: 243199/375342 consumed_samples: 249036800 total_loss: 0.3896 time: 0.5434 s/iter data_time: 0.0513 s/iter total_throughput: 1884.46 samples/s lr: 2.83e-04 [09/20 17:03:34] lb.utils.events INFO: eta: 20:10:21 iteration: 243299/375342 consumed_samples: 249139200 total_loss: 0.3955 time: 0.5434 s/iter data_time: 0.0510 s/iter total_throughput: 1884.45 samples/s lr: 2.83e-04 [09/20 17:04:28] lb.utils.events INFO: eta: 20:09:00 iteration: 243399/375342 consumed_samples: 249241600 total_loss: 0.3925 time: 0.5434 s/iter data_time: 0.0476 s/iter total_throughput: 1884.45 samples/s lr: 2.82e-04 [09/20 17:05:23] lb.utils.events INFO: eta: 20:07:40 iteration: 243499/375342 consumed_samples: 249344000 total_loss: 0.3902 time: 0.5434 s/iter data_time: 0.0515 s/iter total_throughput: 1884.44 samples/s lr: 2.82e-04 [09/20 17:06:18] lb.utils.events INFO: eta: 20:06:04 iteration: 243599/375342 consumed_samples: 249446400 total_loss: 0.3921 time: 0.5434 s/iter data_time: 0.0507 s/iter total_throughput: 1884.43 samples/s lr: 2.82e-04 [09/20 17:07:13] lb.utils.events INFO: eta: 20:04:28 iteration: 243699/375342 consumed_samples: 249548800 total_loss: 0.3871 time: 0.5434 s/iter data_time: 0.0502 s/iter total_throughput: 1884.43 samples/s lr: 2.81e-04 [09/20 17:08:08] lb.utils.events INFO: eta: 20:03:19 iteration: 243799/375342 consumed_samples: 249651200 total_loss: 0.3782 time: 0.5434 s/iter data_time: 0.0521 s/iter total_throughput: 1884.42 samples/s lr: 2.81e-04 [09/20 17:09:02] lb.utils.events INFO: eta: 20:01:54 iteration: 243899/375342 consumed_samples: 249753600 total_loss: 0.3783 time: 0.5434 s/iter data_time: 0.0532 s/iter total_throughput: 1884.41 samples/s lr: 2.81e-04 [09/20 17:09:57] lb.utils.events INFO: eta: 19:59:49 iteration: 243999/375342 consumed_samples: 249856000 total_loss: 0.3835 time: 0.5434 s/iter data_time: 0.0535 s/iter total_throughput: 1884.41 samples/s lr: 2.80e-04 [09/20 17:10:52] lb.utils.events INFO: eta: 19:58:19 iteration: 244099/375342 consumed_samples: 249958400 total_loss: 0.3939 time: 0.5434 s/iter data_time: 0.0536 s/iter total_throughput: 1884.41 samples/s lr: 2.80e-04 [09/20 17:11:46] lb.utils.events INFO: eta: 19:56:42 iteration: 244199/375342 consumed_samples: 250060800 total_loss: 0.3866 time: 0.5434 s/iter data_time: 0.0525 s/iter total_throughput: 1884.40 samples/s lr: 2.79e-04 [09/20 17:12:41] lb.utils.events INFO: eta: 19:55:15 iteration: 244299/375342 consumed_samples: 250163200 total_loss: 0.3839 time: 0.5434 s/iter data_time: 0.0518 s/iter total_throughput: 1884.40 samples/s lr: 2.79e-04 [09/20 17:13:35] lb.utils.events INFO: eta: 19:53:48 iteration: 244399/375342 consumed_samples: 250265600 total_loss: 0.3834 time: 0.5434 s/iter data_time: 0.0511 s/iter total_throughput: 1884.40 samples/s lr: 2.79e-04 [09/20 17:14:30] lb.utils.events INFO: eta: 19:52:10 iteration: 244499/375342 consumed_samples: 250368000 total_loss: 0.3808 time: 0.5434 s/iter data_time: 0.0508 s/iter total_throughput: 1884.40 samples/s lr: 2.78e-04 [09/20 17:15:25] lb.utils.events INFO: eta: 19:50:33 iteration: 244599/375342 consumed_samples: 250470400 total_loss: 0.3815 time: 0.5434 s/iter data_time: 0.0494 s/iter total_throughput: 1884.39 samples/s lr: 2.78e-04 [09/20 17:16:19] lb.utils.events INFO: eta: 19:49:19 iteration: 244699/375342 consumed_samples: 250572800 total_loss: 0.3897 time: 0.5434 s/iter data_time: 0.0513 s/iter total_throughput: 1884.39 samples/s lr: 2.78e-04 [09/20 17:17:14] lb.utils.events INFO: eta: 19:48:04 iteration: 244799/375342 consumed_samples: 250675200 total_loss: 0.3893 time: 0.5434 s/iter data_time: 0.0484 s/iter total_throughput: 1884.39 samples/s lr: 2.77e-04 [09/20 17:18:09] lb.utils.events INFO: eta: 19:46:56 iteration: 244899/375342 consumed_samples: 250777600 total_loss: 0.3815 time: 0.5434 s/iter data_time: 0.0501 s/iter total_throughput: 1884.38 samples/s lr: 2.77e-04 [09/20 17:19:04] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0244999 [09/20 17:19:04] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 17:19:04] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 17:19:09] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0478 s/iter. Inference: 0.2502 s/iter. Eval: 0.0023 s/iter. Total: 0.3003 s/iter. ETA=0:00:11 [09/20 17:19:14] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0661 s/iter. Inference: 0.2649 s/iter. Eval: 0.0023 s/iter. Total: 0.3338 s/iter. ETA=0:00:07 [09/20 17:19:19] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0675 s/iter. Inference: 0.2595 s/iter. Eval: 0.0024 s/iter. Total: 0.3299 s/iter. ETA=0:00:01 [09/20 17:19:21] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 17:19:21] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.246478 (0.000285 s / iter per device, on 8 devices) [09/20 17:19:21] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000226 s / iter per device, on 8 devices) [09/20 17:19:21] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 17:19:21] lb.evaluation.utils INFO: copypaste: Acc@1=77.302 [09/20 17:19:21] lb.evaluation.utils INFO: copypaste: Acc@5=93.748 [09/20 17:19:21] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 77.30200, better than last best score 77.06800 @ iteration 239999. [09/20 17:19:21] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 17:19:22] lb.utils.events INFO: eta: 19:45:58 iteration: 244999/375342 consumed_samples: 250880000 total_loss: 0.3907 time: 0.5434 s/iter data_time: 0.0553 s/iter total_throughput: 1884.37 samples/s lr: 2.76e-04 [09/20 17:20:17] lb.utils.events INFO: eta: 19:45:17 iteration: 245099/375342 consumed_samples: 250982400 total_loss: 0.3897 time: 0.5434 s/iter data_time: 0.0544 s/iter total_throughput: 1884.37 samples/s lr: 2.76e-04 [09/20 17:21:11] lb.utils.events INFO: eta: 19:44:37 iteration: 245199/375342 consumed_samples: 251084800 total_loss: 0.3767 time: 0.5434 s/iter data_time: 0.0538 s/iter total_throughput: 1884.36 samples/s lr: 2.76e-04 [09/20 17:22:06] lb.utils.events INFO: eta: 19:43:50 iteration: 245299/375342 consumed_samples: 251187200 total_loss: 0.3843 time: 0.5434 s/iter data_time: 0.0538 s/iter total_throughput: 1884.35 samples/s lr: 2.75e-04 [09/20 17:23:01] lb.utils.events INFO: eta: 19:43:58 iteration: 245399/375342 consumed_samples: 251289600 total_loss: 0.3907 time: 0.5434 s/iter data_time: 0.0564 s/iter total_throughput: 1884.35 samples/s lr: 2.75e-04 [09/20 17:23:56] lb.utils.events INFO: eta: 19:44:23 iteration: 245499/375342 consumed_samples: 251392000 total_loss: 0.3877 time: 0.5434 s/iter data_time: 0.0557 s/iter total_throughput: 1884.34 samples/s lr: 2.75e-04 [09/20 17:24:51] lb.utils.events INFO: eta: 19:44:20 iteration: 245599/375342 consumed_samples: 251494400 total_loss: 0.3851 time: 0.5434 s/iter data_time: 0.0549 s/iter total_throughput: 1884.33 samples/s lr: 2.74e-04 [09/20 17:25:46] lb.utils.events INFO: eta: 19:44:22 iteration: 245699/375342 consumed_samples: 251596800 total_loss: 0.3837 time: 0.5434 s/iter data_time: 0.0551 s/iter total_throughput: 1884.32 samples/s lr: 2.74e-04 [09/20 17:26:41] lb.utils.events INFO: eta: 19:44:08 iteration: 245799/375342 consumed_samples: 251699200 total_loss: 0.3807 time: 0.5434 s/iter data_time: 0.0545 s/iter total_throughput: 1884.31 samples/s lr: 2.74e-04 [09/20 17:27:36] lb.utils.events INFO: eta: 19:44:11 iteration: 245899/375342 consumed_samples: 251801600 total_loss: 0.3818 time: 0.5434 s/iter data_time: 0.0550 s/iter total_throughput: 1884.30 samples/s lr: 2.73e-04 [09/20 17:28:31] lb.utils.events INFO: eta: 19:43:40 iteration: 245999/375342 consumed_samples: 251904000 total_loss: 0.3831 time: 0.5434 s/iter data_time: 0.0544 s/iter total_throughput: 1884.29 samples/s lr: 2.73e-04 [09/20 17:29:26] lb.utils.events INFO: eta: 19:43:02 iteration: 246099/375342 consumed_samples: 252006400 total_loss: 0.3863 time: 0.5434 s/iter data_time: 0.0547 s/iter total_throughput: 1884.29 samples/s lr: 2.72e-04 [09/20 17:30:21] lb.utils.events INFO: eta: 19:42:19 iteration: 246199/375342 consumed_samples: 252108800 total_loss: 0.3885 time: 0.5434 s/iter data_time: 0.0546 s/iter total_throughput: 1884.28 samples/s lr: 2.72e-04 [09/20 17:31:16] lb.utils.events INFO: eta: 19:41:34 iteration: 246299/375342 consumed_samples: 252211200 total_loss: 0.3876 time: 0.5434 s/iter data_time: 0.0501 s/iter total_throughput: 1884.27 samples/s lr: 2.72e-04 [09/20 17:32:11] lb.utils.events INFO: eta: 19:41:40 iteration: 246399/375342 consumed_samples: 252313600 total_loss: 0.3888 time: 0.5434 s/iter data_time: 0.0492 s/iter total_throughput: 1884.26 samples/s lr: 2.71e-04 [09/20 17:33:06] lb.utils.events INFO: eta: 19:40:17 iteration: 246499/375342 consumed_samples: 252416000 total_loss: 0.3867 time: 0.5435 s/iter data_time: 0.0464 s/iter total_throughput: 1884.25 samples/s lr: 2.71e-04 [09/20 17:34:01] lb.utils.events INFO: eta: 19:38:58 iteration: 246599/375342 consumed_samples: 252518400 total_loss: 0.3797 time: 0.5435 s/iter data_time: 0.0532 s/iter total_throughput: 1884.25 samples/s lr: 2.71e-04 [09/20 17:34:56] lb.utils.events INFO: eta: 19:37:56 iteration: 246699/375342 consumed_samples: 252620800 total_loss: 0.3869 time: 0.5435 s/iter data_time: 0.0494 s/iter total_throughput: 1884.24 samples/s lr: 2.70e-04 [09/20 17:35:51] lb.utils.events INFO: eta: 19:36:45 iteration: 246799/375342 consumed_samples: 252723200 total_loss: 0.3905 time: 0.5435 s/iter data_time: 0.0514 s/iter total_throughput: 1884.23 samples/s lr: 2.70e-04 [09/20 17:36:45] lb.utils.events INFO: eta: 19:35:20 iteration: 246899/375342 consumed_samples: 252825600 total_loss: 0.3873 time: 0.5435 s/iter data_time: 0.0508 s/iter total_throughput: 1884.23 samples/s lr: 2.70e-04 [09/20 17:37:40] lb.utils.events INFO: eta: 19:33:59 iteration: 246999/375342 consumed_samples: 252928000 total_loss: 0.3849 time: 0.5435 s/iter data_time: 0.0530 s/iter total_throughput: 1884.22 samples/s lr: 2.69e-04 [09/20 17:38:35] lb.utils.events INFO: eta: 19:32:40 iteration: 247099/375342 consumed_samples: 253030400 total_loss: 0.3876 time: 0.5435 s/iter data_time: 0.0515 s/iter total_throughput: 1884.21 samples/s lr: 2.69e-04 [09/20 17:39:30] lb.utils.events INFO: eta: 19:31:05 iteration: 247199/375342 consumed_samples: 253132800 total_loss: 0.3874 time: 0.5435 s/iter data_time: 0.0513 s/iter total_throughput: 1884.21 samples/s lr: 2.68e-04 [09/20 17:40:24] lb.utils.events INFO: eta: 19:29:24 iteration: 247299/375342 consumed_samples: 253235200 total_loss: 0.3857 time: 0.5435 s/iter data_time: 0.0522 s/iter total_throughput: 1884.21 samples/s lr: 2.68e-04 [09/20 17:41:19] lb.utils.events INFO: eta: 19:27:53 iteration: 247399/375342 consumed_samples: 253337600 total_loss: 0.3814 time: 0.5435 s/iter data_time: 0.0517 s/iter total_throughput: 1884.20 samples/s lr: 2.68e-04 [09/20 17:42:13] lb.utils.events INFO: eta: 19:26:26 iteration: 247499/375342 consumed_samples: 253440000 total_loss: 0.3873 time: 0.5435 s/iter data_time: 0.0512 s/iter total_throughput: 1884.20 samples/s lr: 2.67e-04 [09/20 17:43:08] lb.utils.events INFO: eta: 19:24:36 iteration: 247599/375342 consumed_samples: 253542400 total_loss: 0.3958 time: 0.5435 s/iter data_time: 0.0517 s/iter total_throughput: 1884.20 samples/s lr: 2.67e-04 [09/20 17:44:03] lb.utils.events INFO: eta: 19:22:48 iteration: 247699/375342 consumed_samples: 253644800 total_loss: 0.3904 time: 0.5435 s/iter data_time: 0.0530 s/iter total_throughput: 1884.19 samples/s lr: 2.67e-04 [09/20 17:44:57] lb.utils.events INFO: eta: 19:21:07 iteration: 247799/375342 consumed_samples: 253747200 total_loss: 0.386 time: 0.5435 s/iter data_time: 0.0521 s/iter total_throughput: 1884.19 samples/s lr: 2.66e-04 [09/20 17:45:52] lb.utils.events INFO: eta: 19:19:24 iteration: 247899/375342 consumed_samples: 253849600 total_loss: 0.3819 time: 0.5435 s/iter data_time: 0.0520 s/iter total_throughput: 1884.19 samples/s lr: 2.66e-04 [09/20 17:46:46] lb.utils.events INFO: eta: 19:18:20 iteration: 247999/375342 consumed_samples: 253952000 total_loss: 0.3833 time: 0.5435 s/iter data_time: 0.0490 s/iter total_throughput: 1884.19 samples/s lr: 2.66e-04 [09/20 17:47:41] lb.utils.events INFO: eta: 19:17:03 iteration: 248099/375342 consumed_samples: 254054400 total_loss: 0.3878 time: 0.5435 s/iter data_time: 0.0482 s/iter total_throughput: 1884.18 samples/s lr: 2.65e-04 [09/20 17:48:35] lb.utils.events INFO: eta: 19:15:45 iteration: 248199/375342 consumed_samples: 254156800 total_loss: 0.3862 time: 0.5435 s/iter data_time: 0.0496 s/iter total_throughput: 1884.18 samples/s lr: 2.65e-04 [09/20 17:49:30] lb.utils.events INFO: eta: 19:14:24 iteration: 248299/375342 consumed_samples: 254259200 total_loss: 0.3843 time: 0.5435 s/iter data_time: 0.0491 s/iter total_throughput: 1884.18 samples/s lr: 2.64e-04 [09/20 17:50:25] lb.utils.events INFO: eta: 19:13:27 iteration: 248399/375342 consumed_samples: 254361600 total_loss: 0.3838 time: 0.5435 s/iter data_time: 0.0523 s/iter total_throughput: 1884.17 samples/s lr: 2.64e-04 [09/20 17:51:20] lb.utils.events INFO: eta: 19:12:45 iteration: 248499/375342 consumed_samples: 254464000 total_loss: 0.3852 time: 0.5435 s/iter data_time: 0.0523 s/iter total_throughput: 1884.16 samples/s lr: 2.64e-04 [09/20 17:52:15] lb.utils.events INFO: eta: 19:12:18 iteration: 248599/375342 consumed_samples: 254566400 total_loss: 0.3862 time: 0.5435 s/iter data_time: 0.0546 s/iter total_throughput: 1884.16 samples/s lr: 2.63e-04 [09/20 17:53:09] lb.utils.events INFO: eta: 19:12:03 iteration: 248699/375342 consumed_samples: 254668800 total_loss: 0.3844 time: 0.5435 s/iter data_time: 0.0545 s/iter total_throughput: 1884.15 samples/s lr: 2.63e-04 [09/20 17:54:04] lb.utils.events INFO: eta: 19:11:59 iteration: 248799/375342 consumed_samples: 254771200 total_loss: 0.3824 time: 0.5435 s/iter data_time: 0.0560 s/iter total_throughput: 1884.15 samples/s lr: 2.63e-04 [09/20 17:54:59] lb.utils.events INFO: eta: 19:11:54 iteration: 248899/375342 consumed_samples: 254873600 total_loss: 0.3827 time: 0.5435 s/iter data_time: 0.0509 s/iter total_throughput: 1884.14 samples/s lr: 2.62e-04 [09/20 17:55:54] lb.utils.events INFO: eta: 19:12:03 iteration: 248999/375342 consumed_samples: 254976000 total_loss: 0.3861 time: 0.5435 s/iter data_time: 0.0530 s/iter total_throughput: 1884.13 samples/s lr: 2.62e-04 [09/20 17:56:49] lb.utils.events INFO: eta: 19:11:52 iteration: 249099/375342 consumed_samples: 255078400 total_loss: 0.3874 time: 0.5435 s/iter data_time: 0.0538 s/iter total_throughput: 1884.12 samples/s lr: 2.62e-04 [09/20 17:57:44] lb.utils.events INFO: eta: 19:11:55 iteration: 249199/375342 consumed_samples: 255180800 total_loss: 0.3888 time: 0.5435 s/iter data_time: 0.0546 s/iter total_throughput: 1884.11 samples/s lr: 2.61e-04 [09/20 17:58:39] lb.utils.events INFO: eta: 19:12:18 iteration: 249299/375342 consumed_samples: 255283200 total_loss: 0.3868 time: 0.5435 s/iter data_time: 0.0577 s/iter total_throughput: 1884.10 samples/s lr: 2.61e-04 [09/20 17:59:34] lb.utils.events INFO: eta: 19:11:58 iteration: 249399/375342 consumed_samples: 255385600 total_loss: 0.384 time: 0.5435 s/iter data_time: 0.0526 s/iter total_throughput: 1884.10 samples/s lr: 2.60e-04 [09/20 18:00:29] lb.utils.events INFO: eta: 19:11:12 iteration: 249499/375342 consumed_samples: 255488000 total_loss: 0.3748 time: 0.5435 s/iter data_time: 0.0562 s/iter total_throughput: 1884.09 samples/s lr: 2.60e-04 [09/20 18:01:24] lb.utils.events INFO: eta: 19:10:21 iteration: 249599/375342 consumed_samples: 255590400 total_loss: 0.3751 time: 0.5435 s/iter data_time: 0.0539 s/iter total_throughput: 1884.09 samples/s lr: 2.60e-04 [09/20 18:02:18] lb.utils.events INFO: eta: 19:09:45 iteration: 249699/375342 consumed_samples: 255692800 total_loss: 0.3801 time: 0.5435 s/iter data_time: 0.0534 s/iter total_throughput: 1884.08 samples/s lr: 2.59e-04 [09/20 18:03:13] lb.utils.events INFO: eta: 19:09:18 iteration: 249799/375342 consumed_samples: 255795200 total_loss: 0.3906 time: 0.5435 s/iter data_time: 0.0472 s/iter total_throughput: 1884.07 samples/s lr: 2.59e-04 [09/20 18:04:08] lb.utils.events INFO: eta: 19:08:50 iteration: 249899/375342 consumed_samples: 255897600 total_loss: 0.3874 time: 0.5435 s/iter data_time: 0.0493 s/iter total_throughput: 1884.06 samples/s lr: 2.59e-04 [09/20 18:05:03] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0249999 [09/20 18:05:04] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 18:05:04] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 18:05:08] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0577 s/iter. Inference: 0.2469 s/iter. Eval: 0.0021 s/iter. Total: 0.3067 s/iter. ETA=0:00:11 [09/20 18:05:13] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0665 s/iter. Inference: 0.2647 s/iter. Eval: 0.0026 s/iter. Total: 0.3341 s/iter. ETA=0:00:07 [09/20 18:05:19] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0635 s/iter. Inference: 0.2640 s/iter. Eval: 0.0024 s/iter. Total: 0.3301 s/iter. ETA=0:00:01 [09/20 18:05:21] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 18:05:21] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.323850 (0.000287 s / iter per device, on 8 devices) [09/20 18:05:21] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000234 s / iter per device, on 8 devices) [09/20 18:05:21] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 18:05:21] lb.evaluation.utils INFO: copypaste: Acc@1=77.542 [09/20 18:05:21] lb.evaluation.utils INFO: copypaste: Acc@5=93.72 [09/20 18:05:21] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 77.54200, better than last best score 77.30200 @ iteration 244999. [09/20 18:05:21] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 18:05:21] lb.utils.events INFO: eta: 19:07:09 iteration: 249999/375342 consumed_samples: 256000000 total_loss: 0.3778 time: 0.5435 s/iter data_time: 0.0503 s/iter total_throughput: 1884.06 samples/s lr: 2.58e-04 [09/20 18:06:16] lb.utils.events INFO: eta: 19:05:36 iteration: 250099/375342 consumed_samples: 256102400 total_loss: 0.3844 time: 0.5435 s/iter data_time: 0.0500 s/iter total_throughput: 1884.05 samples/s lr: 2.58e-04 [09/20 18:07:11] lb.utils.events INFO: eta: 19:04:10 iteration: 250199/375342 consumed_samples: 256204800 total_loss: 0.391 time: 0.5435 s/iter data_time: 0.0501 s/iter total_throughput: 1884.04 samples/s lr: 2.58e-04 [09/20 18:08:06] lb.utils.events INFO: eta: 19:02:36 iteration: 250299/375342 consumed_samples: 256307200 total_loss: 0.3918 time: 0.5435 s/iter data_time: 0.0498 s/iter total_throughput: 1884.04 samples/s lr: 2.57e-04 [09/20 18:09:00] lb.utils.events INFO: eta: 19:01:12 iteration: 250399/375342 consumed_samples: 256409600 total_loss: 0.3843 time: 0.5435 s/iter data_time: 0.0526 s/iter total_throughput: 1884.04 samples/s lr: 2.57e-04 [09/20 18:09:55] lb.utils.events INFO: eta: 18:59:53 iteration: 250499/375342 consumed_samples: 256512000 total_loss: 0.3815 time: 0.5435 s/iter data_time: 0.0506 s/iter total_throughput: 1884.03 samples/s lr: 2.57e-04 [09/20 18:10:50] lb.utils.events INFO: eta: 18:58:35 iteration: 250599/375342 consumed_samples: 256614400 total_loss: 0.3876 time: 0.5435 s/iter data_time: 0.0512 s/iter total_throughput: 1884.03 samples/s lr: 2.56e-04 [09/20 18:11:44] lb.utils.events INFO: eta: 18:57:00 iteration: 250699/375342 consumed_samples: 256716800 total_loss: 0.387 time: 0.5435 s/iter data_time: 0.0510 s/iter total_throughput: 1884.02 samples/s lr: 2.56e-04 [09/20 18:12:39] lb.utils.events INFO: eta: 18:55:41 iteration: 250799/375342 consumed_samples: 256819200 total_loss: 0.3838 time: 0.5435 s/iter data_time: 0.0508 s/iter total_throughput: 1884.02 samples/s lr: 2.55e-04 [09/20 18:13:33] lb.utils.events INFO: eta: 18:54:02 iteration: 250899/375342 consumed_samples: 256921600 total_loss: 0.3869 time: 0.5435 s/iter data_time: 0.0539 s/iter total_throughput: 1884.02 samples/s lr: 2.55e-04 [09/20 18:14:28] lb.utils.events INFO: eta: 18:52:27 iteration: 250999/375342 consumed_samples: 257024000 total_loss: 0.3852 time: 0.5435 s/iter data_time: 0.0510 s/iter total_throughput: 1884.02 samples/s lr: 2.55e-04 [09/20 18:15:23] lb.utils.events INFO: eta: 18:50:49 iteration: 251099/375342 consumed_samples: 257126400 total_loss: 0.3747 time: 0.5435 s/iter data_time: 0.0528 s/iter total_throughput: 1884.01 samples/s lr: 2.54e-04 [09/20 18:16:17] lb.utils.events INFO: eta: 18:49:13 iteration: 251199/375342 consumed_samples: 257228800 total_loss: 0.3809 time: 0.5435 s/iter data_time: 0.0524 s/iter total_throughput: 1884.01 samples/s lr: 2.54e-04 [09/20 18:17:12] lb.utils.events INFO: eta: 18:47:57 iteration: 251299/375342 consumed_samples: 257331200 total_loss: 0.38 time: 0.5435 s/iter data_time: 0.0522 s/iter total_throughput: 1884.01 samples/s lr: 2.54e-04 [09/20 18:18:06] lb.utils.events INFO: eta: 18:46:43 iteration: 251399/375342 consumed_samples: 257433600 total_loss: 0.3817 time: 0.5435 s/iter data_time: 0.0513 s/iter total_throughput: 1884.01 samples/s lr: 2.53e-04 [09/20 18:19:00] lb.utils.events INFO: eta: 18:45:41 iteration: 251499/375342 consumed_samples: 257536000 total_loss: 0.3846 time: 0.5435 s/iter data_time: 0.0485 s/iter total_throughput: 1884.01 samples/s lr: 2.53e-04 [09/20 18:19:55] lb.utils.events INFO: eta: 18:44:19 iteration: 251599/375342 consumed_samples: 257638400 total_loss: 0.379 time: 0.5435 s/iter data_time: 0.0479 s/iter total_throughput: 1884.01 samples/s lr: 2.53e-04 [09/20 18:20:50] lb.utils.events INFO: eta: 18:43:01 iteration: 251699/375342 consumed_samples: 257740800 total_loss: 0.382 time: 0.5435 s/iter data_time: 0.0493 s/iter total_throughput: 1884.00 samples/s lr: 2.52e-04 [09/20 18:21:45] lb.utils.events INFO: eta: 18:41:58 iteration: 251799/375342 consumed_samples: 257843200 total_loss: 0.3933 time: 0.5435 s/iter data_time: 0.0575 s/iter total_throughput: 1884.00 samples/s lr: 2.52e-04 [09/20 18:22:39] lb.utils.events INFO: eta: 18:41:33 iteration: 251899/375342 consumed_samples: 257945600 total_loss: 0.3919 time: 0.5435 s/iter data_time: 0.0538 s/iter total_throughput: 1883.99 samples/s lr: 2.52e-04 [09/20 18:23:34] lb.utils.events INFO: eta: 18:41:16 iteration: 251999/375342 consumed_samples: 258048000 total_loss: 0.3823 time: 0.5435 s/iter data_time: 0.0550 s/iter total_throughput: 1883.98 samples/s lr: 2.51e-04 [09/20 18:24:29] lb.utils.events INFO: eta: 18:40:43 iteration: 252099/375342 consumed_samples: 258150400 total_loss: 0.3824 time: 0.5435 s/iter data_time: 0.0517 s/iter total_throughput: 1883.98 samples/s lr: 2.51e-04 [09/20 18:25:24] lb.utils.events INFO: eta: 18:40:51 iteration: 252199/375342 consumed_samples: 258252800 total_loss: 0.3852 time: 0.5435 s/iter data_time: 0.0559 s/iter total_throughput: 1883.97 samples/s lr: 2.50e-04 [09/20 18:26:19] lb.utils.events INFO: eta: 18:40:23 iteration: 252299/375342 consumed_samples: 258355200 total_loss: 0.3888 time: 0.5435 s/iter data_time: 0.0542 s/iter total_throughput: 1883.97 samples/s lr: 2.50e-04 [09/20 18:27:14] lb.utils.events INFO: eta: 18:40:41 iteration: 252399/375342 consumed_samples: 258457600 total_loss: 0.3883 time: 0.5435 s/iter data_time: 0.0542 s/iter total_throughput: 1883.96 samples/s lr: 2.50e-04 [09/20 18:28:09] lb.utils.events INFO: eta: 18:40:43 iteration: 252499/375342 consumed_samples: 258560000 total_loss: 0.3798 time: 0.5435 s/iter data_time: 0.0545 s/iter total_throughput: 1883.95 samples/s lr: 2.49e-04 [09/20 18:29:04] lb.utils.events INFO: eta: 18:41:09 iteration: 252599/375342 consumed_samples: 258662400 total_loss: 0.3822 time: 0.5435 s/iter data_time: 0.0528 s/iter total_throughput: 1883.94 samples/s lr: 2.49e-04 [09/20 18:29:58] lb.utils.events INFO: eta: 18:40:59 iteration: 252699/375342 consumed_samples: 258764800 total_loss: 0.3857 time: 0.5435 s/iter data_time: 0.0535 s/iter total_throughput: 1883.93 samples/s lr: 2.49e-04 [09/20 18:30:53] lb.utils.events INFO: eta: 18:40:40 iteration: 252799/375342 consumed_samples: 258867200 total_loss: 0.3867 time: 0.5435 s/iter data_time: 0.0552 s/iter total_throughput: 1883.93 samples/s lr: 2.48e-04 [09/20 18:31:48] lb.utils.events INFO: eta: 18:39:45 iteration: 252899/375342 consumed_samples: 258969600 total_loss: 0.3878 time: 0.5435 s/iter data_time: 0.0559 s/iter total_throughput: 1883.92 samples/s lr: 2.48e-04 [09/20 18:32:43] lb.utils.events INFO: eta: 18:38:46 iteration: 252999/375342 consumed_samples: 259072000 total_loss: 0.3846 time: 0.5435 s/iter data_time: 0.0555 s/iter total_throughput: 1883.92 samples/s lr: 2.48e-04 [09/20 18:33:38] lb.utils.events INFO: eta: 18:37:30 iteration: 253099/375342 consumed_samples: 259174400 total_loss: 0.3805 time: 0.5436 s/iter data_time: 0.0494 s/iter total_throughput: 1883.91 samples/s lr: 2.47e-04 [09/20 18:34:33] lb.utils.events INFO: eta: 18:37:04 iteration: 253199/375342 consumed_samples: 259276800 total_loss: 0.3821 time: 0.5436 s/iter data_time: 0.0506 s/iter total_throughput: 1883.90 samples/s lr: 2.47e-04 [09/20 18:35:28] lb.utils.events INFO: eta: 18:36:25 iteration: 253299/375342 consumed_samples: 259379200 total_loss: 0.3857 time: 0.5436 s/iter data_time: 0.0506 s/iter total_throughput: 1883.89 samples/s lr: 2.47e-04 [09/20 18:36:23] lb.utils.events INFO: eta: 18:35:28 iteration: 253399/375342 consumed_samples: 259481600 total_loss: 0.3786 time: 0.5436 s/iter data_time: 0.0490 s/iter total_throughput: 1883.89 samples/s lr: 2.46e-04 [09/20 18:37:18] lb.utils.events INFO: eta: 18:34:26 iteration: 253499/375342 consumed_samples: 259584000 total_loss: 0.3792 time: 0.5436 s/iter data_time: 0.0506 s/iter total_throughput: 1883.88 samples/s lr: 2.46e-04 [09/20 18:38:13] lb.utils.events INFO: eta: 18:33:22 iteration: 253599/375342 consumed_samples: 259686400 total_loss: 0.3797 time: 0.5436 s/iter data_time: 0.0528 s/iter total_throughput: 1883.87 samples/s lr: 2.46e-04 [09/20 18:39:07] lb.utils.events INFO: eta: 18:32:28 iteration: 253699/375342 consumed_samples: 259788800 total_loss: 0.3789 time: 0.5436 s/iter data_time: 0.0518 s/iter total_throughput: 1883.86 samples/s lr: 2.45e-04 [09/20 18:40:02] lb.utils.events INFO: eta: 18:31:17 iteration: 253799/375342 consumed_samples: 259891200 total_loss: 0.382 time: 0.5436 s/iter data_time: 0.0510 s/iter total_throughput: 1883.86 samples/s lr: 2.45e-04 [09/20 18:40:57] lb.utils.events INFO: eta: 18:30:42 iteration: 253899/375342 consumed_samples: 259993600 total_loss: 0.3847 time: 0.5436 s/iter data_time: 0.0522 s/iter total_throughput: 1883.85 samples/s lr: 2.44e-04 [09/20 18:41:52] lb.utils.events INFO: eta: 18:29:59 iteration: 253999/375342 consumed_samples: 260096000 total_loss: 0.3823 time: 0.5436 s/iter data_time: 0.0532 s/iter total_throughput: 1883.84 samples/s lr: 2.44e-04 [09/20 18:42:47] lb.utils.events INFO: eta: 18:29:10 iteration: 254099/375342 consumed_samples: 260198400 total_loss: 0.3833 time: 0.5436 s/iter data_time: 0.0533 s/iter total_throughput: 1883.84 samples/s lr: 2.44e-04 [09/20 18:43:42] lb.utils.events INFO: eta: 18:27:36 iteration: 254199/375342 consumed_samples: 260300800 total_loss: 0.3826 time: 0.5436 s/iter data_time: 0.0520 s/iter total_throughput: 1883.83 samples/s lr: 2.43e-04 [09/20 18:44:36] lb.utils.events INFO: eta: 18:26:05 iteration: 254299/375342 consumed_samples: 260403200 total_loss: 0.3828 time: 0.5436 s/iter data_time: 0.0519 s/iter total_throughput: 1883.83 samples/s lr: 2.43e-04 [09/20 18:45:31] lb.utils.events INFO: eta: 18:24:34 iteration: 254399/375342 consumed_samples: 260505600 total_loss: 0.3865 time: 0.5436 s/iter data_time: 0.0504 s/iter total_throughput: 1883.83 samples/s lr: 2.43e-04 [09/20 18:46:25] lb.utils.events INFO: eta: 18:22:54 iteration: 254499/375342 consumed_samples: 260608000 total_loss: 0.3872 time: 0.5436 s/iter data_time: 0.0507 s/iter total_throughput: 1883.83 samples/s lr: 2.42e-04 [09/20 18:47:20] lb.utils.events INFO: eta: 18:21:15 iteration: 254599/375342 consumed_samples: 260710400 total_loss: 0.3872 time: 0.5436 s/iter data_time: 0.0533 s/iter total_throughput: 1883.82 samples/s lr: 2.42e-04 [09/20 18:48:14] lb.utils.events INFO: eta: 18:19:56 iteration: 254699/375342 consumed_samples: 260812800 total_loss: 0.3826 time: 0.5436 s/iter data_time: 0.0514 s/iter total_throughput: 1883.82 samples/s lr: 2.42e-04 [09/20 18:49:09] lb.utils.events INFO: eta: 18:18:40 iteration: 254799/375342 consumed_samples: 260915200 total_loss: 0.3802 time: 0.5436 s/iter data_time: 0.0523 s/iter total_throughput: 1883.82 samples/s lr: 2.41e-04 [09/20 18:50:04] lb.utils.events INFO: eta: 18:16:48 iteration: 254899/375342 consumed_samples: 261017600 total_loss: 0.3814 time: 0.5436 s/iter data_time: 0.0481 s/iter total_throughput: 1883.82 samples/s lr: 2.41e-04 [09/20 18:50:58] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0254999 [09/20 18:50:59] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 18:50:59] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 18:51:03] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0575 s/iter. Inference: 0.2522 s/iter. Eval: 0.0045 s/iter. Total: 0.3142 s/iter. ETA=0:00:11 [09/20 18:51:09] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0720 s/iter. Inference: 0.2606 s/iter. Eval: 0.0031 s/iter. Total: 0.3359 s/iter. ETA=0:00:07 [09/20 18:51:14] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0700 s/iter. Inference: 0.2585 s/iter. Eval: 0.0029 s/iter. Total: 0.3317 s/iter. ETA=0:00:01 [09/20 18:51:16] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 18:51:16] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.246918 (0.000285 s / iter per device, on 8 devices) [09/20 18:51:16] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000225 s / iter per device, on 8 devices) [09/20 18:51:16] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 18:51:16] lb.evaluation.utils INFO: copypaste: Acc@1=77.694 [09/20 18:51:16] lb.evaluation.utils INFO: copypaste: Acc@5=93.854 [09/20 18:51:16] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 77.69400, better than last best score 77.54200 @ iteration 249999. [09/20 18:51:16] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 18:51:17] lb.utils.events INFO: eta: 18:15:32 iteration: 254999/375342 consumed_samples: 261120000 total_loss: 0.386 time: 0.5436 s/iter data_time: 0.0511 s/iter total_throughput: 1883.81 samples/s lr: 2.41e-04 [09/20 18:52:11] lb.utils.events INFO: eta: 18:13:55 iteration: 255099/375342 consumed_samples: 261222400 total_loss: 0.3831 time: 0.5436 s/iter data_time: 0.0502 s/iter total_throughput: 1883.81 samples/s lr: 2.40e-04 [09/20 18:53:06] lb.utils.events INFO: eta: 18:12:22 iteration: 255199/375342 consumed_samples: 261324800 total_loss: 0.3807 time: 0.5436 s/iter data_time: 0.0501 s/iter total_throughput: 1883.81 samples/s lr: 2.40e-04 [09/20 18:54:01] lb.utils.events INFO: eta: 18:11:17 iteration: 255299/375342 consumed_samples: 261427200 total_loss: 0.3804 time: 0.5436 s/iter data_time: 0.0544 s/iter total_throughput: 1883.80 samples/s lr: 2.40e-04 [09/20 18:54:55] lb.utils.events INFO: eta: 18:10:35 iteration: 255399/375342 consumed_samples: 261529600 total_loss: 0.3793 time: 0.5436 s/iter data_time: 0.0546 s/iter total_throughput: 1883.79 samples/s lr: 2.39e-04 [09/20 18:55:50] lb.utils.events INFO: eta: 18:10:29 iteration: 255499/375342 consumed_samples: 261632000 total_loss: 0.3783 time: 0.5436 s/iter data_time: 0.0556 s/iter total_throughput: 1883.79 samples/s lr: 2.39e-04 [09/20 18:56:45] lb.utils.events INFO: eta: 18:09:50 iteration: 255599/375342 consumed_samples: 261734400 total_loss: 0.3803 time: 0.5436 s/iter data_time: 0.0519 s/iter total_throughput: 1883.78 samples/s lr: 2.38e-04 [09/20 18:57:40] lb.utils.events INFO: eta: 18:09:21 iteration: 255699/375342 consumed_samples: 261836800 total_loss: 0.3776 time: 0.5436 s/iter data_time: 0.0556 s/iter total_throughput: 1883.78 samples/s lr: 2.38e-04 [09/20 18:58:35] lb.utils.events INFO: eta: 18:08:43 iteration: 255799/375342 consumed_samples: 261939200 total_loss: 0.3774 time: 0.5436 s/iter data_time: 0.0556 s/iter total_throughput: 1883.77 samples/s lr: 2.38e-04 [09/20 18:59:30] lb.utils.events INFO: eta: 18:08:37 iteration: 255899/375342 consumed_samples: 262041600 total_loss: 0.3836 time: 0.5436 s/iter data_time: 0.0553 s/iter total_throughput: 1883.76 samples/s lr: 2.37e-04 [09/20 19:00:25] lb.utils.events INFO: eta: 18:08:09 iteration: 255999/375342 consumed_samples: 262144000 total_loss: 0.3812 time: 0.5436 s/iter data_time: 0.0549 s/iter total_throughput: 1883.76 samples/s lr: 2.37e-04 [09/20 19:01:20] lb.utils.events INFO: eta: 18:08:37 iteration: 256099/375342 consumed_samples: 262246400 total_loss: 0.3808 time: 0.5436 s/iter data_time: 0.0548 s/iter total_throughput: 1883.75 samples/s lr: 2.37e-04 [09/20 19:02:14] lb.utils.events INFO: eta: 18:08:24 iteration: 256199/375342 consumed_samples: 262348800 total_loss: 0.3849 time: 0.5436 s/iter data_time: 0.0527 s/iter total_throughput: 1883.74 samples/s lr: 2.36e-04 [09/20 19:03:09] lb.utils.events INFO: eta: 18:08:04 iteration: 256299/375342 consumed_samples: 262451200 total_loss: 0.385 time: 0.5436 s/iter data_time: 0.0572 s/iter total_throughput: 1883.73 samples/s lr: 2.36e-04 [09/20 19:04:04] lb.utils.events INFO: eta: 18:07:25 iteration: 256399/375342 consumed_samples: 262553600 total_loss: 0.3844 time: 0.5436 s/iter data_time: 0.0549 s/iter total_throughput: 1883.73 samples/s lr: 2.36e-04 [09/20 19:04:59] lb.utils.events INFO: eta: 18:06:37 iteration: 256499/375342 consumed_samples: 262656000 total_loss: 0.3844 time: 0.5436 s/iter data_time: 0.0571 s/iter total_throughput: 1883.72 samples/s lr: 2.35e-04 [09/20 19:05:54] lb.utils.events INFO: eta: 18:06:15 iteration: 256599/375342 consumed_samples: 262758400 total_loss: 0.3815 time: 0.5436 s/iter data_time: 0.0525 s/iter total_throughput: 1883.71 samples/s lr: 2.35e-04 [09/20 19:06:49] lb.utils.events INFO: eta: 18:05:25 iteration: 256699/375342 consumed_samples: 262860800 total_loss: 0.3831 time: 0.5436 s/iter data_time: 0.0493 s/iter total_throughput: 1883.71 samples/s lr: 2.35e-04 [09/20 19:07:44] lb.utils.events INFO: eta: 18:04:35 iteration: 256799/375342 consumed_samples: 262963200 total_loss: 0.386 time: 0.5436 s/iter data_time: 0.0502 s/iter total_throughput: 1883.70 samples/s lr: 2.34e-04 [09/20 19:08:39] lb.utils.events INFO: eta: 18:03:29 iteration: 256899/375342 consumed_samples: 263065600 total_loss: 0.3853 time: 0.5436 s/iter data_time: 0.0511 s/iter total_throughput: 1883.69 samples/s lr: 2.34e-04 [09/20 19:09:34] lb.utils.events INFO: eta: 18:02:24 iteration: 256999/375342 consumed_samples: 263168000 total_loss: 0.3896 time: 0.5436 s/iter data_time: 0.0521 s/iter total_throughput: 1883.69 samples/s lr: 2.34e-04 [09/20 19:10:28] lb.utils.events INFO: eta: 18:01:00 iteration: 257099/375342 consumed_samples: 263270400 total_loss: 0.3885 time: 0.5436 s/iter data_time: 0.0514 s/iter total_throughput: 1883.68 samples/s lr: 2.33e-04 [09/20 19:11:23] lb.utils.events INFO: eta: 17:59:52 iteration: 257199/375342 consumed_samples: 263372800 total_loss: 0.3815 time: 0.5436 s/iter data_time: 0.0511 s/iter total_throughput: 1883.68 samples/s lr: 2.33e-04 [09/20 19:12:18] lb.utils.events INFO: eta: 17:58:45 iteration: 257299/375342 consumed_samples: 263475200 total_loss: 0.3766 time: 0.5436 s/iter data_time: 0.0498 s/iter total_throughput: 1883.67 samples/s lr: 2.33e-04 [09/20 19:13:13] lb.utils.events INFO: eta: 17:57:29 iteration: 257399/375342 consumed_samples: 263577600 total_loss: 0.3754 time: 0.5436 s/iter data_time: 0.0514 s/iter total_throughput: 1883.67 samples/s lr: 2.32e-04 [09/20 19:14:07] lb.utils.events INFO: eta: 17:56:13 iteration: 257499/375342 consumed_samples: 263680000 total_loss: 0.3748 time: 0.5436 s/iter data_time: 0.0518 s/iter total_throughput: 1883.66 samples/s lr: 2.32e-04 [09/20 19:15:02] lb.utils.events INFO: eta: 17:54:42 iteration: 257599/375342 consumed_samples: 263782400 total_loss: 0.3829 time: 0.5436 s/iter data_time: 0.0522 s/iter total_throughput: 1883.66 samples/s lr: 2.32e-04 [09/20 19:15:57] lb.utils.events INFO: eta: 17:53:23 iteration: 257699/375342 consumed_samples: 263884800 total_loss: 0.39 time: 0.5436 s/iter data_time: 0.0512 s/iter total_throughput: 1883.66 samples/s lr: 2.31e-04 [09/20 19:16:51] lb.utils.events INFO: eta: 17:51:49 iteration: 257799/375342 consumed_samples: 263987200 total_loss: 0.3836 time: 0.5436 s/iter data_time: 0.0515 s/iter total_throughput: 1883.65 samples/s lr: 2.31e-04 [09/20 19:17:46] lb.utils.events INFO: eta: 17:50:09 iteration: 257899/375342 consumed_samples: 264089600 total_loss: 0.3815 time: 0.5436 s/iter data_time: 0.0530 s/iter total_throughput: 1883.65 samples/s lr: 2.31e-04 [09/20 19:18:40] lb.utils.events INFO: eta: 17:48:41 iteration: 257999/375342 consumed_samples: 264192000 total_loss: 0.3805 time: 0.5436 s/iter data_time: 0.0534 s/iter total_throughput: 1883.65 samples/s lr: 2.30e-04 [09/20 19:19:35] lb.utils.events INFO: eta: 17:47:26 iteration: 258099/375342 consumed_samples: 264294400 total_loss: 0.3815 time: 0.5436 s/iter data_time: 0.0524 s/iter total_throughput: 1883.65 samples/s lr: 2.30e-04 [09/20 19:20:29] lb.utils.events INFO: eta: 17:46:06 iteration: 258199/375342 consumed_samples: 264396800 total_loss: 0.3804 time: 0.5436 s/iter data_time: 0.0522 s/iter total_throughput: 1883.64 samples/s lr: 2.29e-04 [09/20 19:21:24] lb.utils.events INFO: eta: 17:44:37 iteration: 258299/375342 consumed_samples: 264499200 total_loss: 0.3809 time: 0.5436 s/iter data_time: 0.0520 s/iter total_throughput: 1883.64 samples/s lr: 2.29e-04 [09/20 19:22:18] lb.utils.events INFO: eta: 17:43:21 iteration: 258399/375342 consumed_samples: 264601600 total_loss: 0.3821 time: 0.5436 s/iter data_time: 0.0463 s/iter total_throughput: 1883.64 samples/s lr: 2.29e-04 [09/20 19:23:13] lb.utils.events INFO: eta: 17:42:15 iteration: 258499/375342 consumed_samples: 264704000 total_loss: 0.3781 time: 0.5436 s/iter data_time: 0.0501 s/iter total_throughput: 1883.64 samples/s lr: 2.28e-04 [09/20 19:24:08] lb.utils.events INFO: eta: 17:41:23 iteration: 258599/375342 consumed_samples: 264806400 total_loss: 0.3789 time: 0.5436 s/iter data_time: 0.0499 s/iter total_throughput: 1883.63 samples/s lr: 2.28e-04 [09/20 19:25:03] lb.utils.events INFO: eta: 17:40:26 iteration: 258699/375342 consumed_samples: 264908800 total_loss: 0.378 time: 0.5436 s/iter data_time: 0.0574 s/iter total_throughput: 1883.62 samples/s lr: 2.28e-04 [09/20 19:25:58] lb.utils.events INFO: eta: 17:40:14 iteration: 258799/375342 consumed_samples: 265011200 total_loss: 0.375 time: 0.5436 s/iter data_time: 0.0532 s/iter total_throughput: 1883.62 samples/s lr: 2.27e-04 [09/20 19:26:52] lb.utils.events INFO: eta: 17:39:55 iteration: 258899/375342 consumed_samples: 265113600 total_loss: 0.3815 time: 0.5436 s/iter data_time: 0.0533 s/iter total_throughput: 1883.61 samples/s lr: 2.27e-04 [09/20 19:27:47] lb.utils.events INFO: eta: 17:39:05 iteration: 258999/375342 consumed_samples: 265216000 total_loss: 0.3784 time: 0.5436 s/iter data_time: 0.0544 s/iter total_throughput: 1883.61 samples/s lr: 2.27e-04 [09/20 19:28:42] lb.utils.events INFO: eta: 17:38:52 iteration: 259099/375342 consumed_samples: 265318400 total_loss: 0.3754 time: 0.5436 s/iter data_time: 0.0542 s/iter total_throughput: 1883.60 samples/s lr: 2.26e-04 [09/20 19:29:37] lb.utils.events INFO: eta: 17:38:27 iteration: 259199/375342 consumed_samples: 265420800 total_loss: 0.3805 time: 0.5436 s/iter data_time: 0.0565 s/iter total_throughput: 1883.60 samples/s lr: 2.26e-04 [09/20 19:30:32] lb.utils.events INFO: eta: 17:38:43 iteration: 259299/375342 consumed_samples: 265523200 total_loss: 0.3824 time: 0.5436 s/iter data_time: 0.0536 s/iter total_throughput: 1883.59 samples/s lr: 2.26e-04 [09/20 19:31:27] lb.utils.events INFO: eta: 17:38:53 iteration: 259399/375342 consumed_samples: 265625600 total_loss: 0.374 time: 0.5436 s/iter data_time: 0.0518 s/iter total_throughput: 1883.58 samples/s lr: 2.25e-04 [09/20 19:32:22] lb.utils.events INFO: eta: 17:38:14 iteration: 259499/375342 consumed_samples: 265728000 total_loss: 0.3687 time: 0.5436 s/iter data_time: 0.0575 s/iter total_throughput: 1883.57 samples/s lr: 2.25e-04 [09/20 19:33:16] lb.utils.events INFO: eta: 17:37:43 iteration: 259599/375342 consumed_samples: 265830400 total_loss: 0.3759 time: 0.5436 s/iter data_time: 0.0557 s/iter total_throughput: 1883.57 samples/s lr: 2.25e-04 [09/20 19:34:11] lb.utils.events INFO: eta: 17:37:03 iteration: 259699/375342 consumed_samples: 265932800 total_loss: 0.3856 time: 0.5437 s/iter data_time: 0.0557 s/iter total_throughput: 1883.56 samples/s lr: 2.24e-04 [09/20 19:35:06] lb.utils.events INFO: eta: 17:36:08 iteration: 259799/375342 consumed_samples: 266035200 total_loss: 0.3801 time: 0.5437 s/iter data_time: 0.0562 s/iter total_throughput: 1883.56 samples/s lr: 2.24e-04 [09/20 19:36:01] lb.utils.events INFO: eta: 17:35:13 iteration: 259899/375342 consumed_samples: 266137600 total_loss: 0.3718 time: 0.5437 s/iter data_time: 0.0543 s/iter total_throughput: 1883.55 samples/s lr: 2.24e-04 [09/20 19:36:56] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0259999 [09/20 19:36:56] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 19:36:56] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 19:37:01] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0479 s/iter. Inference: 0.2503 s/iter. Eval: 0.0023 s/iter. Total: 0.3005 s/iter. ETA=0:00:11 [09/20 19:37:06] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0786 s/iter. Inference: 0.2517 s/iter. Eval: 0.0022 s/iter. Total: 0.3329 s/iter. ETA=0:00:07 [09/20 19:37:12] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0778 s/iter. Inference: 0.2501 s/iter. Eval: 0.0022 s/iter. Total: 0.3304 s/iter. ETA=0:00:01 [09/20 19:37:14] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 19:37:14] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.235490 (0.000285 s / iter per device, on 8 devices) [09/20 19:37:14] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000219 s / iter per device, on 8 devices) [09/20 19:37:14] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 19:37:14] lb.evaluation.utils INFO: copypaste: Acc@1=77.534 [09/20 19:37:14] lb.evaluation.utils INFO: copypaste: Acc@5=93.85 [09/20 19:37:14] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 77.53400, not better than best score 77.69400 @ iteration 254999. [09/20 19:37:14] lb.utils.events INFO: eta: 17:34:41 iteration: 259999/375342 consumed_samples: 266240000 total_loss: 0.3799 time: 0.5437 s/iter data_time: 0.0508 s/iter total_throughput: 1883.54 samples/s lr: 2.23e-04 [09/20 19:38:08] lb.utils.events INFO: eta: 17:33:46 iteration: 260099/375342 consumed_samples: 266342400 total_loss: 0.383 time: 0.5437 s/iter data_time: 0.0489 s/iter total_throughput: 1883.54 samples/s lr: 2.23e-04 [09/20 19:39:03] lb.utils.events INFO: eta: 17:32:40 iteration: 260199/375342 consumed_samples: 266444800 total_loss: 0.3806 time: 0.5437 s/iter data_time: 0.0485 s/iter total_throughput: 1883.53 samples/s lr: 2.23e-04 [09/20 19:39:58] lb.utils.events INFO: eta: 17:31:42 iteration: 260299/375342 consumed_samples: 266547200 total_loss: 0.3818 time: 0.5437 s/iter data_time: 0.0496 s/iter total_throughput: 1883.53 samples/s lr: 2.22e-04 [09/20 19:40:53] lb.utils.events INFO: eta: 17:30:16 iteration: 260399/375342 consumed_samples: 266649600 total_loss: 0.3787 time: 0.5437 s/iter data_time: 0.0518 s/iter total_throughput: 1883.52 samples/s lr: 2.22e-04 [09/20 19:41:48] lb.utils.events INFO: eta: 17:29:10 iteration: 260499/375342 consumed_samples: 266752000 total_loss: 0.3723 time: 0.5437 s/iter data_time: 0.0509 s/iter total_throughput: 1883.52 samples/s lr: 2.22e-04 [09/20 19:42:42] lb.utils.events INFO: eta: 17:27:58 iteration: 260599/375342 consumed_samples: 266854400 total_loss: 0.3789 time: 0.5437 s/iter data_time: 0.0514 s/iter total_throughput: 1883.51 samples/s lr: 2.21e-04 [09/20 19:43:37] lb.utils.events INFO: eta: 17:26:46 iteration: 260699/375342 consumed_samples: 266956800 total_loss: 0.3815 time: 0.5437 s/iter data_time: 0.0507 s/iter total_throughput: 1883.51 samples/s lr: 2.21e-04 [09/20 19:44:32] lb.utils.events INFO: eta: 17:25:25 iteration: 260799/375342 consumed_samples: 267059200 total_loss: 0.3809 time: 0.5437 s/iter data_time: 0.0514 s/iter total_throughput: 1883.51 samples/s lr: 2.21e-04 [09/20 19:45:26] lb.utils.events INFO: eta: 17:24:18 iteration: 260899/375342 consumed_samples: 267161600 total_loss: 0.3803 time: 0.5437 s/iter data_time: 0.0504 s/iter total_throughput: 1883.50 samples/s lr: 2.20e-04 [09/20 19:46:21] lb.utils.events INFO: eta: 17:22:55 iteration: 260999/375342 consumed_samples: 267264000 total_loss: 0.3758 time: 0.5437 s/iter data_time: 0.0515 s/iter total_throughput: 1883.50 samples/s lr: 2.20e-04 [09/20 19:47:16] lb.utils.events INFO: eta: 17:21:22 iteration: 261099/375342 consumed_samples: 267366400 total_loss: 0.3717 time: 0.5437 s/iter data_time: 0.0520 s/iter total_throughput: 1883.50 samples/s lr: 2.20e-04 [09/20 19:48:10] lb.utils.events INFO: eta: 17:20:12 iteration: 261199/375342 consumed_samples: 267468800 total_loss: 0.3719 time: 0.5437 s/iter data_time: 0.0510 s/iter total_throughput: 1883.49 samples/s lr: 2.19e-04 [09/20 19:49:05] lb.utils.events INFO: eta: 17:18:38 iteration: 261299/375342 consumed_samples: 267571200 total_loss: 0.3734 time: 0.5437 s/iter data_time: 0.0522 s/iter total_throughput: 1883.49 samples/s lr: 2.19e-04 [09/20 19:49:59] lb.utils.events INFO: eta: 17:17:20 iteration: 261399/375342 consumed_samples: 267673600 total_loss: 0.3788 time: 0.5437 s/iter data_time: 0.0499 s/iter total_throughput: 1883.49 samples/s lr: 2.19e-04 [09/20 19:50:54] lb.utils.events INFO: eta: 17:16:00 iteration: 261499/375342 consumed_samples: 267776000 total_loss: 0.3753 time: 0.5437 s/iter data_time: 0.0525 s/iter total_throughput: 1883.48 samples/s lr: 2.18e-04 [09/20 19:51:49] lb.utils.events INFO: eta: 17:15:09 iteration: 261599/375342 consumed_samples: 267878400 total_loss: 0.3686 time: 0.5437 s/iter data_time: 0.0483 s/iter total_throughput: 1883.47 samples/s lr: 2.18e-04 [09/20 19:52:44] lb.utils.events INFO: eta: 17:14:31 iteration: 261699/375342 consumed_samples: 267980800 total_loss: 0.3724 time: 0.5437 s/iter data_time: 0.0504 s/iter total_throughput: 1883.47 samples/s lr: 2.18e-04 [09/20 19:53:39] lb.utils.events INFO: eta: 17:13:49 iteration: 261799/375342 consumed_samples: 268083200 total_loss: 0.3739 time: 0.5437 s/iter data_time: 0.0524 s/iter total_throughput: 1883.46 samples/s lr: 2.17e-04 [09/20 19:54:34] lb.utils.events INFO: eta: 17:13:32 iteration: 261899/375342 consumed_samples: 268185600 total_loss: 0.3718 time: 0.5437 s/iter data_time: 0.0528 s/iter total_throughput: 1883.45 samples/s lr: 2.17e-04 [09/20 19:55:29] lb.utils.events INFO: eta: 17:13:41 iteration: 261999/375342 consumed_samples: 268288000 total_loss: 0.3746 time: 0.5437 s/iter data_time: 0.0585 s/iter total_throughput: 1883.44 samples/s lr: 2.17e-04 [09/20 19:56:24] lb.utils.events INFO: eta: 17:13:34 iteration: 262099/375342 consumed_samples: 268390400 total_loss: 0.3812 time: 0.5437 s/iter data_time: 0.0508 s/iter total_throughput: 1883.43 samples/s lr: 2.16e-04 [09/20 19:57:20] lb.utils.events INFO: eta: 17:13:15 iteration: 262199/375342 consumed_samples: 268492800 total_loss: 0.3827 time: 0.5437 s/iter data_time: 0.0536 s/iter total_throughput: 1883.42 samples/s lr: 2.16e-04 [09/20 19:58:15] lb.utils.events INFO: eta: 17:12:45 iteration: 262299/375342 consumed_samples: 268595200 total_loss: 0.3804 time: 0.5437 s/iter data_time: 0.0497 s/iter total_throughput: 1883.41 samples/s lr: 2.16e-04 [09/20 19:59:10] lb.utils.events INFO: eta: 17:13:01 iteration: 262399/375342 consumed_samples: 268697600 total_loss: 0.3767 time: 0.5437 s/iter data_time: 0.0551 s/iter total_throughput: 1883.40 samples/s lr: 2.15e-04 [09/20 20:00:05] lb.utils.events INFO: eta: 17:12:42 iteration: 262499/375342 consumed_samples: 268800000 total_loss: 0.3752 time: 0.5437 s/iter data_time: 0.0525 s/iter total_throughput: 1883.40 samples/s lr: 2.15e-04 [09/20 20:01:00] lb.utils.events INFO: eta: 17:11:50 iteration: 262599/375342 consumed_samples: 268902400 total_loss: 0.3755 time: 0.5437 s/iter data_time: 0.0507 s/iter total_throughput: 1883.39 samples/s lr: 2.15e-04 [09/20 20:01:54] lb.utils.events INFO: eta: 17:11:20 iteration: 262699/375342 consumed_samples: 269004800 total_loss: 0.3783 time: 0.5437 s/iter data_time: 0.0577 s/iter total_throughput: 1883.38 samples/s lr: 2.14e-04 [09/20 20:02:49] lb.utils.events INFO: eta: 17:10:19 iteration: 262799/375342 consumed_samples: 269107200 total_loss: 0.3794 time: 0.5437 s/iter data_time: 0.0525 s/iter total_throughput: 1883.37 samples/s lr: 2.14e-04 [09/20 20:03:44] lb.utils.events INFO: eta: 17:09:10 iteration: 262899/375342 consumed_samples: 269209600 total_loss: 0.3815 time: 0.5437 s/iter data_time: 0.0547 s/iter total_throughput: 1883.37 samples/s lr: 2.14e-04 [09/20 20:04:39] lb.utils.events INFO: eta: 17:08:15 iteration: 262999/375342 consumed_samples: 269312000 total_loss: 0.3818 time: 0.5437 s/iter data_time: 0.0494 s/iter total_throughput: 1883.36 samples/s lr: 2.13e-04 [09/20 20:05:34] lb.utils.events INFO: eta: 17:06:55 iteration: 263099/375342 consumed_samples: 269414400 total_loss: 0.3812 time: 0.5437 s/iter data_time: 0.0553 s/iter total_throughput: 1883.36 samples/s lr: 2.13e-04 [09/20 20:06:29] lb.utils.events INFO: eta: 17:05:44 iteration: 263199/375342 consumed_samples: 269516800 total_loss: 0.3851 time: 0.5437 s/iter data_time: 0.0518 s/iter total_throughput: 1883.35 samples/s lr: 2.13e-04 [09/20 20:07:24] lb.utils.events INFO: eta: 17:04:50 iteration: 263299/375342 consumed_samples: 269619200 total_loss: 0.3838 time: 0.5437 s/iter data_time: 0.0520 s/iter total_throughput: 1883.34 samples/s lr: 2.12e-04 [09/20 20:08:19] lb.utils.events INFO: eta: 17:03:41 iteration: 263399/375342 consumed_samples: 269721600 total_loss: 0.3722 time: 0.5437 s/iter data_time: 0.0500 s/iter total_throughput: 1883.34 samples/s lr: 2.12e-04 [09/20 20:09:13] lb.utils.events INFO: eta: 17:02:41 iteration: 263499/375342 consumed_samples: 269824000 total_loss: 0.374 time: 0.5437 s/iter data_time: 0.0524 s/iter total_throughput: 1883.33 samples/s lr: 2.12e-04 [09/20 20:10:08] lb.utils.events INFO: eta: 17:01:29 iteration: 263599/375342 consumed_samples: 269926400 total_loss: 0.3793 time: 0.5437 s/iter data_time: 0.0519 s/iter total_throughput: 1883.33 samples/s lr: 2.11e-04 [09/20 20:11:03] lb.utils.events INFO: eta: 17:00:32 iteration: 263699/375342 consumed_samples: 270028800 total_loss: 0.3767 time: 0.5437 s/iter data_time: 0.0536 s/iter total_throughput: 1883.32 samples/s lr: 2.11e-04 [09/20 20:11:58] lb.utils.events INFO: eta: 16:59:32 iteration: 263799/375342 consumed_samples: 270131200 total_loss: 0.3716 time: 0.5437 s/iter data_time: 0.0491 s/iter total_throughput: 1883.32 samples/s lr: 2.11e-04 [09/20 20:12:53] lb.utils.events INFO: eta: 16:58:37 iteration: 263899/375342 consumed_samples: 270233600 total_loss: 0.3767 time: 0.5437 s/iter data_time: 0.0542 s/iter total_throughput: 1883.31 samples/s lr: 2.10e-04 [09/20 20:13:48] lb.utils.events INFO: eta: 16:57:18 iteration: 263999/375342 consumed_samples: 270336000 total_loss: 0.3799 time: 0.5437 s/iter data_time: 0.0539 s/iter total_throughput: 1883.31 samples/s lr: 2.10e-04 [09/20 20:14:42] lb.utils.events INFO: eta: 16:56:26 iteration: 264099/375342 consumed_samples: 270438400 total_loss: 0.3815 time: 0.5437 s/iter data_time: 0.0538 s/iter total_throughput: 1883.30 samples/s lr: 2.10e-04 [09/20 20:15:37] lb.utils.events INFO: eta: 16:55:48 iteration: 264199/375342 consumed_samples: 270540800 total_loss: 0.3814 time: 0.5437 s/iter data_time: 0.0573 s/iter total_throughput: 1883.29 samples/s lr: 2.09e-04 [09/20 20:16:32] lb.utils.events INFO: eta: 16:54:38 iteration: 264299/375342 consumed_samples: 270643200 total_loss: 0.3789 time: 0.5437 s/iter data_time: 0.0561 s/iter total_throughput: 1883.29 samples/s lr: 2.09e-04 [09/20 20:17:27] lb.utils.events INFO: eta: 16:54:03 iteration: 264399/375342 consumed_samples: 270745600 total_loss: 0.3777 time: 0.5437 s/iter data_time: 0.0513 s/iter total_throughput: 1883.28 samples/s lr: 2.09e-04 [09/20 20:18:22] lb.utils.events INFO: eta: 16:53:23 iteration: 264499/375342 consumed_samples: 270848000 total_loss: 0.3776 time: 0.5437 s/iter data_time: 0.0524 s/iter total_throughput: 1883.27 samples/s lr: 2.08e-04 [09/20 20:19:17] lb.utils.events INFO: eta: 16:53:17 iteration: 264599/375342 consumed_samples: 270950400 total_loss: 0.3753 time: 0.5437 s/iter data_time: 0.0514 s/iter total_throughput: 1883.26 samples/s lr: 2.08e-04 [09/20 20:20:12] lb.utils.events INFO: eta: 16:52:41 iteration: 264699/375342 consumed_samples: 271052800 total_loss: 0.3708 time: 0.5437 s/iter data_time: 0.0498 s/iter total_throughput: 1883.26 samples/s lr: 2.08e-04 [09/20 20:21:07] lb.utils.events INFO: eta: 16:52:12 iteration: 264799/375342 consumed_samples: 271155200 total_loss: 0.3793 time: 0.5437 s/iter data_time: 0.0518 s/iter total_throughput: 1883.25 samples/s lr: 2.07e-04 [09/20 20:22:02] lb.utils.events INFO: eta: 16:51:23 iteration: 264899/375342 consumed_samples: 271257600 total_loss: 0.3817 time: 0.5437 s/iter data_time: 0.0531 s/iter total_throughput: 1883.24 samples/s lr: 2.07e-04 [09/20 20:22:57] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0264999 [09/20 20:22:58] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 20:22:58] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 20:23:02] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0481 s/iter. Inference: 0.2544 s/iter. Eval: 0.0034 s/iter. Total: 0.3059 s/iter. ETA=0:00:11 [09/20 20:23:08] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0619 s/iter. Inference: 0.2700 s/iter. Eval: 0.0032 s/iter. Total: 0.3353 s/iter. ETA=0:00:07 [09/20 20:23:13] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0609 s/iter. Inference: 0.2661 s/iter. Eval: 0.0032 s/iter. Total: 0.3305 s/iter. ETA=0:00:01 [09/20 20:23:15] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 20:23:15] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.255894 (0.000285 s / iter per device, on 8 devices) [09/20 20:23:15] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000231 s / iter per device, on 8 devices) [09/20 20:23:15] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 20:23:15] lb.evaluation.utils INFO: copypaste: Acc@1=77.998 [09/20 20:23:15] lb.evaluation.utils INFO: copypaste: Acc@5=93.948 [09/20 20:23:15] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 77.99800, better than last best score 77.69400 @ iteration 254999. [09/20 20:23:15] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 20:23:15] lb.utils.events INFO: eta: 16:50:32 iteration: 264999/375342 consumed_samples: 271360000 total_loss: 0.3767 time: 0.5437 s/iter data_time: 0.0524 s/iter total_throughput: 1883.23 samples/s lr: 2.07e-04 [09/20 20:24:10] lb.utils.events INFO: eta: 16:49:48 iteration: 265099/375342 consumed_samples: 271462400 total_loss: 0.3751 time: 0.5437 s/iter data_time: 0.0503 s/iter total_throughput: 1883.22 samples/s lr: 2.06e-04 [09/20 20:25:06] lb.utils.events INFO: eta: 16:49:14 iteration: 265199/375342 consumed_samples: 271564800 total_loss: 0.3755 time: 0.5438 s/iter data_time: 0.0522 s/iter total_throughput: 1883.21 samples/s lr: 2.06e-04 [09/20 20:26:01] lb.utils.events INFO: eta: 16:48:35 iteration: 265299/375342 consumed_samples: 271667200 total_loss: 0.374 time: 0.5438 s/iter data_time: 0.0514 s/iter total_throughput: 1883.21 samples/s lr: 2.06e-04 [09/20 20:26:56] lb.utils.events INFO: eta: 16:47:45 iteration: 265399/375342 consumed_samples: 271769600 total_loss: 0.3719 time: 0.5438 s/iter data_time: 0.0545 s/iter total_throughput: 1883.20 samples/s lr: 2.05e-04 [09/20 20:27:51] lb.utils.events INFO: eta: 16:46:56 iteration: 265499/375342 consumed_samples: 271872000 total_loss: 0.38 time: 0.5438 s/iter data_time: 0.0521 s/iter total_throughput: 1883.19 samples/s lr: 2.05e-04 [09/20 20:28:47] lb.utils.events INFO: eta: 16:46:23 iteration: 265599/375342 consumed_samples: 271974400 total_loss: 0.3836 time: 0.5438 s/iter data_time: 0.0514 s/iter total_throughput: 1883.17 samples/s lr: 2.05e-04 [09/20 20:29:41] lb.utils.events INFO: eta: 16:45:03 iteration: 265699/375342 consumed_samples: 272076800 total_loss: 0.3721 time: 0.5438 s/iter data_time: 0.0542 s/iter total_throughput: 1883.16 samples/s lr: 2.04e-04 [09/20 20:30:36] lb.utils.events INFO: eta: 16:43:38 iteration: 265799/375342 consumed_samples: 272179200 total_loss: 0.3719 time: 0.5438 s/iter data_time: 0.0526 s/iter total_throughput: 1883.16 samples/s lr: 2.04e-04 [09/20 20:31:31] lb.utils.events INFO: eta: 16:42:28 iteration: 265899/375342 consumed_samples: 272281600 total_loss: 0.3781 time: 0.5438 s/iter data_time: 0.0500 s/iter total_throughput: 1883.15 samples/s lr: 2.04e-04 [09/20 20:32:26] lb.utils.events INFO: eta: 16:41:33 iteration: 265999/375342 consumed_samples: 272384000 total_loss: 0.3729 time: 0.5438 s/iter data_time: 0.0487 s/iter total_throughput: 1883.15 samples/s lr: 2.03e-04 [09/20 20:33:21] lb.utils.events INFO: eta: 16:40:43 iteration: 266099/375342 consumed_samples: 272486400 total_loss: 0.3662 time: 0.5438 s/iter data_time: 0.0500 s/iter total_throughput: 1883.14 samples/s lr: 2.03e-04 [09/20 20:34:16] lb.utils.events INFO: eta: 16:39:27 iteration: 266199/375342 consumed_samples: 272588800 total_loss: 0.3731 time: 0.5438 s/iter data_time: 0.0499 s/iter total_throughput: 1883.13 samples/s lr: 2.03e-04 [09/20 20:35:11] lb.utils.events INFO: eta: 16:38:11 iteration: 266299/375342 consumed_samples: 272691200 total_loss: 0.3727 time: 0.5438 s/iter data_time: 0.0538 s/iter total_throughput: 1883.13 samples/s lr: 2.02e-04 [09/20 20:36:06] lb.utils.events INFO: eta: 16:37:02 iteration: 266399/375342 consumed_samples: 272793600 total_loss: 0.3729 time: 0.5438 s/iter data_time: 0.0541 s/iter total_throughput: 1883.12 samples/s lr: 2.02e-04 [09/20 20:37:01] lb.utils.events INFO: eta: 16:35:43 iteration: 266499/375342 consumed_samples: 272896000 total_loss: 0.3792 time: 0.5438 s/iter data_time: 0.0553 s/iter total_throughput: 1883.11 samples/s lr: 2.02e-04 [09/20 20:37:56] lb.utils.events INFO: eta: 16:34:24 iteration: 266599/375342 consumed_samples: 272998400 total_loss: 0.3804 time: 0.5438 s/iter data_time: 0.0565 s/iter total_throughput: 1883.11 samples/s lr: 2.01e-04 [09/20 20:38:51] lb.utils.events INFO: eta: 16:34:09 iteration: 266699/375342 consumed_samples: 273100800 total_loss: 0.3797 time: 0.5438 s/iter data_time: 0.0537 s/iter total_throughput: 1883.10 samples/s lr: 2.01e-04 [09/20 20:39:45] lb.utils.events INFO: eta: 16:33:21 iteration: 266799/375342 consumed_samples: 273203200 total_loss: 0.3747 time: 0.5438 s/iter data_time: 0.0528 s/iter total_throughput: 1883.09 samples/s lr: 2.01e-04 [09/20 20:40:41] lb.utils.events INFO: eta: 16:32:46 iteration: 266899/375342 consumed_samples: 273305600 total_loss: 0.3735 time: 0.5438 s/iter data_time: 0.0555 s/iter total_throughput: 1883.09 samples/s lr: 2.00e-04 [09/20 20:41:35] lb.utils.events INFO: eta: 16:32:07 iteration: 266999/375342 consumed_samples: 273408000 total_loss: 0.381 time: 0.5438 s/iter data_time: 0.0553 s/iter total_throughput: 1883.08 samples/s lr: 2.00e-04 [09/20 20:42:30] lb.utils.events INFO: eta: 16:31:10 iteration: 267099/375342 consumed_samples: 273510400 total_loss: 0.3781 time: 0.5438 s/iter data_time: 0.0556 s/iter total_throughput: 1883.07 samples/s lr: 2.00e-04 [09/20 20:43:25] lb.utils.events INFO: eta: 16:30:34 iteration: 267199/375342 consumed_samples: 273612800 total_loss: 0.3711 time: 0.5438 s/iter data_time: 0.0516 s/iter total_throughput: 1883.06 samples/s lr: 1.99e-04 [09/20 20:44:21] lb.utils.events INFO: eta: 16:29:56 iteration: 267299/375342 consumed_samples: 273715200 total_loss: 0.3767 time: 0.5438 s/iter data_time: 0.0557 s/iter total_throughput: 1883.05 samples/s lr: 1.99e-04 [09/20 20:45:15] lb.utils.events INFO: eta: 16:29:01 iteration: 267399/375342 consumed_samples: 273817600 total_loss: 0.3758 time: 0.5438 s/iter data_time: 0.0595 s/iter total_throughput: 1883.05 samples/s lr: 1.99e-04 [09/20 20:46:10] lb.utils.events INFO: eta: 16:28:36 iteration: 267499/375342 consumed_samples: 273920000 total_loss: 0.3775 time: 0.5438 s/iter data_time: 0.0540 s/iter total_throughput: 1883.04 samples/s lr: 1.98e-04 [09/20 20:47:05] lb.utils.events INFO: eta: 16:27:49 iteration: 267599/375342 consumed_samples: 274022400 total_loss: 0.3732 time: 0.5438 s/iter data_time: 0.0516 s/iter total_throughput: 1883.03 samples/s lr: 1.98e-04 [09/20 20:48:01] lb.utils.events INFO: eta: 16:27:10 iteration: 267699/375342 consumed_samples: 274124800 total_loss: 0.3729 time: 0.5438 s/iter data_time: 0.0534 s/iter total_throughput: 1883.02 samples/s lr: 1.98e-04 [09/20 20:48:56] lb.utils.events INFO: eta: 16:26:31 iteration: 267799/375342 consumed_samples: 274227200 total_loss: 0.3716 time: 0.5438 s/iter data_time: 0.0507 s/iter total_throughput: 1883.01 samples/s lr: 1.97e-04 [09/20 20:49:51] lb.utils.events INFO: eta: 16:26:18 iteration: 267899/375342 consumed_samples: 274329600 total_loss: 0.3716 time: 0.5438 s/iter data_time: 0.0560 s/iter total_throughput: 1883.00 samples/s lr: 1.97e-04 [09/20 20:50:47] lb.utils.events INFO: eta: 16:26:00 iteration: 267999/375342 consumed_samples: 274432000 total_loss: 0.3756 time: 0.5438 s/iter data_time: 0.0524 s/iter total_throughput: 1882.99 samples/s lr: 1.97e-04 [09/20 20:51:42] lb.utils.events INFO: eta: 16:25:32 iteration: 268099/375342 consumed_samples: 274534400 total_loss: 0.3758 time: 0.5438 s/iter data_time: 0.0510 s/iter total_throughput: 1882.97 samples/s lr: 1.96e-04 [09/20 20:52:38] lb.utils.events INFO: eta: 16:25:13 iteration: 268199/375342 consumed_samples: 274636800 total_loss: 0.3713 time: 0.5438 s/iter data_time: 0.0508 s/iter total_throughput: 1882.96 samples/s lr: 1.96e-04 [09/20 20:53:33] lb.utils.events INFO: eta: 16:25:07 iteration: 268299/375342 consumed_samples: 274739200 total_loss: 0.3733 time: 0.5438 s/iter data_time: 0.0550 s/iter total_throughput: 1882.94 samples/s lr: 1.96e-04 [09/20 20:54:29] lb.utils.events INFO: eta: 16:25:24 iteration: 268399/375342 consumed_samples: 274841600 total_loss: 0.3749 time: 0.5438 s/iter data_time: 0.0553 s/iter total_throughput: 1882.93 samples/s lr: 1.95e-04 [09/20 20:55:24] lb.utils.events INFO: eta: 16:25:53 iteration: 268499/375342 consumed_samples: 274944000 total_loss: 0.3751 time: 0.5438 s/iter data_time: 0.0539 s/iter total_throughput: 1882.91 samples/s lr: 1.95e-04 [09/20 20:56:20] lb.utils.events INFO: eta: 16:26:07 iteration: 268599/375342 consumed_samples: 275046400 total_loss: 0.3816 time: 0.5438 s/iter data_time: 0.0534 s/iter total_throughput: 1882.90 samples/s lr: 1.95e-04 [09/20 20:57:15] lb.utils.events INFO: eta: 16:25:41 iteration: 268699/375342 consumed_samples: 275148800 total_loss: 0.3877 time: 0.5438 s/iter data_time: 0.0514 s/iter total_throughput: 1882.89 samples/s lr: 1.94e-04 [09/20 20:58:11] lb.utils.events INFO: eta: 16:25:10 iteration: 268799/375342 consumed_samples: 275251200 total_loss: 0.3812 time: 0.5438 s/iter data_time: 0.0486 s/iter total_throughput: 1882.87 samples/s lr: 1.94e-04 [09/20 20:59:06] lb.utils.events INFO: eta: 16:24:38 iteration: 268899/375342 consumed_samples: 275353600 total_loss: 0.3716 time: 0.5439 s/iter data_time: 0.0560 s/iter total_throughput: 1882.86 samples/s lr: 1.94e-04 [09/20 21:00:02] lb.utils.events INFO: eta: 16:23:47 iteration: 268999/375342 consumed_samples: 275456000 total_loss: 0.3726 time: 0.5439 s/iter data_time: 0.0477 s/iter total_throughput: 1882.85 samples/s lr: 1.93e-04 [09/20 21:00:57] lb.utils.events INFO: eta: 16:22:42 iteration: 269099/375342 consumed_samples: 275558400 total_loss: 0.3769 time: 0.5439 s/iter data_time: 0.0548 s/iter total_throughput: 1882.83 samples/s lr: 1.93e-04 [09/20 21:01:52] lb.utils.events INFO: eta: 16:20:48 iteration: 269199/375342 consumed_samples: 275660800 total_loss: 0.3814 time: 0.5439 s/iter data_time: 0.0513 s/iter total_throughput: 1882.82 samples/s lr: 1.93e-04 [09/20 21:02:47] lb.utils.events INFO: eta: 16:18:17 iteration: 269299/375342 consumed_samples: 275763200 total_loss: 0.378 time: 0.5439 s/iter data_time: 0.0540 s/iter total_throughput: 1882.82 samples/s lr: 1.93e-04 [09/20 21:03:42] lb.utils.events INFO: eta: 16:15:43 iteration: 269399/375342 consumed_samples: 275865600 total_loss: 0.3771 time: 0.5439 s/iter data_time: 0.0552 s/iter total_throughput: 1882.81 samples/s lr: 1.92e-04 [09/20 21:04:37] lb.utils.events INFO: eta: 16:13:08 iteration: 269499/375342 consumed_samples: 275968000 total_loss: 0.3769 time: 0.5439 s/iter data_time: 0.0516 s/iter total_throughput: 1882.80 samples/s lr: 1.92e-04 [09/20 21:05:32] lb.utils.events INFO: eta: 16:10:57 iteration: 269599/375342 consumed_samples: 276070400 total_loss: 0.3805 time: 0.5439 s/iter data_time: 0.0546 s/iter total_throughput: 1882.80 samples/s lr: 1.92e-04 [09/20 21:06:27] lb.utils.events INFO: eta: 16:09:27 iteration: 269699/375342 consumed_samples: 276172800 total_loss: 0.3792 time: 0.5439 s/iter data_time: 0.0549 s/iter total_throughput: 1882.79 samples/s lr: 1.91e-04 [09/20 21:07:22] lb.utils.events INFO: eta: 16:07:49 iteration: 269799/375342 consumed_samples: 276275200 total_loss: 0.3751 time: 0.5439 s/iter data_time: 0.0566 s/iter total_throughput: 1882.78 samples/s lr: 1.91e-04 [09/20 21:08:17] lb.utils.events INFO: eta: 16:06:09 iteration: 269899/375342 consumed_samples: 276377600 total_loss: 0.3701 time: 0.5439 s/iter data_time: 0.0547 s/iter total_throughput: 1882.77 samples/s lr: 1.91e-04 [09/20 21:09:12] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0269999 [09/20 21:09:13] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 21:09:13] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 21:09:18] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0583 s/iter. Inference: 0.2476 s/iter. Eval: 0.0022 s/iter. Total: 0.3081 s/iter. ETA=0:00:11 [09/20 21:09:23] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0706 s/iter. Inference: 0.2609 s/iter. Eval: 0.0022 s/iter. Total: 0.3341 s/iter. ETA=0:00:07 [09/20 21:09:28] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0715 s/iter. Inference: 0.2558 s/iter. Eval: 0.0022 s/iter. Total: 0.3298 s/iter. ETA=0:00:01 [09/20 21:09:30] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 21:09:30] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.203085 (0.000284 s / iter per device, on 8 devices) [09/20 21:09:30] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/20 21:09:30] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 21:09:30] lb.evaluation.utils INFO: copypaste: Acc@1=78.084 [09/20 21:09:30] lb.evaluation.utils INFO: copypaste: Acc@5=93.95 [09/20 21:09:30] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 78.08400, better than last best score 77.99800 @ iteration 264999. [09/20 21:09:30] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 21:09:31] lb.utils.events INFO: eta: 16:04:29 iteration: 269999/375342 consumed_samples: 276480000 total_loss: 0.3698 time: 0.5439 s/iter data_time: 0.0508 s/iter total_throughput: 1882.76 samples/s lr: 1.90e-04 [09/20 21:10:26] lb.utils.events INFO: eta: 16:03:46 iteration: 270099/375342 consumed_samples: 276582400 total_loss: 0.3715 time: 0.5439 s/iter data_time: 0.0538 s/iter total_throughput: 1882.76 samples/s lr: 1.90e-04 [09/20 21:11:21] lb.utils.events INFO: eta: 16:03:12 iteration: 270199/375342 consumed_samples: 276684800 total_loss: 0.3762 time: 0.5439 s/iter data_time: 0.0522 s/iter total_throughput: 1882.75 samples/s lr: 1.90e-04 [09/20 21:12:16] lb.utils.events INFO: eta: 16:02:37 iteration: 270299/375342 consumed_samples: 276787200 total_loss: 0.3804 time: 0.5439 s/iter data_time: 0.0495 s/iter total_throughput: 1882.74 samples/s lr: 1.89e-04 [09/20 21:13:11] lb.utils.events INFO: eta: 16:02:01 iteration: 270399/375342 consumed_samples: 276889600 total_loss: 0.3714 time: 0.5439 s/iter data_time: 0.0526 s/iter total_throughput: 1882.73 samples/s lr: 1.89e-04 [09/20 21:14:06] lb.utils.events INFO: eta: 16:01:38 iteration: 270499/375342 consumed_samples: 276992000 total_loss: 0.3702 time: 0.5439 s/iter data_time: 0.0507 s/iter total_throughput: 1882.72 samples/s lr: 1.89e-04 [09/20 21:15:01] lb.utils.events INFO: eta: 16:01:26 iteration: 270599/375342 consumed_samples: 277094400 total_loss: 0.3765 time: 0.5439 s/iter data_time: 0.0576 s/iter total_throughput: 1882.71 samples/s lr: 1.88e-04 [09/20 21:15:56] lb.utils.events INFO: eta: 16:01:05 iteration: 270699/375342 consumed_samples: 277196800 total_loss: 0.3732 time: 0.5439 s/iter data_time: 0.0557 s/iter total_throughput: 1882.70 samples/s lr: 1.88e-04 [09/20 21:16:52] lb.utils.events INFO: eta: 16:00:31 iteration: 270799/375342 consumed_samples: 277299200 total_loss: 0.3727 time: 0.5439 s/iter data_time: 0.0462 s/iter total_throughput: 1882.69 samples/s lr: 1.88e-04 [09/20 21:17:47] lb.utils.events INFO: eta: 16:00:32 iteration: 270899/375342 consumed_samples: 277401600 total_loss: 0.3743 time: 0.5439 s/iter data_time: 0.0494 s/iter total_throughput: 1882.67 samples/s lr: 1.87e-04 [09/20 21:18:43] lb.utils.events INFO: eta: 16:00:40 iteration: 270999/375342 consumed_samples: 277504000 total_loss: 0.376 time: 0.5439 s/iter data_time: 0.0534 s/iter total_throughput: 1882.66 samples/s lr: 1.87e-04 [09/20 21:19:39] lb.utils.events INFO: eta: 16:00:25 iteration: 271099/375342 consumed_samples: 277606400 total_loss: 0.3757 time: 0.5439 s/iter data_time: 0.0535 s/iter total_throughput: 1882.65 samples/s lr: 1.87e-04 [09/20 21:20:34] lb.utils.events INFO: eta: 16:00:39 iteration: 271199/375342 consumed_samples: 277708800 total_loss: 0.3743 time: 0.5439 s/iter data_time: 0.0543 s/iter total_throughput: 1882.63 samples/s lr: 1.86e-04 [09/20 21:21:30] lb.utils.events INFO: eta: 16:00:31 iteration: 271299/375342 consumed_samples: 277811200 total_loss: 0.3741 time: 0.5439 s/iter data_time: 0.0520 s/iter total_throughput: 1882.62 samples/s lr: 1.86e-04 [09/20 21:22:25] lb.utils.events INFO: eta: 16:00:34 iteration: 271399/375342 consumed_samples: 277913600 total_loss: 0.3779 time: 0.5439 s/iter data_time: 0.0568 s/iter total_throughput: 1882.60 samples/s lr: 1.86e-04 [09/20 21:23:21] lb.utils.events INFO: eta: 16:00:24 iteration: 271499/375342 consumed_samples: 278016000 total_loss: 0.3779 time: 0.5439 s/iter data_time: 0.0497 s/iter total_throughput: 1882.59 samples/s lr: 1.85e-04 [09/20 21:24:16] lb.utils.events INFO: eta: 15:59:46 iteration: 271599/375342 consumed_samples: 278118400 total_loss: 0.3799 time: 0.5439 s/iter data_time: 0.0530 s/iter total_throughput: 1882.58 samples/s lr: 1.85e-04 [09/20 21:25:12] lb.utils.events INFO: eta: 15:59:14 iteration: 271699/375342 consumed_samples: 278220800 total_loss: 0.3705 time: 0.5439 s/iter data_time: 0.0514 s/iter total_throughput: 1882.56 samples/s lr: 1.85e-04 [09/20 21:26:07] lb.utils.events INFO: eta: 15:58:34 iteration: 271799/375342 consumed_samples: 278323200 total_loss: 0.3658 time: 0.5439 s/iter data_time: 0.0490 s/iter total_throughput: 1882.55 samples/s lr: 1.85e-04 [09/20 21:27:02] lb.utils.events INFO: eta: 15:57:08 iteration: 271899/375342 consumed_samples: 278425600 total_loss: 0.3725 time: 0.5439 s/iter data_time: 0.0541 s/iter total_throughput: 1882.54 samples/s lr: 1.84e-04 [09/20 21:27:58] lb.utils.events INFO: eta: 15:55:54 iteration: 271999/375342 consumed_samples: 278528000 total_loss: 0.3711 time: 0.5440 s/iter data_time: 0.0567 s/iter total_throughput: 1882.52 samples/s lr: 1.84e-04 [09/20 21:28:54] lb.utils.events INFO: eta: 15:54:54 iteration: 272099/375342 consumed_samples: 278630400 total_loss: 0.3744 time: 0.5440 s/iter data_time: 0.0550 s/iter total_throughput: 1882.51 samples/s lr: 1.84e-04 [09/20 21:29:49] lb.utils.events INFO: eta: 15:53:57 iteration: 272199/375342 consumed_samples: 278732800 total_loss: 0.3804 time: 0.5440 s/iter data_time: 0.0551 s/iter total_throughput: 1882.50 samples/s lr: 1.83e-04 [09/20 21:30:44] lb.utils.events INFO: eta: 15:52:16 iteration: 272299/375342 consumed_samples: 278835200 total_loss: 0.3804 time: 0.5440 s/iter data_time: 0.0553 s/iter total_throughput: 1882.49 samples/s lr: 1.83e-04 [09/20 21:31:40] lb.utils.events INFO: eta: 15:50:29 iteration: 272399/375342 consumed_samples: 278937600 total_loss: 0.3773 time: 0.5440 s/iter data_time: 0.0539 s/iter total_throughput: 1882.48 samples/s lr: 1.83e-04 [09/20 21:32:35] lb.utils.events INFO: eta: 15:48:58 iteration: 272499/375342 consumed_samples: 279040000 total_loss: 0.3724 time: 0.5440 s/iter data_time: 0.0578 s/iter total_throughput: 1882.46 samples/s lr: 1.82e-04 [09/20 21:33:30] lb.utils.events INFO: eta: 15:46:47 iteration: 272599/375342 consumed_samples: 279142400 total_loss: 0.3795 time: 0.5440 s/iter data_time: 0.0534 s/iter total_throughput: 1882.45 samples/s lr: 1.82e-04 [09/20 21:34:25] lb.utils.events INFO: eta: 15:44:59 iteration: 272699/375342 consumed_samples: 279244800 total_loss: 0.3799 time: 0.5440 s/iter data_time: 0.0550 s/iter total_throughput: 1882.45 samples/s lr: 1.82e-04 [09/20 21:35:20] lb.utils.events INFO: eta: 15:43:37 iteration: 272799/375342 consumed_samples: 279347200 total_loss: 0.3777 time: 0.5440 s/iter data_time: 0.0506 s/iter total_throughput: 1882.44 samples/s lr: 1.81e-04 [09/20 21:36:15] lb.utils.events INFO: eta: 15:42:09 iteration: 272899/375342 consumed_samples: 279449600 total_loss: 0.3776 time: 0.5440 s/iter data_time: 0.0531 s/iter total_throughput: 1882.43 samples/s lr: 1.81e-04 [09/20 21:37:10] lb.utils.events INFO: eta: 15:40:24 iteration: 272999/375342 consumed_samples: 279552000 total_loss: 0.3741 time: 0.5440 s/iter data_time: 0.0549 s/iter total_throughput: 1882.43 samples/s lr: 1.81e-04 [09/20 21:38:05] lb.utils.events INFO: eta: 15:38:31 iteration: 273099/375342 consumed_samples: 279654400 total_loss: 0.3754 time: 0.5440 s/iter data_time: 0.0509 s/iter total_throughput: 1882.42 samples/s lr: 1.80e-04 [09/20 21:39:00] lb.utils.events INFO: eta: 15:37:07 iteration: 273199/375342 consumed_samples: 279756800 total_loss: 0.3781 time: 0.5440 s/iter data_time: 0.0523 s/iter total_throughput: 1882.41 samples/s lr: 1.80e-04 [09/20 21:39:55] lb.utils.events INFO: eta: 15:36:00 iteration: 273299/375342 consumed_samples: 279859200 total_loss: 0.3776 time: 0.5440 s/iter data_time: 0.0533 s/iter total_throughput: 1882.40 samples/s lr: 1.80e-04 [09/20 21:40:51] lb.utils.events INFO: eta: 15:35:38 iteration: 273399/375342 consumed_samples: 279961600 total_loss: 0.3725 time: 0.5440 s/iter data_time: 0.0500 s/iter total_throughput: 1882.39 samples/s lr: 1.80e-04 [09/20 21:41:46] lb.utils.events INFO: eta: 15:34:45 iteration: 273499/375342 consumed_samples: 280064000 total_loss: 0.3748 time: 0.5440 s/iter data_time: 0.0510 s/iter total_throughput: 1882.38 samples/s lr: 1.79e-04 [09/20 21:42:41] lb.utils.events INFO: eta: 15:34:00 iteration: 273599/375342 consumed_samples: 280166400 total_loss: 0.3723 time: 0.5440 s/iter data_time: 0.0504 s/iter total_throughput: 1882.37 samples/s lr: 1.79e-04 [09/20 21:43:36] lb.utils.events INFO: eta: 15:33:35 iteration: 273699/375342 consumed_samples: 280268800 total_loss: 0.3673 time: 0.5440 s/iter data_time: 0.0537 s/iter total_throughput: 1882.36 samples/s lr: 1.79e-04 [09/20 21:44:31] lb.utils.events INFO: eta: 15:33:23 iteration: 273799/375342 consumed_samples: 280371200 total_loss: 0.3699 time: 0.5440 s/iter data_time: 0.0566 s/iter total_throughput: 1882.35 samples/s lr: 1.78e-04 [09/20 21:45:26] lb.utils.events INFO: eta: 15:32:34 iteration: 273899/375342 consumed_samples: 280473600 total_loss: 0.377 time: 0.5440 s/iter data_time: 0.0533 s/iter total_throughput: 1882.34 samples/s lr: 1.78e-04 [09/20 21:46:22] lb.utils.events INFO: eta: 15:32:16 iteration: 273999/375342 consumed_samples: 280576000 total_loss: 0.3756 time: 0.5440 s/iter data_time: 0.0465 s/iter total_throughput: 1882.33 samples/s lr: 1.78e-04 [09/20 21:47:17] lb.utils.events INFO: eta: 15:31:48 iteration: 274099/375342 consumed_samples: 280678400 total_loss: 0.371 time: 0.5440 s/iter data_time: 0.0460 s/iter total_throughput: 1882.32 samples/s lr: 1.77e-04 [09/20 21:48:13] lb.utils.events INFO: eta: 15:31:32 iteration: 274199/375342 consumed_samples: 280780800 total_loss: 0.3713 time: 0.5440 s/iter data_time: 0.0438 s/iter total_throughput: 1882.31 samples/s lr: 1.77e-04 [09/20 21:49:08] lb.utils.events INFO: eta: 15:31:02 iteration: 274299/375342 consumed_samples: 280883200 total_loss: 0.3792 time: 0.5440 s/iter data_time: 0.0509 s/iter total_throughput: 1882.29 samples/s lr: 1.77e-04 [09/20 21:50:04] lb.utils.events INFO: eta: 15:29:55 iteration: 274399/375342 consumed_samples: 280985600 total_loss: 0.3775 time: 0.5440 s/iter data_time: 0.0525 s/iter total_throughput: 1882.28 samples/s lr: 1.76e-04 [09/20 21:50:59] lb.utils.events INFO: eta: 15:29:34 iteration: 274499/375342 consumed_samples: 281088000 total_loss: 0.3711 time: 0.5440 s/iter data_time: 0.0531 s/iter total_throughput: 1882.27 samples/s lr: 1.76e-04 [09/20 21:51:55] lb.utils.events INFO: eta: 15:29:40 iteration: 274599/375342 consumed_samples: 281190400 total_loss: 0.3752 time: 0.5440 s/iter data_time: 0.0506 s/iter total_throughput: 1882.25 samples/s lr: 1.76e-04 [09/20 21:52:50] lb.utils.events INFO: eta: 15:29:33 iteration: 274699/375342 consumed_samples: 281292800 total_loss: 0.3777 time: 0.5440 s/iter data_time: 0.0498 s/iter total_throughput: 1882.24 samples/s lr: 1.75e-04 [09/20 21:53:46] lb.utils.events INFO: eta: 15:29:23 iteration: 274799/375342 consumed_samples: 281395200 total_loss: 0.3743 time: 0.5440 s/iter data_time: 0.0476 s/iter total_throughput: 1882.23 samples/s lr: 1.75e-04 [09/20 21:54:41] lb.utils.events INFO: eta: 15:28:51 iteration: 274899/375342 consumed_samples: 281497600 total_loss: 0.3742 time: 0.5440 s/iter data_time: 0.0465 s/iter total_throughput: 1882.21 samples/s lr: 1.75e-04 [09/20 21:55:37] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0274999 [09/20 21:55:37] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 21:55:37] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 21:55:42] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0626 s/iter. Inference: 0.2574 s/iter. Eval: 0.0028 s/iter. Total: 0.3228 s/iter. ETA=0:00:11 [09/20 21:55:47] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0732 s/iter. Inference: 0.2601 s/iter. Eval: 0.0029 s/iter. Total: 0.3371 s/iter. ETA=0:00:07 [09/20 21:55:52] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0711 s/iter. Inference: 0.2573 s/iter. Eval: 0.0030 s/iter. Total: 0.3324 s/iter. ETA=0:00:01 [09/20 21:55:54] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 21:55:54] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.231395 (0.000285 s / iter per device, on 8 devices) [09/20 21:55:54] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000225 s / iter per device, on 8 devices) [09/20 21:55:54] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 21:55:54] lb.evaluation.utils INFO: copypaste: Acc@1=78.396 [09/20 21:55:54] lb.evaluation.utils INFO: copypaste: Acc@5=94.07799999999999 [09/20 21:55:54] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 78.39600, better than last best score 78.08400 @ iteration 269999. [09/20 21:55:54] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 21:55:55] lb.utils.events INFO: eta: 15:27:56 iteration: 274999/375342 consumed_samples: 281600000 total_loss: 0.3804 time: 0.5440 s/iter data_time: 0.0533 s/iter total_throughput: 1882.20 samples/s lr: 1.75e-04 [09/20 21:56:51] lb.utils.events INFO: eta: 15:27:22 iteration: 275099/375342 consumed_samples: 281702400 total_loss: 0.3779 time: 0.5440 s/iter data_time: 0.0520 s/iter total_throughput: 1882.19 samples/s lr: 1.74e-04 [09/20 21:57:46] lb.utils.events INFO: eta: 15:26:33 iteration: 275199/375342 consumed_samples: 281804800 total_loss: 0.3693 time: 0.5441 s/iter data_time: 0.0480 s/iter total_throughput: 1882.17 samples/s lr: 1.74e-04 [09/20 21:58:42] lb.utils.events INFO: eta: 15:25:39 iteration: 275299/375342 consumed_samples: 281907200 total_loss: 0.3672 time: 0.5441 s/iter data_time: 0.0536 s/iter total_throughput: 1882.16 samples/s lr: 1.74e-04 [09/20 21:59:37] lb.utils.events INFO: eta: 15:24:34 iteration: 275399/375342 consumed_samples: 282009600 total_loss: 0.3663 time: 0.5441 s/iter data_time: 0.0547 s/iter total_throughput: 1882.15 samples/s lr: 1.73e-04 [09/20 22:00:32] lb.utils.events INFO: eta: 15:23:08 iteration: 275499/375342 consumed_samples: 282112000 total_loss: 0.3691 time: 0.5441 s/iter data_time: 0.0532 s/iter total_throughput: 1882.14 samples/s lr: 1.73e-04 [09/20 22:01:28] lb.utils.events INFO: eta: 15:21:15 iteration: 275599/375342 consumed_samples: 282214400 total_loss: 0.3724 time: 0.5441 s/iter data_time: 0.0560 s/iter total_throughput: 1882.13 samples/s lr: 1.73e-04 [09/20 22:02:23] lb.utils.events INFO: eta: 15:19:55 iteration: 275699/375342 consumed_samples: 282316800 total_loss: 0.37 time: 0.5441 s/iter data_time: 0.0538 s/iter total_throughput: 1882.12 samples/s lr: 1.72e-04 [09/20 22:03:18] lb.utils.events INFO: eta: 15:18:24 iteration: 275799/375342 consumed_samples: 282419200 total_loss: 0.3734 time: 0.5441 s/iter data_time: 0.0501 s/iter total_throughput: 1882.11 samples/s lr: 1.72e-04 [09/20 22:04:13] lb.utils.events INFO: eta: 15:17:08 iteration: 275899/375342 consumed_samples: 282521600 total_loss: 0.3734 time: 0.5441 s/iter data_time: 0.0483 s/iter total_throughput: 1882.10 samples/s lr: 1.72e-04 [09/20 22:05:09] lb.utils.events INFO: eta: 15:15:24 iteration: 275999/375342 consumed_samples: 282624000 total_loss: 0.3733 time: 0.5441 s/iter data_time: 0.0546 s/iter total_throughput: 1882.09 samples/s lr: 1.71e-04 [09/20 22:06:04] lb.utils.events INFO: eta: 15:13:09 iteration: 276099/375342 consumed_samples: 282726400 total_loss: 0.3727 time: 0.5441 s/iter data_time: 0.0526 s/iter total_throughput: 1882.08 samples/s lr: 1.71e-04 [09/20 22:06:59] lb.utils.events INFO: eta: 15:11:34 iteration: 276199/375342 consumed_samples: 282828800 total_loss: 0.3742 time: 0.5441 s/iter data_time: 0.0524 s/iter total_throughput: 1882.07 samples/s lr: 1.71e-04 [09/20 22:07:54] lb.utils.events INFO: eta: 15:10:17 iteration: 276299/375342 consumed_samples: 282931200 total_loss: 0.3758 time: 0.5441 s/iter data_time: 0.0560 s/iter total_throughput: 1882.07 samples/s lr: 1.71e-04 [09/20 22:08:48] lb.utils.events INFO: eta: 15:08:37 iteration: 276399/375342 consumed_samples: 283033600 total_loss: 0.3771 time: 0.5441 s/iter data_time: 0.0552 s/iter total_throughput: 1882.06 samples/s lr: 1.70e-04 [09/20 22:09:44] lb.utils.events INFO: eta: 15:07:21 iteration: 276499/375342 consumed_samples: 283136000 total_loss: 0.3778 time: 0.5441 s/iter data_time: 0.0539 s/iter total_throughput: 1882.05 samples/s lr: 1.70e-04 [09/20 22:10:39] lb.utils.events INFO: eta: 15:06:20 iteration: 276599/375342 consumed_samples: 283238400 total_loss: 0.3756 time: 0.5441 s/iter data_time: 0.0481 s/iter total_throughput: 1882.04 samples/s lr: 1.70e-04 [09/20 22:11:34] lb.utils.events INFO: eta: 15:05:31 iteration: 276699/375342 consumed_samples: 283340800 total_loss: 0.373 time: 0.5441 s/iter data_time: 0.0523 s/iter total_throughput: 1882.03 samples/s lr: 1.69e-04 [09/20 22:12:30] lb.utils.events INFO: eta: 15:05:07 iteration: 276799/375342 consumed_samples: 283443200 total_loss: 0.3726 time: 0.5441 s/iter data_time: 0.0503 s/iter total_throughput: 1882.02 samples/s lr: 1.69e-04 [09/20 22:13:25] lb.utils.events INFO: eta: 15:04:28 iteration: 276899/375342 consumed_samples: 283545600 total_loss: 0.3728 time: 0.5441 s/iter data_time: 0.0507 s/iter total_throughput: 1882.01 samples/s lr: 1.69e-04 [09/20 22:14:20] lb.utils.events INFO: eta: 15:03:47 iteration: 276999/375342 consumed_samples: 283648000 total_loss: 0.3697 time: 0.5441 s/iter data_time: 0.0519 s/iter total_throughput: 1882.00 samples/s lr: 1.68e-04 [09/20 22:15:15] lb.utils.events INFO: eta: 15:03:02 iteration: 277099/375342 consumed_samples: 283750400 total_loss: 0.3697 time: 0.5441 s/iter data_time: 0.0512 s/iter total_throughput: 1881.99 samples/s lr: 1.68e-04 [09/20 22:16:10] lb.utils.events INFO: eta: 15:02:09 iteration: 277199/375342 consumed_samples: 283852800 total_loss: 0.3709 time: 0.5441 s/iter data_time: 0.0528 s/iter total_throughput: 1881.98 samples/s lr: 1.68e-04 [09/20 22:17:05] lb.utils.events INFO: eta: 15:01:23 iteration: 277299/375342 consumed_samples: 283955200 total_loss: 0.3722 time: 0.5441 s/iter data_time: 0.0446 s/iter total_throughput: 1881.98 samples/s lr: 1.68e-04 [09/20 22:18:01] lb.utils.events INFO: eta: 15:01:08 iteration: 277399/375342 consumed_samples: 284057600 total_loss: 0.3738 time: 0.5441 s/iter data_time: 0.0471 s/iter total_throughput: 1881.97 samples/s lr: 1.67e-04 [09/20 22:18:56] lb.utils.events INFO: eta: 15:00:37 iteration: 277499/375342 consumed_samples: 284160000 total_loss: 0.3747 time: 0.5441 s/iter data_time: 0.0454 s/iter total_throughput: 1881.96 samples/s lr: 1.67e-04 [09/20 22:19:51] lb.utils.events INFO: eta: 15:00:00 iteration: 277599/375342 consumed_samples: 284262400 total_loss: 0.3711 time: 0.5441 s/iter data_time: 0.0505 s/iter total_throughput: 1881.94 samples/s lr: 1.67e-04 [09/20 22:20:47] lb.utils.events INFO: eta: 14:59:30 iteration: 277699/375342 consumed_samples: 284364800 total_loss: 0.3728 time: 0.5441 s/iter data_time: 0.0510 s/iter total_throughput: 1881.93 samples/s lr: 1.66e-04 [09/20 22:21:42] lb.utils.events INFO: eta: 14:58:24 iteration: 277799/375342 consumed_samples: 284467200 total_loss: 0.3733 time: 0.5441 s/iter data_time: 0.0557 s/iter total_throughput: 1881.92 samples/s lr: 1.66e-04 [09/20 22:22:38] lb.utils.events INFO: eta: 14:58:06 iteration: 277899/375342 consumed_samples: 284569600 total_loss: 0.3713 time: 0.5441 s/iter data_time: 0.0515 s/iter total_throughput: 1881.90 samples/s lr: 1.66e-04 [09/20 22:23:33] lb.utils.events INFO: eta: 14:57:33 iteration: 277999/375342 consumed_samples: 284672000 total_loss: 0.3717 time: 0.5441 s/iter data_time: 0.0483 s/iter total_throughput: 1881.89 samples/s lr: 1.65e-04 [09/20 22:24:29] lb.utils.events INFO: eta: 14:57:33 iteration: 278099/375342 consumed_samples: 284774400 total_loss: 0.3715 time: 0.5441 s/iter data_time: 0.0470 s/iter total_throughput: 1881.88 samples/s lr: 1.65e-04 [09/20 22:25:24] lb.utils.events INFO: eta: 14:57:24 iteration: 278199/375342 consumed_samples: 284876800 total_loss: 0.3728 time: 0.5441 s/iter data_time: 0.0473 s/iter total_throughput: 1881.87 samples/s lr: 1.65e-04 [09/20 22:26:20] lb.utils.events INFO: eta: 14:57:00 iteration: 278299/375342 consumed_samples: 284979200 total_loss: 0.3709 time: 0.5441 s/iter data_time: 0.0577 s/iter total_throughput: 1881.85 samples/s lr: 1.65e-04 [09/20 22:27:15] lb.utils.events INFO: eta: 14:55:49 iteration: 278399/375342 consumed_samples: 285081600 total_loss: 0.3665 time: 0.5441 s/iter data_time: 0.0517 s/iter total_throughput: 1881.84 samples/s lr: 1.64e-04 [09/20 22:28:10] lb.utils.events INFO: eta: 14:54:56 iteration: 278499/375342 consumed_samples: 285184000 total_loss: 0.3643 time: 0.5442 s/iter data_time: 0.0511 s/iter total_throughput: 1881.83 samples/s lr: 1.64e-04 [09/20 22:29:06] lb.utils.events INFO: eta: 14:53:43 iteration: 278599/375342 consumed_samples: 285286400 total_loss: 0.3681 time: 0.5442 s/iter data_time: 0.0518 s/iter total_throughput: 1881.82 samples/s lr: 1.64e-04 [09/20 22:30:01] lb.utils.events INFO: eta: 14:52:16 iteration: 278699/375342 consumed_samples: 285388800 total_loss: 0.3722 time: 0.5442 s/iter data_time: 0.0559 s/iter total_throughput: 1881.81 samples/s lr: 1.63e-04 [09/20 22:30:56] lb.utils.events INFO: eta: 14:51:00 iteration: 278799/375342 consumed_samples: 285491200 total_loss: 0.3714 time: 0.5442 s/iter data_time: 0.0574 s/iter total_throughput: 1881.80 samples/s lr: 1.63e-04 [09/20 22:31:51] lb.utils.events INFO: eta: 14:49:19 iteration: 278899/375342 consumed_samples: 285593600 total_loss: 0.3667 time: 0.5442 s/iter data_time: 0.0540 s/iter total_throughput: 1881.79 samples/s lr: 1.63e-04 [09/20 22:32:46] lb.utils.events INFO: eta: 14:47:42 iteration: 278999/375342 consumed_samples: 285696000 total_loss: 0.3652 time: 0.5442 s/iter data_time: 0.0535 s/iter total_throughput: 1881.79 samples/s lr: 1.62e-04 [09/20 22:33:41] lb.utils.events INFO: eta: 14:46:12 iteration: 279099/375342 consumed_samples: 285798400 total_loss: 0.367 time: 0.5442 s/iter data_time: 0.0438 s/iter total_throughput: 1881.78 samples/s lr: 1.62e-04 [09/20 22:34:37] lb.utils.events INFO: eta: 14:44:36 iteration: 279199/375342 consumed_samples: 285900800 total_loss: 0.3726 time: 0.5442 s/iter data_time: 0.0458 s/iter total_throughput: 1881.77 samples/s lr: 1.62e-04 [09/20 22:35:32] lb.utils.events INFO: eta: 14:43:10 iteration: 279299/375342 consumed_samples: 286003200 total_loss: 0.3684 time: 0.5442 s/iter data_time: 0.0496 s/iter total_throughput: 1881.76 samples/s lr: 1.62e-04 [09/20 22:36:28] lb.utils.events INFO: eta: 14:42:06 iteration: 279399/375342 consumed_samples: 286105600 total_loss: 0.3586 time: 0.5442 s/iter data_time: 0.0582 s/iter total_throughput: 1881.74 samples/s lr: 1.61e-04 [09/20 22:37:22] lb.utils.events INFO: eta: 14:40:27 iteration: 279499/375342 consumed_samples: 286208000 total_loss: 0.3637 time: 0.5442 s/iter data_time: 0.0543 s/iter total_throughput: 1881.74 samples/s lr: 1.61e-04 [09/20 22:38:17] lb.utils.events INFO: eta: 14:38:14 iteration: 279599/375342 consumed_samples: 286310400 total_loss: 0.3743 time: 0.5442 s/iter data_time: 0.0533 s/iter total_throughput: 1881.73 samples/s lr: 1.61e-04 [09/20 22:39:12] lb.utils.events INFO: eta: 14:36:52 iteration: 279699/375342 consumed_samples: 286412800 total_loss: 0.3729 time: 0.5442 s/iter data_time: 0.0559 s/iter total_throughput: 1881.73 samples/s lr: 1.60e-04 [09/20 22:40:07] lb.utils.events INFO: eta: 14:35:38 iteration: 279799/375342 consumed_samples: 286515200 total_loss: 0.3697 time: 0.5442 s/iter data_time: 0.0566 s/iter total_throughput: 1881.72 samples/s lr: 1.60e-04 [09/20 22:41:02] lb.utils.events INFO: eta: 14:34:52 iteration: 279899/375342 consumed_samples: 286617600 total_loss: 0.3657 time: 0.5442 s/iter data_time: 0.0574 s/iter total_throughput: 1881.71 samples/s lr: 1.60e-04 [09/20 22:41:57] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0279999 [09/20 22:41:58] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 22:41:58] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 22:42:03] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0598 s/iter. Inference: 0.2519 s/iter. Eval: 0.0025 s/iter. Total: 0.3142 s/iter. ETA=0:00:11 [09/20 22:42:08] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0634 s/iter. Inference: 0.2690 s/iter. Eval: 0.0023 s/iter. Total: 0.3352 s/iter. ETA=0:00:07 [09/20 22:42:13] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0701 s/iter. Inference: 0.2635 s/iter. Eval: 0.0023 s/iter. Total: 0.3363 s/iter. ETA=0:00:02 [09/20 22:42:15] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 22:42:15] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.421935 (0.000288 s / iter per device, on 8 devices) [09/20 22:42:15] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000229 s / iter per device, on 8 devices) [09/20 22:42:15] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 22:42:15] lb.evaluation.utils INFO: copypaste: Acc@1=78.446 [09/20 22:42:15] lb.evaluation.utils INFO: copypaste: Acc@5=94.234 [09/20 22:42:15] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 78.44600, better than last best score 78.39600 @ iteration 274999. [09/20 22:42:15] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 22:42:16] lb.utils.events INFO: eta: 14:34:12 iteration: 279999/375342 consumed_samples: 286720000 total_loss: 0.3682 time: 0.5442 s/iter data_time: 0.0502 s/iter total_throughput: 1881.70 samples/s lr: 1.59e-04 [09/20 22:43:12] lb.utils.events INFO: eta: 14:34:12 iteration: 280099/375342 consumed_samples: 286822400 total_loss: 0.3679 time: 0.5442 s/iter data_time: 0.0532 s/iter total_throughput: 1881.69 samples/s lr: 1.59e-04 [09/20 22:44:07] lb.utils.events INFO: eta: 14:33:56 iteration: 280199/375342 consumed_samples: 286924800 total_loss: 0.3713 time: 0.5442 s/iter data_time: 0.0526 s/iter total_throughput: 1881.68 samples/s lr: 1.59e-04 [09/20 22:45:03] lb.utils.events INFO: eta: 14:33:14 iteration: 280299/375342 consumed_samples: 287027200 total_loss: 0.3713 time: 0.5442 s/iter data_time: 0.0515 s/iter total_throughput: 1881.66 samples/s lr: 1.59e-04 [09/20 22:45:58] lb.utils.events INFO: eta: 14:32:13 iteration: 280399/375342 consumed_samples: 287129600 total_loss: 0.3691 time: 0.5442 s/iter data_time: 0.0485 s/iter total_throughput: 1881.66 samples/s lr: 1.58e-04 [09/20 22:46:53] lb.utils.events INFO: eta: 14:32:14 iteration: 280499/375342 consumed_samples: 287232000 total_loss: 0.3666 time: 0.5442 s/iter data_time: 0.0512 s/iter total_throughput: 1881.65 samples/s lr: 1.58e-04 [09/20 22:47:48] lb.utils.events INFO: eta: 14:32:08 iteration: 280599/375342 consumed_samples: 287334400 total_loss: 0.3648 time: 0.5442 s/iter data_time: 0.0551 s/iter total_throughput: 1881.64 samples/s lr: 1.58e-04 [09/20 22:48:43] lb.utils.events INFO: eta: 14:31:32 iteration: 280699/375342 consumed_samples: 287436800 total_loss: 0.3728 time: 0.5442 s/iter data_time: 0.0443 s/iter total_throughput: 1881.63 samples/s lr: 1.57e-04 [09/20 22:49:38] lb.utils.events INFO: eta: 14:30:53 iteration: 280799/375342 consumed_samples: 287539200 total_loss: 0.3825 time: 0.5442 s/iter data_time: 0.0480 s/iter total_throughput: 1881.62 samples/s lr: 1.57e-04 [09/20 22:50:34] lb.utils.events INFO: eta: 14:30:09 iteration: 280899/375342 consumed_samples: 287641600 total_loss: 0.3777 time: 0.5442 s/iter data_time: 0.0478 s/iter total_throughput: 1881.61 samples/s lr: 1.57e-04 [09/20 22:51:29] lb.utils.events INFO: eta: 14:29:26 iteration: 280999/375342 consumed_samples: 287744000 total_loss: 0.3676 time: 0.5442 s/iter data_time: 0.0471 s/iter total_throughput: 1881.60 samples/s lr: 1.56e-04 [09/20 22:52:25] lb.utils.events INFO: eta: 14:28:15 iteration: 281099/375342 consumed_samples: 287846400 total_loss: 0.3721 time: 0.5442 s/iter data_time: 0.0494 s/iter total_throughput: 1881.58 samples/s lr: 1.56e-04 [09/20 22:53:20] lb.utils.events INFO: eta: 14:27:17 iteration: 281199/375342 consumed_samples: 287948800 total_loss: 0.3715 time: 0.5442 s/iter data_time: 0.0476 s/iter total_throughput: 1881.57 samples/s lr: 1.56e-04 [09/20 22:54:16] lb.utils.events INFO: eta: 14:26:46 iteration: 281299/375342 consumed_samples: 288051200 total_loss: 0.3674 time: 0.5442 s/iter data_time: 0.0484 s/iter total_throughput: 1881.56 samples/s lr: 1.56e-04 [09/20 22:55:11] lb.utils.events INFO: eta: 14:26:18 iteration: 281399/375342 consumed_samples: 288153600 total_loss: 0.369 time: 0.5442 s/iter data_time: 0.0517 s/iter total_throughput: 1881.55 samples/s lr: 1.55e-04 [09/20 22:56:07] lb.utils.events INFO: eta: 14:26:05 iteration: 281499/375342 consumed_samples: 288256000 total_loss: 0.3697 time: 0.5442 s/iter data_time: 0.0507 s/iter total_throughput: 1881.53 samples/s lr: 1.55e-04 [09/20 22:57:02] lb.utils.events INFO: eta: 14:25:40 iteration: 281599/375342 consumed_samples: 288358400 total_loss: 0.3714 time: 0.5442 s/iter data_time: 0.0534 s/iter total_throughput: 1881.52 samples/s lr: 1.55e-04 [09/20 22:57:58] lb.utils.events INFO: eta: 14:25:20 iteration: 281699/375342 consumed_samples: 288460800 total_loss: 0.371 time: 0.5442 s/iter data_time: 0.0521 s/iter total_throughput: 1881.51 samples/s lr: 1.54e-04 [09/20 22:58:53] lb.utils.events INFO: eta: 14:24:26 iteration: 281799/375342 consumed_samples: 288563200 total_loss: 0.3736 time: 0.5442 s/iter data_time: 0.0538 s/iter total_throughput: 1881.50 samples/s lr: 1.54e-04 [09/20 22:59:48] lb.utils.events INFO: eta: 14:23:18 iteration: 281899/375342 consumed_samples: 288665600 total_loss: 0.3748 time: 0.5443 s/iter data_time: 0.0511 s/iter total_throughput: 1881.49 samples/s lr: 1.54e-04 [09/20 23:00:43] lb.utils.events INFO: eta: 14:21:47 iteration: 281999/375342 consumed_samples: 288768000 total_loss: 0.37 time: 0.5443 s/iter data_time: 0.0555 s/iter total_throughput: 1881.48 samples/s lr: 1.54e-04 [09/20 23:01:38] lb.utils.events INFO: eta: 14:19:56 iteration: 282099/375342 consumed_samples: 288870400 total_loss: 0.3678 time: 0.5443 s/iter data_time: 0.0539 s/iter total_throughput: 1881.47 samples/s lr: 1.53e-04 [09/20 23:02:33] lb.utils.events INFO: eta: 14:18:06 iteration: 282199/375342 consumed_samples: 288972800 total_loss: 0.3708 time: 0.5443 s/iter data_time: 0.0537 s/iter total_throughput: 1881.47 samples/s lr: 1.53e-04 [09/20 23:03:28] lb.utils.events INFO: eta: 14:16:03 iteration: 282299/375342 consumed_samples: 289075200 total_loss: 0.373 time: 0.5443 s/iter data_time: 0.0516 s/iter total_throughput: 1881.46 samples/s lr: 1.53e-04 [09/20 23:04:23] lb.utils.events INFO: eta: 14:14:05 iteration: 282399/375342 consumed_samples: 289177600 total_loss: 0.375 time: 0.5443 s/iter data_time: 0.0535 s/iter total_throughput: 1881.46 samples/s lr: 1.52e-04 [09/20 23:05:18] lb.utils.events INFO: eta: 14:12:12 iteration: 282499/375342 consumed_samples: 289280000 total_loss: 0.3735 time: 0.5443 s/iter data_time: 0.0505 s/iter total_throughput: 1881.45 samples/s lr: 1.52e-04 [09/20 23:06:13] lb.utils.events INFO: eta: 14:11:00 iteration: 282599/375342 consumed_samples: 289382400 total_loss: 0.3699 time: 0.5443 s/iter data_time: 0.0528 s/iter total_throughput: 1881.44 samples/s lr: 1.52e-04 [09/20 23:07:08] lb.utils.events INFO: eta: 14:09:19 iteration: 282699/375342 consumed_samples: 289484800 total_loss: 0.3627 time: 0.5443 s/iter data_time: 0.0523 s/iter total_throughput: 1881.43 samples/s lr: 1.52e-04 [09/20 23:08:03] lb.utils.events INFO: eta: 14:07:50 iteration: 282799/375342 consumed_samples: 289587200 total_loss: 0.3713 time: 0.5443 s/iter data_time: 0.0541 s/iter total_throughput: 1881.43 samples/s lr: 1.51e-04 [09/20 23:08:59] lb.utils.events INFO: eta: 14:06:22 iteration: 282899/375342 consumed_samples: 289689600 total_loss: 0.3763 time: 0.5443 s/iter data_time: 0.0525 s/iter total_throughput: 1881.41 samples/s lr: 1.51e-04 [09/20 23:09:54] lb.utils.events INFO: eta: 14:05:00 iteration: 282999/375342 consumed_samples: 289792000 total_loss: 0.3682 time: 0.5443 s/iter data_time: 0.0548 s/iter total_throughput: 1881.41 samples/s lr: 1.51e-04 [09/20 23:10:49] lb.utils.events INFO: eta: 14:04:00 iteration: 283099/375342 consumed_samples: 289894400 total_loss: 0.3706 time: 0.5443 s/iter data_time: 0.0558 s/iter total_throughput: 1881.40 samples/s lr: 1.50e-04 [09/20 23:11:44] lb.utils.events INFO: eta: 14:03:16 iteration: 283199/375342 consumed_samples: 289996800 total_loss: 0.3621 time: 0.5443 s/iter data_time: 0.0558 s/iter total_throughput: 1881.40 samples/s lr: 1.50e-04 [09/20 23:12:39] lb.utils.events INFO: eta: 14:03:01 iteration: 283299/375342 consumed_samples: 290099200 total_loss: 0.3604 time: 0.5443 s/iter data_time: 0.0560 s/iter total_throughput: 1881.39 samples/s lr: 1.50e-04 [09/20 23:13:34] lb.utils.events INFO: eta: 14:02:38 iteration: 283399/375342 consumed_samples: 290201600 total_loss: 0.3688 time: 0.5443 s/iter data_time: 0.0577 s/iter total_throughput: 1881.38 samples/s lr: 1.49e-04 [09/20 23:14:29] lb.utils.events INFO: eta: 14:02:22 iteration: 283499/375342 consumed_samples: 290304000 total_loss: 0.3733 time: 0.5443 s/iter data_time: 0.0503 s/iter total_throughput: 1881.37 samples/s lr: 1.49e-04 [09/20 23:15:25] lb.utils.events INFO: eta: 14:01:33 iteration: 283599/375342 consumed_samples: 290406400 total_loss: 0.3703 time: 0.5443 s/iter data_time: 0.0512 s/iter total_throughput: 1881.36 samples/s lr: 1.49e-04 [09/20 23:16:20] lb.utils.events INFO: eta: 14:00:38 iteration: 283699/375342 consumed_samples: 290508800 total_loss: 0.366 time: 0.5443 s/iter data_time: 0.0516 s/iter total_throughput: 1881.35 samples/s lr: 1.49e-04 [09/20 23:17:15] lb.utils.events INFO: eta: 14:00:04 iteration: 283799/375342 consumed_samples: 290611200 total_loss: 0.3646 time: 0.5443 s/iter data_time: 0.0536 s/iter total_throughput: 1881.34 samples/s lr: 1.48e-04 [09/20 23:18:10] lb.utils.events INFO: eta: 13:59:20 iteration: 283899/375342 consumed_samples: 290713600 total_loss: 0.3652 time: 0.5443 s/iter data_time: 0.0497 s/iter total_throughput: 1881.34 samples/s lr: 1.48e-04 [09/20 23:19:05] lb.utils.events INFO: eta: 13:58:38 iteration: 283999/375342 consumed_samples: 290816000 total_loss: 0.3675 time: 0.5443 s/iter data_time: 0.0536 s/iter total_throughput: 1881.33 samples/s lr: 1.48e-04 [09/20 23:20:00] lb.utils.events INFO: eta: 13:58:20 iteration: 284099/375342 consumed_samples: 290918400 total_loss: 0.3703 time: 0.5443 s/iter data_time: 0.0407 s/iter total_throughput: 1881.32 samples/s lr: 1.47e-04 [09/20 23:20:55] lb.utils.events INFO: eta: 13:57:50 iteration: 284199/375342 consumed_samples: 291020800 total_loss: 0.3687 time: 0.5443 s/iter data_time: 0.0448 s/iter total_throughput: 1881.31 samples/s lr: 1.47e-04 [09/20 23:21:50] lb.utils.events INFO: eta: 13:57:20 iteration: 284299/375342 consumed_samples: 291123200 total_loss: 0.363 time: 0.5443 s/iter data_time: 0.0499 s/iter total_throughput: 1881.30 samples/s lr: 1.47e-04 [09/20 23:22:46] lb.utils.events INFO: eta: 13:56:48 iteration: 284399/375342 consumed_samples: 291225600 total_loss: 0.3665 time: 0.5443 s/iter data_time: 0.0534 s/iter total_throughput: 1881.29 samples/s lr: 1.47e-04 [09/20 23:23:41] lb.utils.events INFO: eta: 13:56:16 iteration: 284499/375342 consumed_samples: 291328000 total_loss: 0.369 time: 0.5443 s/iter data_time: 0.0524 s/iter total_throughput: 1881.28 samples/s lr: 1.46e-04 [09/20 23:24:37] lb.utils.events INFO: eta: 13:55:18 iteration: 284599/375342 consumed_samples: 291430400 total_loss: 0.3669 time: 0.5443 s/iter data_time: 0.0464 s/iter total_throughput: 1881.27 samples/s lr: 1.46e-04 [09/20 23:25:32] lb.utils.events INFO: eta: 13:54:27 iteration: 284699/375342 consumed_samples: 291532800 total_loss: 0.3673 time: 0.5443 s/iter data_time: 0.0541 s/iter total_throughput: 1881.26 samples/s lr: 1.46e-04 [09/20 23:26:28] lb.utils.events INFO: eta: 13:53:54 iteration: 284799/375342 consumed_samples: 291635200 total_loss: 0.3682 time: 0.5443 s/iter data_time: 0.0555 s/iter total_throughput: 1881.25 samples/s lr: 1.45e-04 [09/20 23:27:23] lb.utils.events INFO: eta: 13:53:20 iteration: 284899/375342 consumed_samples: 291737600 total_loss: 0.3724 time: 0.5443 s/iter data_time: 0.0569 s/iter total_throughput: 1881.24 samples/s lr: 1.45e-04 [09/20 23:28:18] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0284999 [09/20 23:28:18] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/20 23:28:18] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/20 23:28:23] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0502 s/iter. Inference: 0.2525 s/iter. Eval: 0.0022 s/iter. Total: 0.3049 s/iter. ETA=0:00:11 [09/20 23:28:28] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0646 s/iter. Inference: 0.2674 s/iter. Eval: 0.0023 s/iter. Total: 0.3347 s/iter. ETA=0:00:07 [09/20 23:28:34] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0656 s/iter. Inference: 0.2616 s/iter. Eval: 0.0031 s/iter. Total: 0.3307 s/iter. ETA=0:00:01 [09/20 23:28:36] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/20 23:28:36] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.279911 (0.000286 s / iter per device, on 8 devices) [09/20 23:28:36] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000228 s / iter per device, on 8 devices) [09/20 23:28:36] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/20 23:28:36] lb.evaluation.utils INFO: copypaste: Acc@1=78.642 [09/20 23:28:36] lb.evaluation.utils INFO: copypaste: Acc@5=94.266 [09/20 23:28:36] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 78.64200, better than last best score 78.44600 @ iteration 279999. [09/20 23:28:36] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/20 23:28:36] lb.utils.events INFO: eta: 13:52:26 iteration: 284999/375342 consumed_samples: 291840000 total_loss: 0.3725 time: 0.5443 s/iter data_time: 0.0540 s/iter total_throughput: 1881.23 samples/s lr: 1.45e-04 [09/20 23:29:31] lb.utils.events INFO: eta: 13:51:15 iteration: 285099/375342 consumed_samples: 291942400 total_loss: 0.3705 time: 0.5443 s/iter data_time: 0.0543 s/iter total_throughput: 1881.22 samples/s lr: 1.45e-04 [09/20 23:30:27] lb.utils.events INFO: eta: 13:50:18 iteration: 285199/375342 consumed_samples: 292044800 total_loss: 0.3663 time: 0.5443 s/iter data_time: 0.0552 s/iter total_throughput: 1881.21 samples/s lr: 1.44e-04 [09/20 23:31:22] lb.utils.events INFO: eta: 13:48:45 iteration: 285299/375342 consumed_samples: 292147200 total_loss: 0.3684 time: 0.5443 s/iter data_time: 0.0519 s/iter total_throughput: 1881.20 samples/s lr: 1.44e-04 [09/20 23:32:16] lb.utils.events INFO: eta: 13:47:10 iteration: 285399/375342 consumed_samples: 292249600 total_loss: 0.3693 time: 0.5443 s/iter data_time: 0.0525 s/iter total_throughput: 1881.20 samples/s lr: 1.44e-04 [09/20 23:33:11] lb.utils.events INFO: eta: 13:45:08 iteration: 285499/375342 consumed_samples: 292352000 total_loss: 0.3665 time: 0.5443 s/iter data_time: 0.0524 s/iter total_throughput: 1881.19 samples/s lr: 1.43e-04 [09/20 23:34:06] lb.utils.events INFO: eta: 13:43:08 iteration: 285599/375342 consumed_samples: 292454400 total_loss: 0.3626 time: 0.5443 s/iter data_time: 0.0509 s/iter total_throughput: 1881.19 samples/s lr: 1.43e-04 [09/20 23:35:00] lb.utils.events INFO: eta: 13:41:25 iteration: 285699/375342 consumed_samples: 292556800 total_loss: 0.3668 time: 0.5443 s/iter data_time: 0.0528 s/iter total_throughput: 1881.19 samples/s lr: 1.43e-04 [09/20 23:35:55] lb.utils.events INFO: eta: 13:39:08 iteration: 285799/375342 consumed_samples: 292659200 total_loss: 0.3675 time: 0.5443 s/iter data_time: 0.0523 s/iter total_throughput: 1881.19 samples/s lr: 1.43e-04 [09/20 23:36:50] lb.utils.events INFO: eta: 13:37:17 iteration: 285899/375342 consumed_samples: 292761600 total_loss: 0.3655 time: 0.5443 s/iter data_time: 0.0532 s/iter total_throughput: 1881.19 samples/s lr: 1.42e-04 [09/20 23:37:44] lb.utils.events INFO: eta: 13:35:43 iteration: 285999/375342 consumed_samples: 292864000 total_loss: 0.366 time: 0.5443 s/iter data_time: 0.0501 s/iter total_throughput: 1881.18 samples/s lr: 1.42e-04 [09/20 23:38:39] lb.utils.events INFO: eta: 13:34:06 iteration: 286099/375342 consumed_samples: 292966400 total_loss: 0.3716 time: 0.5443 s/iter data_time: 0.0493 s/iter total_throughput: 1881.18 samples/s lr: 1.42e-04 [09/20 23:39:34] lb.utils.events INFO: eta: 13:32:35 iteration: 286199/375342 consumed_samples: 293068800 total_loss: 0.3708 time: 0.5443 s/iter data_time: 0.0492 s/iter total_throughput: 1881.18 samples/s lr: 1.42e-04 [09/20 23:40:29] lb.utils.events INFO: eta: 13:31:17 iteration: 286299/375342 consumed_samples: 293171200 total_loss: 0.3679 time: 0.5443 s/iter data_time: 0.0566 s/iter total_throughput: 1881.17 samples/s lr: 1.41e-04 [09/20 23:41:24] lb.utils.events INFO: eta: 13:30:02 iteration: 286399/375342 consumed_samples: 293273600 total_loss: 0.3683 time: 0.5443 s/iter data_time: 0.0550 s/iter total_throughput: 1881.17 samples/s lr: 1.41e-04 [09/20 23:42:19] lb.utils.events INFO: eta: 13:29:06 iteration: 286499/375342 consumed_samples: 293376000 total_loss: 0.3679 time: 0.5443 s/iter data_time: 0.0534 s/iter total_throughput: 1881.16 samples/s lr: 1.41e-04 [09/20 23:43:14] lb.utils.events INFO: eta: 13:28:35 iteration: 286599/375342 consumed_samples: 293478400 total_loss: 0.3711 time: 0.5443 s/iter data_time: 0.0528 s/iter total_throughput: 1881.15 samples/s lr: 1.40e-04 [09/20 23:44:09] lb.utils.events INFO: eta: 13:28:14 iteration: 286699/375342 consumed_samples: 293580800 total_loss: 0.36 time: 0.5443 s/iter data_time: 0.0588 s/iter total_throughput: 1881.15 samples/s lr: 1.40e-04 [09/20 23:45:04] lb.utils.events INFO: eta: 13:28:13 iteration: 286799/375342 consumed_samples: 293683200 total_loss: 0.3601 time: 0.5444 s/iter data_time: 0.0557 s/iter total_throughput: 1881.14 samples/s lr: 1.40e-04 [09/20 23:45:59] lb.utils.events INFO: eta: 13:28:05 iteration: 286899/375342 consumed_samples: 293785600 total_loss: 0.363 time: 0.5444 s/iter data_time: 0.0550 s/iter total_throughput: 1881.13 samples/s lr: 1.40e-04 [09/20 23:46:54] lb.utils.events INFO: eta: 13:28:02 iteration: 286999/375342 consumed_samples: 293888000 total_loss: 0.3638 time: 0.5444 s/iter data_time: 0.0500 s/iter total_throughput: 1881.12 samples/s lr: 1.39e-04 [09/20 23:47:49] lb.utils.events INFO: eta: 13:27:51 iteration: 287099/375342 consumed_samples: 293990400 total_loss: 0.3655 time: 0.5444 s/iter data_time: 0.0515 s/iter total_throughput: 1881.12 samples/s lr: 1.39e-04 [09/20 23:48:44] lb.utils.events INFO: eta: 13:27:33 iteration: 287199/375342 consumed_samples: 294092800 total_loss: 0.3715 time: 0.5444 s/iter data_time: 0.0596 s/iter total_throughput: 1881.11 samples/s lr: 1.39e-04 [09/20 23:49:39] lb.utils.events INFO: eta: 13:27:24 iteration: 287299/375342 consumed_samples: 294195200 total_loss: 0.368 time: 0.5444 s/iter data_time: 0.0488 s/iter total_throughput: 1881.10 samples/s lr: 1.38e-04 [09/20 23:50:34] lb.utils.events INFO: eta: 13:27:01 iteration: 287399/375342 consumed_samples: 294297600 total_loss: 0.3566 time: 0.5444 s/iter data_time: 0.0533 s/iter total_throughput: 1881.09 samples/s lr: 1.38e-04 [09/20 23:51:29] lb.utils.events INFO: eta: 13:26:16 iteration: 287499/375342 consumed_samples: 294400000 total_loss: 0.364 time: 0.5444 s/iter data_time: 0.0458 s/iter total_throughput: 1881.09 samples/s lr: 1.38e-04 [09/20 23:52:24] lb.utils.events INFO: eta: 13:25:41 iteration: 287599/375342 consumed_samples: 294502400 total_loss: 0.3741 time: 0.5444 s/iter data_time: 0.0426 s/iter total_throughput: 1881.08 samples/s lr: 1.38e-04 [09/20 23:53:20] lb.utils.events INFO: eta: 13:24:45 iteration: 287699/375342 consumed_samples: 294604800 total_loss: 0.376 time: 0.5444 s/iter data_time: 0.0529 s/iter total_throughput: 1881.07 samples/s lr: 1.37e-04 [09/20 23:54:15] lb.utils.events INFO: eta: 13:24:14 iteration: 287799/375342 consumed_samples: 294707200 total_loss: 0.3718 time: 0.5444 s/iter data_time: 0.0524 s/iter total_throughput: 1881.06 samples/s lr: 1.37e-04 [09/20 23:55:10] lb.utils.events INFO: eta: 13:23:35 iteration: 287899/375342 consumed_samples: 294809600 total_loss: 0.3672 time: 0.5444 s/iter data_time: 0.0536 s/iter total_throughput: 1881.05 samples/s lr: 1.37e-04 [09/20 23:56:05] lb.utils.events INFO: eta: 13:22:37 iteration: 287999/375342 consumed_samples: 294912000 total_loss: 0.3696 time: 0.5444 s/iter data_time: 0.0527 s/iter total_throughput: 1881.04 samples/s lr: 1.36e-04 [09/20 23:57:01] lb.utils.events INFO: eta: 13:21:55 iteration: 288099/375342 consumed_samples: 295014400 total_loss: 0.366 time: 0.5444 s/iter data_time: 0.0528 s/iter total_throughput: 1881.03 samples/s lr: 1.36e-04 [09/20 23:57:56] lb.utils.events INFO: eta: 13:21:16 iteration: 288199/375342 consumed_samples: 295116800 total_loss: 0.3628 time: 0.5444 s/iter data_time: 0.0542 s/iter total_throughput: 1881.02 samples/s lr: 1.36e-04 [09/20 23:58:51] lb.utils.events INFO: eta: 13:20:21 iteration: 288299/375342 consumed_samples: 295219200 total_loss: 0.3638 time: 0.5444 s/iter data_time: 0.0537 s/iter total_throughput: 1881.02 samples/s lr: 1.36e-04 [09/20 23:59:46] lb.utils.events INFO: eta: 13:19:14 iteration: 288399/375342 consumed_samples: 295321600 total_loss: 0.3638 time: 0.5444 s/iter data_time: 0.0543 s/iter total_throughput: 1881.01 samples/s lr: 1.35e-04 [09/21 00:00:41] lb.utils.events INFO: eta: 13:17:58 iteration: 288499/375342 consumed_samples: 295424000 total_loss: 0.364 time: 0.5444 s/iter data_time: 0.0521 s/iter total_throughput: 1881.01 samples/s lr: 1.35e-04 [09/21 00:01:36] lb.utils.events INFO: eta: 13:16:43 iteration: 288599/375342 consumed_samples: 295526400 total_loss: 0.3677 time: 0.5444 s/iter data_time: 0.0496 s/iter total_throughput: 1881.00 samples/s lr: 1.35e-04 [09/21 00:02:30] lb.utils.events INFO: eta: 13:15:19 iteration: 288699/375342 consumed_samples: 295628800 total_loss: 0.3696 time: 0.5444 s/iter data_time: 0.0520 s/iter total_throughput: 1881.00 samples/s lr: 1.35e-04 [09/21 00:03:25] lb.utils.events INFO: eta: 13:13:27 iteration: 288799/375342 consumed_samples: 295731200 total_loss: 0.3667 time: 0.5444 s/iter data_time: 0.0530 s/iter total_throughput: 1880.99 samples/s lr: 1.34e-04 [09/21 00:04:20] lb.utils.events INFO: eta: 13:11:48 iteration: 288899/375342 consumed_samples: 295833600 total_loss: 0.3686 time: 0.5444 s/iter data_time: 0.0516 s/iter total_throughput: 1880.99 samples/s lr: 1.34e-04 [09/21 00:05:14] lb.utils.events INFO: eta: 13:10:03 iteration: 288999/375342 consumed_samples: 295936000 total_loss: 0.3684 time: 0.5444 s/iter data_time: 0.0507 s/iter total_throughput: 1880.99 samples/s lr: 1.34e-04 [09/21 00:06:09] lb.utils.events INFO: eta: 13:07:46 iteration: 289099/375342 consumed_samples: 296038400 total_loss: 0.3601 time: 0.5444 s/iter data_time: 0.0520 s/iter total_throughput: 1880.99 samples/s lr: 1.33e-04 [09/21 00:07:03] lb.utils.events INFO: eta: 13:05:52 iteration: 289199/375342 consumed_samples: 296140800 total_loss: 0.3566 time: 0.5444 s/iter data_time: 0.0517 s/iter total_throughput: 1880.99 samples/s lr: 1.33e-04 [09/21 00:07:58] lb.utils.events INFO: eta: 13:03:58 iteration: 289299/375342 consumed_samples: 296243200 total_loss: 0.3639 time: 0.5444 s/iter data_time: 0.0528 s/iter total_throughput: 1880.99 samples/s lr: 1.33e-04 [09/21 00:08:52] lb.utils.events INFO: eta: 13:02:28 iteration: 289399/375342 consumed_samples: 296345600 total_loss: 0.3666 time: 0.5444 s/iter data_time: 0.0501 s/iter total_throughput: 1880.99 samples/s lr: 1.33e-04 [09/21 00:09:47] lb.utils.events INFO: eta: 13:01:01 iteration: 289499/375342 consumed_samples: 296448000 total_loss: 0.3712 time: 0.5444 s/iter data_time: 0.0506 s/iter total_throughput: 1880.99 samples/s lr: 1.32e-04 [09/21 00:10:42] lb.utils.events INFO: eta: 12:59:45 iteration: 289599/375342 consumed_samples: 296550400 total_loss: 0.371 time: 0.5444 s/iter data_time: 0.0499 s/iter total_throughput: 1880.99 samples/s lr: 1.32e-04 [09/21 00:11:36] lb.utils.events INFO: eta: 12:58:27 iteration: 289699/375342 consumed_samples: 296652800 total_loss: 0.3711 time: 0.5444 s/iter data_time: 0.0502 s/iter total_throughput: 1880.99 samples/s lr: 1.32e-04 [09/21 00:12:31] lb.utils.events INFO: eta: 12:57:28 iteration: 289799/375342 consumed_samples: 296755200 total_loss: 0.37 time: 0.5444 s/iter data_time: 0.0523 s/iter total_throughput: 1880.98 samples/s lr: 1.32e-04 [09/21 00:13:26] lb.utils.events INFO: eta: 12:56:41 iteration: 289899/375342 consumed_samples: 296857600 total_loss: 0.3692 time: 0.5444 s/iter data_time: 0.0504 s/iter total_throughput: 1880.98 samples/s lr: 1.31e-04 [09/21 00:14:21] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0289999 [09/21 00:14:21] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 00:14:21] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 00:14:26] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0545 s/iter. Inference: 0.2515 s/iter. Eval: 0.0023 s/iter. Total: 0.3084 s/iter. ETA=0:00:11 [09/21 00:14:31] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0762 s/iter. Inference: 0.2568 s/iter. Eval: 0.0023 s/iter. Total: 0.3356 s/iter. ETA=0:00:07 [09/21 00:14:36] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0738 s/iter. Inference: 0.2560 s/iter. Eval: 0.0022 s/iter. Total: 0.3325 s/iter. ETA=0:00:01 [09/21 00:14:38] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 00:14:38] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.284371 (0.000286 s / iter per device, on 8 devices) [09/21 00:14:38] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/21 00:14:38] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 00:14:38] lb.evaluation.utils INFO: copypaste: Acc@1=78.892 [09/21 00:14:38] lb.evaluation.utils INFO: copypaste: Acc@5=94.28999999999999 [09/21 00:14:38] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 78.89200, better than last best score 78.64200 @ iteration 284999. [09/21 00:14:38] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 00:14:39] lb.utils.events INFO: eta: 12:56:19 iteration: 289999/375342 consumed_samples: 296960000 total_loss: 0.3696 time: 0.5444 s/iter data_time: 0.0544 s/iter total_throughput: 1880.97 samples/s lr: 1.31e-04 [09/21 00:15:34] lb.utils.events INFO: eta: 12:56:00 iteration: 290099/375342 consumed_samples: 297062400 total_loss: 0.3646 time: 0.5444 s/iter data_time: 0.0552 s/iter total_throughput: 1880.97 samples/s lr: 1.31e-04 [09/21 00:16:29] lb.utils.events INFO: eta: 12:55:51 iteration: 290199/375342 consumed_samples: 297164800 total_loss: 0.3647 time: 0.5444 s/iter data_time: 0.0520 s/iter total_throughput: 1880.96 samples/s lr: 1.30e-04 [09/21 00:17:24] lb.utils.events INFO: eta: 12:56:09 iteration: 290299/375342 consumed_samples: 297267200 total_loss: 0.3654 time: 0.5444 s/iter data_time: 0.0557 s/iter total_throughput: 1880.95 samples/s lr: 1.30e-04 [09/21 00:18:19] lb.utils.events INFO: eta: 12:55:51 iteration: 290399/375342 consumed_samples: 297369600 total_loss: 0.3577 time: 0.5444 s/iter data_time: 0.0534 s/iter total_throughput: 1880.94 samples/s lr: 1.30e-04 [09/21 00:19:14] lb.utils.events INFO: eta: 12:55:25 iteration: 290499/375342 consumed_samples: 297472000 total_loss: 0.3653 time: 0.5444 s/iter data_time: 0.0536 s/iter total_throughput: 1880.93 samples/s lr: 1.30e-04 [09/21 00:20:09] lb.utils.events INFO: eta: 12:55:18 iteration: 290599/375342 consumed_samples: 297574400 total_loss: 0.3668 time: 0.5444 s/iter data_time: 0.0534 s/iter total_throughput: 1880.93 samples/s lr: 1.29e-04 [09/21 00:21:05] lb.utils.events INFO: eta: 12:55:25 iteration: 290699/375342 consumed_samples: 297676800 total_loss: 0.3541 time: 0.5444 s/iter data_time: 0.0525 s/iter total_throughput: 1880.92 samples/s lr: 1.29e-04 [09/21 00:22:00] lb.utils.events INFO: eta: 12:54:52 iteration: 290799/375342 consumed_samples: 297779200 total_loss: 0.3538 time: 0.5444 s/iter data_time: 0.0519 s/iter total_throughput: 1880.91 samples/s lr: 1.29e-04 [09/21 00:22:55] lb.utils.events INFO: eta: 12:54:30 iteration: 290899/375342 consumed_samples: 297881600 total_loss: 0.3613 time: 0.5444 s/iter data_time: 0.0515 s/iter total_throughput: 1880.91 samples/s lr: 1.29e-04 [09/21 00:23:50] lb.utils.events INFO: eta: 12:53:49 iteration: 290999/375342 consumed_samples: 297984000 total_loss: 0.3659 time: 0.5444 s/iter data_time: 0.0448 s/iter total_throughput: 1880.90 samples/s lr: 1.28e-04 [09/21 00:24:45] lb.utils.events INFO: eta: 12:53:26 iteration: 291099/375342 consumed_samples: 298086400 total_loss: 0.368 time: 0.5444 s/iter data_time: 0.0526 s/iter total_throughput: 1880.89 samples/s lr: 1.28e-04 [09/21 00:25:40] lb.utils.events INFO: eta: 12:52:34 iteration: 291199/375342 consumed_samples: 298188800 total_loss: 0.368 time: 0.5444 s/iter data_time: 0.0533 s/iter total_throughput: 1880.88 samples/s lr: 1.28e-04 [09/21 00:26:35] lb.utils.events INFO: eta: 12:51:40 iteration: 291299/375342 consumed_samples: 298291200 total_loss: 0.3653 time: 0.5444 s/iter data_time: 0.0521 s/iter total_throughput: 1880.88 samples/s lr: 1.28e-04 [09/21 00:27:30] lb.utils.events INFO: eta: 12:50:39 iteration: 291399/375342 consumed_samples: 298393600 total_loss: 0.362 time: 0.5444 s/iter data_time: 0.0519 s/iter total_throughput: 1880.87 samples/s lr: 1.27e-04 [09/21 00:28:25] lb.utils.events INFO: eta: 12:49:37 iteration: 291499/375342 consumed_samples: 298496000 total_loss: 0.3662 time: 0.5444 s/iter data_time: 0.0519 s/iter total_throughput: 1880.86 samples/s lr: 1.27e-04 [09/21 00:29:20] lb.utils.events INFO: eta: 12:48:27 iteration: 291599/375342 consumed_samples: 298598400 total_loss: 0.3651 time: 0.5444 s/iter data_time: 0.0510 s/iter total_throughput: 1880.86 samples/s lr: 1.27e-04 [09/21 00:30:15] lb.utils.events INFO: eta: 12:47:08 iteration: 291699/375342 consumed_samples: 298700800 total_loss: 0.3594 time: 0.5444 s/iter data_time: 0.0514 s/iter total_throughput: 1880.85 samples/s lr: 1.26e-04 [09/21 00:31:10] lb.utils.events INFO: eta: 12:46:09 iteration: 291799/375342 consumed_samples: 298803200 total_loss: 0.3611 time: 0.5444 s/iter data_time: 0.0505 s/iter total_throughput: 1880.85 samples/s lr: 1.26e-04 [09/21 00:32:05] lb.utils.events INFO: eta: 12:44:43 iteration: 291899/375342 consumed_samples: 298905600 total_loss: 0.3617 time: 0.5444 s/iter data_time: 0.0522 s/iter total_throughput: 1880.84 samples/s lr: 1.26e-04 [09/21 00:33:00] lb.utils.events INFO: eta: 12:43:12 iteration: 291999/375342 consumed_samples: 299008000 total_loss: 0.3574 time: 0.5444 s/iter data_time: 0.0525 s/iter total_throughput: 1880.84 samples/s lr: 1.26e-04 [09/21 00:33:54] lb.utils.events INFO: eta: 12:41:52 iteration: 292099/375342 consumed_samples: 299110400 total_loss: 0.3635 time: 0.5444 s/iter data_time: 0.0509 s/iter total_throughput: 1880.84 samples/s lr: 1.25e-04 [09/21 00:34:49] lb.utils.events INFO: eta: 12:40:28 iteration: 292199/375342 consumed_samples: 299212800 total_loss: 0.3646 time: 0.5444 s/iter data_time: 0.0526 s/iter total_throughput: 1880.83 samples/s lr: 1.25e-04 [09/21 00:35:44] lb.utils.events INFO: eta: 12:38:41 iteration: 292299/375342 consumed_samples: 299315200 total_loss: 0.3639 time: 0.5444 s/iter data_time: 0.0527 s/iter total_throughput: 1880.83 samples/s lr: 1.25e-04 [09/21 00:36:38] lb.utils.events INFO: eta: 12:37:01 iteration: 292399/375342 consumed_samples: 299417600 total_loss: 0.3675 time: 0.5444 s/iter data_time: 0.0529 s/iter total_throughput: 1880.83 samples/s lr: 1.25e-04 [09/21 00:37:33] lb.utils.events INFO: eta: 12:35:14 iteration: 292499/375342 consumed_samples: 299520000 total_loss: 0.3681 time: 0.5444 s/iter data_time: 0.0496 s/iter total_throughput: 1880.83 samples/s lr: 1.24e-04 [09/21 00:38:27] lb.utils.events INFO: eta: 12:33:22 iteration: 292599/375342 consumed_samples: 299622400 total_loss: 0.3663 time: 0.5444 s/iter data_time: 0.0516 s/iter total_throughput: 1880.83 samples/s lr: 1.24e-04 [09/21 00:39:21] lb.utils.events INFO: eta: 12:31:58 iteration: 292699/375342 consumed_samples: 299724800 total_loss: 0.3674 time: 0.5444 s/iter data_time: 0.0517 s/iter total_throughput: 1880.83 samples/s lr: 1.24e-04 [09/21 00:40:16] lb.utils.events INFO: eta: 12:30:26 iteration: 292799/375342 consumed_samples: 299827200 total_loss: 0.3671 time: 0.5444 s/iter data_time: 0.0512 s/iter total_throughput: 1880.83 samples/s lr: 1.24e-04 [09/21 00:41:10] lb.utils.events INFO: eta: 12:29:09 iteration: 292899/375342 consumed_samples: 299929600 total_loss: 0.3608 time: 0.5444 s/iter data_time: 0.0469 s/iter total_throughput: 1880.83 samples/s lr: 1.23e-04 [09/21 00:42:05] lb.utils.events INFO: eta: 12:27:50 iteration: 292999/375342 consumed_samples: 300032000 total_loss: 0.3658 time: 0.5444 s/iter data_time: 0.0489 s/iter total_throughput: 1880.83 samples/s lr: 1.23e-04 [09/21 00:43:00] lb.utils.events INFO: eta: 12:26:45 iteration: 293099/375342 consumed_samples: 300134400 total_loss: 0.3674 time: 0.5444 s/iter data_time: 0.0493 s/iter total_throughput: 1880.83 samples/s lr: 1.23e-04 [09/21 00:43:55] lb.utils.events INFO: eta: 12:25:47 iteration: 293199/375342 consumed_samples: 300236800 total_loss: 0.3617 time: 0.5444 s/iter data_time: 0.0539 s/iter total_throughput: 1880.82 samples/s lr: 1.22e-04 [09/21 00:44:50] lb.utils.events INFO: eta: 12:25:07 iteration: 293299/375342 consumed_samples: 300339200 total_loss: 0.3631 time: 0.5444 s/iter data_time: 0.0537 s/iter total_throughput: 1880.82 samples/s lr: 1.22e-04 [09/21 00:45:44] lb.utils.events INFO: eta: 12:24:54 iteration: 293399/375342 consumed_samples: 300441600 total_loss: 0.3668 time: 0.5444 s/iter data_time: 0.0513 s/iter total_throughput: 1880.81 samples/s lr: 1.22e-04 [09/21 00:46:39] lb.utils.events INFO: eta: 12:24:32 iteration: 293499/375342 consumed_samples: 300544000 total_loss: 0.3652 time: 0.5444 s/iter data_time: 0.0547 s/iter total_throughput: 1880.81 samples/s lr: 1.22e-04 [09/21 00:47:34] lb.utils.events INFO: eta: 12:24:31 iteration: 293599/375342 consumed_samples: 300646400 total_loss: 0.3647 time: 0.5444 s/iter data_time: 0.0549 s/iter total_throughput: 1880.80 samples/s lr: 1.21e-04 [09/21 00:48:29] lb.utils.events INFO: eta: 12:24:25 iteration: 293699/375342 consumed_samples: 300748800 total_loss: 0.367 time: 0.5445 s/iter data_time: 0.0564 s/iter total_throughput: 1880.79 samples/s lr: 1.21e-04 [09/21 00:49:25] lb.utils.events INFO: eta: 12:24:32 iteration: 293799/375342 consumed_samples: 300851200 total_loss: 0.3666 time: 0.5445 s/iter data_time: 0.0548 s/iter total_throughput: 1880.78 samples/s lr: 1.21e-04 [09/21 00:50:20] lb.utils.events INFO: eta: 12:24:45 iteration: 293899/375342 consumed_samples: 300953600 total_loss: 0.36 time: 0.5445 s/iter data_time: 0.0520 s/iter total_throughput: 1880.78 samples/s lr: 1.21e-04 [09/21 00:51:15] lb.utils.events INFO: eta: 12:24:43 iteration: 293999/375342 consumed_samples: 301056000 total_loss: 0.3634 time: 0.5445 s/iter data_time: 0.0505 s/iter total_throughput: 1880.77 samples/s lr: 1.20e-04 [09/21 00:52:10] lb.utils.events INFO: eta: 12:24:43 iteration: 294099/375342 consumed_samples: 301158400 total_loss: 0.3688 time: 0.5445 s/iter data_time: 0.0543 s/iter total_throughput: 1880.76 samples/s lr: 1.20e-04 [09/21 00:53:05] lb.utils.events INFO: eta: 12:24:12 iteration: 294199/375342 consumed_samples: 301260800 total_loss: 0.3626 time: 0.5445 s/iter data_time: 0.0508 s/iter total_throughput: 1880.75 samples/s lr: 1.20e-04 [09/21 00:54:00] lb.utils.events INFO: eta: 12:23:26 iteration: 294299/375342 consumed_samples: 301363200 total_loss: 0.3467 time: 0.5445 s/iter data_time: 0.0520 s/iter total_throughput: 1880.75 samples/s lr: 1.20e-04 [09/21 00:54:55] lb.utils.events INFO: eta: 12:22:50 iteration: 294399/375342 consumed_samples: 301465600 total_loss: 0.3585 time: 0.5445 s/iter data_time: 0.0495 s/iter total_throughput: 1880.74 samples/s lr: 1.19e-04 [09/21 00:55:50] lb.utils.events INFO: eta: 12:22:17 iteration: 294499/375342 consumed_samples: 301568000 total_loss: 0.3659 time: 0.5445 s/iter data_time: 0.0476 s/iter total_throughput: 1880.73 samples/s lr: 1.19e-04 [09/21 00:56:45] lb.utils.events INFO: eta: 12:21:55 iteration: 294599/375342 consumed_samples: 301670400 total_loss: 0.3675 time: 0.5445 s/iter data_time: 0.0478 s/iter total_throughput: 1880.73 samples/s lr: 1.19e-04 [09/21 00:57:41] lb.utils.events INFO: eta: 12:21:12 iteration: 294699/375342 consumed_samples: 301772800 total_loss: 0.3613 time: 0.5445 s/iter data_time: 0.0517 s/iter total_throughput: 1880.72 samples/s lr: 1.19e-04 [09/21 00:58:36] lb.utils.events INFO: eta: 12:20:01 iteration: 294799/375342 consumed_samples: 301875200 total_loss: 0.3608 time: 0.5445 s/iter data_time: 0.0508 s/iter total_throughput: 1880.71 samples/s lr: 1.18e-04 [09/21 00:59:31] lb.utils.events INFO: eta: 12:18:56 iteration: 294899/375342 consumed_samples: 301977600 total_loss: 0.3693 time: 0.5445 s/iter data_time: 0.0510 s/iter total_throughput: 1880.70 samples/s lr: 1.18e-04 [09/21 01:00:26] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0294999 [09/21 01:00:26] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 01:00:26] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 01:00:31] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0659 s/iter. Inference: 0.2519 s/iter. Eval: 0.0022 s/iter. Total: 0.3201 s/iter. ETA=0:00:11 [09/21 01:00:36] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0822 s/iter. Inference: 0.2518 s/iter. Eval: 0.0022 s/iter. Total: 0.3363 s/iter. ETA=0:00:07 [09/21 01:00:41] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0797 s/iter. Inference: 0.2487 s/iter. Eval: 0.0021 s/iter. Total: 0.3308 s/iter. ETA=0:00:01 [09/21 01:00:43] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 01:00:43] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.216647 (0.000284 s / iter per device, on 8 devices) [09/21 01:00:43] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000218 s / iter per device, on 8 devices) [09/21 01:00:43] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 01:00:43] lb.evaluation.utils INFO: copypaste: Acc@1=79.042 [09/21 01:00:43] lb.evaluation.utils INFO: copypaste: Acc@5=94.46 [09/21 01:00:43] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 79.04200, better than last best score 78.89200 @ iteration 289999. [09/21 01:00:43] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 01:00:44] lb.utils.events INFO: eta: 12:17:26 iteration: 294999/375342 consumed_samples: 302080000 total_loss: 0.3621 time: 0.5445 s/iter data_time: 0.0511 s/iter total_throughput: 1880.70 samples/s lr: 1.18e-04 [09/21 01:01:39] lb.utils.events INFO: eta: 12:16:13 iteration: 295099/375342 consumed_samples: 302182400 total_loss: 0.3598 time: 0.5445 s/iter data_time: 0.0528 s/iter total_throughput: 1880.69 samples/s lr: 1.18e-04 [09/21 01:02:34] lb.utils.events INFO: eta: 12:15:09 iteration: 295199/375342 consumed_samples: 302284800 total_loss: 0.3632 time: 0.5445 s/iter data_time: 0.0514 s/iter total_throughput: 1880.69 samples/s lr: 1.17e-04 [09/21 01:03:29] lb.utils.events INFO: eta: 12:13:36 iteration: 295299/375342 consumed_samples: 302387200 total_loss: 0.3612 time: 0.5445 s/iter data_time: 0.0511 s/iter total_throughput: 1880.69 samples/s lr: 1.17e-04 [09/21 01:04:23] lb.utils.events INFO: eta: 12:11:48 iteration: 295399/375342 consumed_samples: 302489600 total_loss: 0.3633 time: 0.5445 s/iter data_time: 0.0527 s/iter total_throughput: 1880.68 samples/s lr: 1.17e-04 [09/21 01:05:18] lb.utils.events INFO: eta: 12:10:16 iteration: 295499/375342 consumed_samples: 302592000 total_loss: 0.3591 time: 0.5445 s/iter data_time: 0.0518 s/iter total_throughput: 1880.68 samples/s lr: 1.16e-04 [09/21 01:06:13] lb.utils.events INFO: eta: 12:08:47 iteration: 295599/375342 consumed_samples: 302694400 total_loss: 0.355 time: 0.5445 s/iter data_time: 0.0528 s/iter total_throughput: 1880.68 samples/s lr: 1.16e-04 [09/21 01:07:07] lb.utils.events INFO: eta: 12:07:13 iteration: 295699/375342 consumed_samples: 302796800 total_loss: 0.3588 time: 0.5445 s/iter data_time: 0.0530 s/iter total_throughput: 1880.68 samples/s lr: 1.16e-04 [09/21 01:08:02] lb.utils.events INFO: eta: 12:05:34 iteration: 295799/375342 consumed_samples: 302899200 total_loss: 0.3653 time: 0.5445 s/iter data_time: 0.0512 s/iter total_throughput: 1880.68 samples/s lr: 1.16e-04 [09/21 01:08:56] lb.utils.events INFO: eta: 12:04:08 iteration: 295899/375342 consumed_samples: 303001600 total_loss: 0.3644 time: 0.5445 s/iter data_time: 0.0525 s/iter total_throughput: 1880.68 samples/s lr: 1.15e-04 [09/21 01:09:51] lb.utils.events INFO: eta: 12:02:43 iteration: 295999/375342 consumed_samples: 303104000 total_loss: 0.3591 time: 0.5445 s/iter data_time: 0.0509 s/iter total_throughput: 1880.68 samples/s lr: 1.15e-04 [09/21 01:10:45] lb.utils.events INFO: eta: 12:01:11 iteration: 296099/375342 consumed_samples: 303206400 total_loss: 0.3601 time: 0.5445 s/iter data_time: 0.0522 s/iter total_throughput: 1880.68 samples/s lr: 1.15e-04 [09/21 01:11:40] lb.utils.events INFO: eta: 11:59:49 iteration: 296199/375342 consumed_samples: 303308800 total_loss: 0.3665 time: 0.5445 s/iter data_time: 0.0519 s/iter total_throughput: 1880.68 samples/s lr: 1.15e-04 [09/21 01:12:34] lb.utils.events INFO: eta: 11:58:28 iteration: 296299/375342 consumed_samples: 303411200 total_loss: 0.3703 time: 0.5445 s/iter data_time: 0.0489 s/iter total_throughput: 1880.68 samples/s lr: 1.14e-04 [09/21 01:13:29] lb.utils.events INFO: eta: 11:57:25 iteration: 296399/375342 consumed_samples: 303513600 total_loss: 0.3665 time: 0.5445 s/iter data_time: 0.0484 s/iter total_throughput: 1880.67 samples/s lr: 1.14e-04 [09/21 01:14:23] lb.utils.events INFO: eta: 11:56:30 iteration: 296499/375342 consumed_samples: 303616000 total_loss: 0.3604 time: 0.5445 s/iter data_time: 0.0498 s/iter total_throughput: 1880.67 samples/s lr: 1.14e-04 [09/21 01:15:18] lb.utils.events INFO: eta: 11:55:29 iteration: 296599/375342 consumed_samples: 303718400 total_loss: 0.363 time: 0.5445 s/iter data_time: 0.0493 s/iter total_throughput: 1880.67 samples/s lr: 1.14e-04 [09/21 01:16:13] lb.utils.events INFO: eta: 11:54:50 iteration: 296699/375342 consumed_samples: 303820800 total_loss: 0.3652 time: 0.5445 s/iter data_time: 0.0513 s/iter total_throughput: 1880.66 samples/s lr: 1.13e-04 [09/21 01:17:08] lb.utils.events INFO: eta: 11:54:24 iteration: 296799/375342 consumed_samples: 303923200 total_loss: 0.3666 time: 0.5445 s/iter data_time: 0.0521 s/iter total_throughput: 1880.66 samples/s lr: 1.13e-04 [09/21 01:18:03] lb.utils.events INFO: eta: 11:53:37 iteration: 296899/375342 consumed_samples: 304025600 total_loss: 0.3678 time: 0.5445 s/iter data_time: 0.0546 s/iter total_throughput: 1880.65 samples/s lr: 1.13e-04 [09/21 01:18:58] lb.utils.events INFO: eta: 11:53:18 iteration: 296999/375342 consumed_samples: 304128000 total_loss: 0.3687 time: 0.5445 s/iter data_time: 0.0533 s/iter total_throughput: 1880.65 samples/s lr: 1.13e-04 [09/21 01:19:53] lb.utils.events INFO: eta: 11:53:12 iteration: 297099/375342 consumed_samples: 304230400 total_loss: 0.3621 time: 0.5445 s/iter data_time: 0.0572 s/iter total_throughput: 1880.64 samples/s lr: 1.12e-04 [09/21 01:20:48] lb.utils.events INFO: eta: 11:52:52 iteration: 297199/375342 consumed_samples: 304332800 total_loss: 0.3621 time: 0.5445 s/iter data_time: 0.0538 s/iter total_throughput: 1880.64 samples/s lr: 1.12e-04 [09/21 01:21:43] lb.utils.events INFO: eta: 11:52:28 iteration: 297299/375342 consumed_samples: 304435200 total_loss: 0.3643 time: 0.5445 s/iter data_time: 0.0522 s/iter total_throughput: 1880.63 samples/s lr: 1.12e-04 [09/21 01:22:38] lb.utils.events INFO: eta: 11:52:06 iteration: 297399/375342 consumed_samples: 304537600 total_loss: 0.3633 time: 0.5445 s/iter data_time: 0.0537 s/iter total_throughput: 1880.63 samples/s lr: 1.12e-04 [09/21 01:23:33] lb.utils.events INFO: eta: 11:51:47 iteration: 297499/375342 consumed_samples: 304640000 total_loss: 0.3606 time: 0.5445 s/iter data_time: 0.0568 s/iter total_throughput: 1880.62 samples/s lr: 1.11e-04 [09/21 01:24:28] lb.utils.events INFO: eta: 11:51:29 iteration: 297599/375342 consumed_samples: 304742400 total_loss: 0.3624 time: 0.5445 s/iter data_time: 0.0573 s/iter total_throughput: 1880.61 samples/s lr: 1.11e-04 [09/21 01:25:23] lb.utils.events INFO: eta: 11:51:06 iteration: 297699/375342 consumed_samples: 304844800 total_loss: 0.366 time: 0.5445 s/iter data_time: 0.0561 s/iter total_throughput: 1880.61 samples/s lr: 1.11e-04 [09/21 01:26:18] lb.utils.events INFO: eta: 11:50:23 iteration: 297799/375342 consumed_samples: 304947200 total_loss: 0.3658 time: 0.5445 s/iter data_time: 0.0527 s/iter total_throughput: 1880.60 samples/s lr: 1.11e-04 [09/21 01:27:13] lb.utils.events INFO: eta: 11:49:42 iteration: 297899/375342 consumed_samples: 305049600 total_loss: 0.3637 time: 0.5445 s/iter data_time: 0.0501 s/iter total_throughput: 1880.60 samples/s lr: 1.10e-04 [09/21 01:28:08] lb.utils.events INFO: eta: 11:49:11 iteration: 297999/375342 consumed_samples: 305152000 total_loss: 0.3704 time: 0.5445 s/iter data_time: 0.0460 s/iter total_throughput: 1880.59 samples/s lr: 1.10e-04 [09/21 01:29:03] lb.utils.events INFO: eta: 11:48:26 iteration: 298099/375342 consumed_samples: 305254400 total_loss: 0.3661 time: 0.5445 s/iter data_time: 0.0491 s/iter total_throughput: 1880.58 samples/s lr: 1.10e-04 [09/21 01:29:58] lb.utils.events INFO: eta: 11:47:35 iteration: 298199/375342 consumed_samples: 305356800 total_loss: 0.3584 time: 0.5445 s/iter data_time: 0.0501 s/iter total_throughput: 1880.58 samples/s lr: 1.10e-04 [09/21 01:30:53] lb.utils.events INFO: eta: 11:46:28 iteration: 298299/375342 consumed_samples: 305459200 total_loss: 0.3607 time: 0.5445 s/iter data_time: 0.0506 s/iter total_throughput: 1880.57 samples/s lr: 1.09e-04 [09/21 01:31:48] lb.utils.events INFO: eta: 11:45:20 iteration: 298399/375342 consumed_samples: 305561600 total_loss: 0.3607 time: 0.5445 s/iter data_time: 0.0513 s/iter total_throughput: 1880.57 samples/s lr: 1.09e-04 [09/21 01:32:42] lb.utils.events INFO: eta: 11:44:04 iteration: 298499/375342 consumed_samples: 305664000 total_loss: 0.3627 time: 0.5445 s/iter data_time: 0.0499 s/iter total_throughput: 1880.56 samples/s lr: 1.09e-04 [09/21 01:33:37] lb.utils.events INFO: eta: 11:42:50 iteration: 298599/375342 consumed_samples: 305766400 total_loss: 0.3674 time: 0.5445 s/iter data_time: 0.0506 s/iter total_throughput: 1880.56 samples/s lr: 1.09e-04 [09/21 01:34:32] lb.utils.events INFO: eta: 11:41:29 iteration: 298699/375342 consumed_samples: 305868800 total_loss: 0.3647 time: 0.5445 s/iter data_time: 0.0496 s/iter total_throughput: 1880.56 samples/s lr: 1.08e-04 [09/21 01:35:26] lb.utils.events INFO: eta: 11:40:05 iteration: 298799/375342 consumed_samples: 305971200 total_loss: 0.3589 time: 0.5445 s/iter data_time: 0.0504 s/iter total_throughput: 1880.56 samples/s lr: 1.08e-04 [09/21 01:36:21] lb.utils.events INFO: eta: 11:38:34 iteration: 298899/375342 consumed_samples: 306073600 total_loss: 0.3609 time: 0.5445 s/iter data_time: 0.0508 s/iter total_throughput: 1880.55 samples/s lr: 1.08e-04 [09/21 01:37:16] lb.utils.events INFO: eta: 11:37:02 iteration: 298999/375342 consumed_samples: 306176000 total_loss: 0.3685 time: 0.5445 s/iter data_time: 0.0501 s/iter total_throughput: 1880.55 samples/s lr: 1.08e-04 [09/21 01:38:10] lb.utils.events INFO: eta: 11:35:39 iteration: 299099/375342 consumed_samples: 306278400 total_loss: 0.3661 time: 0.5445 s/iter data_time: 0.0511 s/iter total_throughput: 1880.55 samples/s lr: 1.07e-04 [09/21 01:39:05] lb.utils.events INFO: eta: 11:33:51 iteration: 299199/375342 consumed_samples: 306380800 total_loss: 0.3602 time: 0.5445 s/iter data_time: 0.0511 s/iter total_throughput: 1880.55 samples/s lr: 1.07e-04 [09/21 01:39:59] lb.utils.events INFO: eta: 11:32:27 iteration: 299299/375342 consumed_samples: 306483200 total_loss: 0.36 time: 0.5445 s/iter data_time: 0.0509 s/iter total_throughput: 1880.55 samples/s lr: 1.07e-04 [09/21 01:40:54] lb.utils.events INFO: eta: 11:30:40 iteration: 299399/375342 consumed_samples: 306585600 total_loss: 0.3584 time: 0.5445 s/iter data_time: 0.0513 s/iter total_throughput: 1880.55 samples/s lr: 1.07e-04 [09/21 01:41:48] lb.utils.events INFO: eta: 11:29:18 iteration: 299499/375342 consumed_samples: 306688000 total_loss: 0.3584 time: 0.5445 s/iter data_time: 0.0527 s/iter total_throughput: 1880.55 samples/s lr: 1.06e-04 [09/21 01:42:43] lb.utils.events INFO: eta: 11:27:43 iteration: 299599/375342 consumed_samples: 306790400 total_loss: 0.3569 time: 0.5445 s/iter data_time: 0.0514 s/iter total_throughput: 1880.55 samples/s lr: 1.06e-04 [09/21 01:43:37] lb.utils.events INFO: eta: 11:26:35 iteration: 299699/375342 consumed_samples: 306892800 total_loss: 0.3537 time: 0.5445 s/iter data_time: 0.0510 s/iter total_throughput: 1880.55 samples/s lr: 1.06e-04 [09/21 01:44:32] lb.utils.events INFO: eta: 11:25:42 iteration: 299799/375342 consumed_samples: 306995200 total_loss: 0.3537 time: 0.5445 s/iter data_time: 0.0501 s/iter total_throughput: 1880.55 samples/s lr: 1.06e-04 [09/21 01:45:26] lb.utils.events INFO: eta: 11:24:47 iteration: 299899/375342 consumed_samples: 307097600 total_loss: 0.3591 time: 0.5445 s/iter data_time: 0.0509 s/iter total_throughput: 1880.55 samples/s lr: 1.05e-04 [09/21 01:46:21] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0299999 [09/21 01:46:21] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 01:46:21] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 01:46:26] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0537 s/iter. Inference: 0.2500 s/iter. Eval: 0.0032 s/iter. Total: 0.3070 s/iter. ETA=0:00:11 [09/21 01:46:31] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0769 s/iter. Inference: 0.2553 s/iter. Eval: 0.0029 s/iter. Total: 0.3352 s/iter. ETA=0:00:07 [09/21 01:46:36] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0763 s/iter. Inference: 0.2526 s/iter. Eval: 0.0031 s/iter. Total: 0.3322 s/iter. ETA=0:00:01 [09/21 01:46:38] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 01:46:38] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.253006 (0.000285 s / iter per device, on 8 devices) [09/21 01:46:38] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000221 s / iter per device, on 8 devices) [09/21 01:46:38] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 01:46:38] lb.evaluation.utils INFO: copypaste: Acc@1=79.092 [09/21 01:46:38] lb.evaluation.utils INFO: copypaste: Acc@5=94.376 [09/21 01:46:38] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 79.09200, better than last best score 79.04200 @ iteration 294999. [09/21 01:46:38] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 01:46:39] lb.utils.events INFO: eta: 11:23:53 iteration: 299999/375342 consumed_samples: 307200000 total_loss: 0.3629 time: 0.5445 s/iter data_time: 0.0509 s/iter total_throughput: 1880.55 samples/s lr: 1.05e-04 [09/21 01:47:34] lb.utils.events INFO: eta: 11:22:56 iteration: 300099/375342 consumed_samples: 307302400 total_loss: 0.3635 time: 0.5445 s/iter data_time: 0.0567 s/iter total_throughput: 1880.54 samples/s lr: 1.05e-04 [09/21 01:48:29] lb.utils.events INFO: eta: 11:22:22 iteration: 300199/375342 consumed_samples: 307404800 total_loss: 0.3617 time: 0.5445 s/iter data_time: 0.0528 s/iter total_throughput: 1880.54 samples/s lr: 1.05e-04 [09/21 01:49:23] lb.utils.events INFO: eta: 11:21:48 iteration: 300299/375342 consumed_samples: 307507200 total_loss: 0.367 time: 0.5445 s/iter data_time: 0.0538 s/iter total_throughput: 1880.54 samples/s lr: 1.04e-04 [09/21 01:50:18] lb.utils.events INFO: eta: 11:21:29 iteration: 300399/375342 consumed_samples: 307609600 total_loss: 0.3659 time: 0.5445 s/iter data_time: 0.0521 s/iter total_throughput: 1880.53 samples/s lr: 1.04e-04 [09/21 01:51:13] lb.utils.events INFO: eta: 11:21:04 iteration: 300499/375342 consumed_samples: 307712000 total_loss: 0.3647 time: 0.5445 s/iter data_time: 0.0513 s/iter total_throughput: 1880.53 samples/s lr: 1.04e-04 [09/21 01:52:08] lb.utils.events INFO: eta: 11:20:43 iteration: 300599/375342 consumed_samples: 307814400 total_loss: 0.3666 time: 0.5445 s/iter data_time: 0.0510 s/iter total_throughput: 1880.53 samples/s lr: 1.04e-04 [09/21 01:53:03] lb.utils.events INFO: eta: 11:20:30 iteration: 300699/375342 consumed_samples: 307916800 total_loss: 0.3694 time: 0.5445 s/iter data_time: 0.0549 s/iter total_throughput: 1880.52 samples/s lr: 1.04e-04 [09/21 01:53:58] lb.utils.events INFO: eta: 11:20:07 iteration: 300799/375342 consumed_samples: 308019200 total_loss: 0.3709 time: 0.5445 s/iter data_time: 0.0526 s/iter total_throughput: 1880.51 samples/s lr: 1.03e-04 [09/21 01:54:53] lb.utils.events INFO: eta: 11:19:41 iteration: 300899/375342 consumed_samples: 308121600 total_loss: 0.3699 time: 0.5445 s/iter data_time: 0.0508 s/iter total_throughput: 1880.51 samples/s lr: 1.03e-04 [09/21 01:55:48] lb.utils.events INFO: eta: 11:19:16 iteration: 300999/375342 consumed_samples: 308224000 total_loss: 0.3661 time: 0.5445 s/iter data_time: 0.0555 s/iter total_throughput: 1880.50 samples/s lr: 1.03e-04 [09/21 01:56:43] lb.utils.events INFO: eta: 11:18:51 iteration: 301099/375342 consumed_samples: 308326400 total_loss: 0.3641 time: 0.5445 s/iter data_time: 0.0558 s/iter total_throughput: 1880.50 samples/s lr: 1.03e-04 [09/21 01:57:38] lb.utils.events INFO: eta: 11:18:18 iteration: 301199/375342 consumed_samples: 308428800 total_loss: 0.3641 time: 0.5445 s/iter data_time: 0.0560 s/iter total_throughput: 1880.49 samples/s lr: 1.02e-04 [09/21 01:58:33] lb.utils.events INFO: eta: 11:17:29 iteration: 301299/375342 consumed_samples: 308531200 total_loss: 0.3628 time: 0.5445 s/iter data_time: 0.0516 s/iter total_throughput: 1880.49 samples/s lr: 1.02e-04 [09/21 01:59:28] lb.utils.events INFO: eta: 11:16:39 iteration: 301399/375342 consumed_samples: 308633600 total_loss: 0.3671 time: 0.5445 s/iter data_time: 0.0547 s/iter total_throughput: 1880.48 samples/s lr: 1.02e-04 [09/21 02:00:23] lb.utils.events INFO: eta: 11:15:51 iteration: 301499/375342 consumed_samples: 308736000 total_loss: 0.3651 time: 0.5445 s/iter data_time: 0.0516 s/iter total_throughput: 1880.48 samples/s lr: 1.02e-04 [09/21 02:01:17] lb.utils.events INFO: eta: 11:14:45 iteration: 301599/375342 consumed_samples: 308838400 total_loss: 0.3579 time: 0.5445 s/iter data_time: 0.0548 s/iter total_throughput: 1880.47 samples/s lr: 1.01e-04 [09/21 02:02:13] lb.utils.events INFO: eta: 11:13:41 iteration: 301699/375342 consumed_samples: 308940800 total_loss: 0.3589 time: 0.5445 s/iter data_time: 0.0544 s/iter total_throughput: 1880.46 samples/s lr: 1.01e-04 [09/21 02:03:08] lb.utils.events INFO: eta: 11:12:39 iteration: 301799/375342 consumed_samples: 309043200 total_loss: 0.3654 time: 0.5445 s/iter data_time: 0.0524 s/iter total_throughput: 1880.46 samples/s lr: 1.01e-04 [09/21 02:04:02] lb.utils.events INFO: eta: 11:11:43 iteration: 301899/375342 consumed_samples: 309145600 total_loss: 0.3646 time: 0.5445 s/iter data_time: 0.0540 s/iter total_throughput: 1880.45 samples/s lr: 1.01e-04 [09/21 02:04:57] lb.utils.events INFO: eta: 11:11:05 iteration: 301999/375342 consumed_samples: 309248000 total_loss: 0.3649 time: 0.5446 s/iter data_time: 0.0537 s/iter total_throughput: 1880.45 samples/s lr: 1.00e-04 [09/21 02:05:52] lb.utils.events INFO: eta: 11:10:13 iteration: 302099/375342 consumed_samples: 309350400 total_loss: 0.3603 time: 0.5446 s/iter data_time: 0.0549 s/iter total_throughput: 1880.44 samples/s lr: 1.00e-04 [09/21 02:06:47] lb.utils.events INFO: eta: 11:09:05 iteration: 302199/375342 consumed_samples: 309452800 total_loss: 0.3553 time: 0.5446 s/iter data_time: 0.0549 s/iter total_throughput: 1880.44 samples/s lr: 9.99e-05 [09/21 02:07:42] lb.utils.events INFO: eta: 11:08:06 iteration: 302299/375342 consumed_samples: 309555200 total_loss: 0.3556 time: 0.5446 s/iter data_time: 0.0538 s/iter total_throughput: 1880.44 samples/s lr: 9.97e-05 [09/21 02:08:37] lb.utils.events INFO: eta: 11:07:15 iteration: 302399/375342 consumed_samples: 309657600 total_loss: 0.3614 time: 0.5446 s/iter data_time: 0.0543 s/iter total_throughput: 1880.43 samples/s lr: 9.94e-05 [09/21 02:09:32] lb.utils.events INFO: eta: 11:06:32 iteration: 302499/375342 consumed_samples: 309760000 total_loss: 0.3622 time: 0.5446 s/iter data_time: 0.0514 s/iter total_throughput: 1880.42 samples/s lr: 9.92e-05 [09/21 02:10:27] lb.utils.events INFO: eta: 11:05:54 iteration: 302599/375342 consumed_samples: 309862400 total_loss: 0.3607 time: 0.5446 s/iter data_time: 0.0520 s/iter total_throughput: 1880.42 samples/s lr: 9.89e-05 [09/21 02:11:22] lb.utils.events INFO: eta: 11:05:02 iteration: 302699/375342 consumed_samples: 309964800 total_loss: 0.3602 time: 0.5446 s/iter data_time: 0.0502 s/iter total_throughput: 1880.41 samples/s lr: 9.87e-05 [09/21 02:12:17] lb.utils.events INFO: eta: 11:04:09 iteration: 302799/375342 consumed_samples: 310067200 total_loss: 0.3558 time: 0.5446 s/iter data_time: 0.0510 s/iter total_throughput: 1880.41 samples/s lr: 9.85e-05 [09/21 02:13:12] lb.utils.events INFO: eta: 11:03:17 iteration: 302899/375342 consumed_samples: 310169600 total_loss: 0.3633 time: 0.5446 s/iter data_time: 0.0532 s/iter total_throughput: 1880.40 samples/s lr: 9.82e-05 [09/21 02:14:07] lb.utils.events INFO: eta: 11:02:11 iteration: 302999/375342 consumed_samples: 310272000 total_loss: 0.3656 time: 0.5446 s/iter data_time: 0.0513 s/iter total_throughput: 1880.40 samples/s lr: 9.80e-05 [09/21 02:15:02] lb.utils.events INFO: eta: 11:01:23 iteration: 303099/375342 consumed_samples: 310374400 total_loss: 0.3589 time: 0.5446 s/iter data_time: 0.0509 s/iter total_throughput: 1880.39 samples/s lr: 9.78e-05 [09/21 02:15:57] lb.utils.events INFO: eta: 11:00:41 iteration: 303199/375342 consumed_samples: 310476800 total_loss: 0.36 time: 0.5446 s/iter data_time: 0.0505 s/iter total_throughput: 1880.38 samples/s lr: 9.75e-05 [09/21 02:16:52] lb.utils.events INFO: eta: 11:00:09 iteration: 303299/375342 consumed_samples: 310579200 total_loss: 0.3582 time: 0.5446 s/iter data_time: 0.0516 s/iter total_throughput: 1880.38 samples/s lr: 9.73e-05 [09/21 02:17:47] lb.utils.events INFO: eta: 10:59:13 iteration: 303399/375342 consumed_samples: 310681600 total_loss: 0.3604 time: 0.5446 s/iter data_time: 0.0503 s/iter total_throughput: 1880.37 samples/s lr: 9.71e-05 [09/21 02:18:42] lb.utils.events INFO: eta: 10:58:15 iteration: 303499/375342 consumed_samples: 310784000 total_loss: 0.3595 time: 0.5446 s/iter data_time: 0.0518 s/iter total_throughput: 1880.37 samples/s lr: 9.68e-05 [09/21 02:19:37] lb.utils.events INFO: eta: 10:57:07 iteration: 303599/375342 consumed_samples: 310886400 total_loss: 0.3594 time: 0.5446 s/iter data_time: 0.0519 s/iter total_throughput: 1880.36 samples/s lr: 9.66e-05 [09/21 02:20:32] lb.utils.events INFO: eta: 10:55:56 iteration: 303699/375342 consumed_samples: 310988800 total_loss: 0.3595 time: 0.5446 s/iter data_time: 0.0546 s/iter total_throughput: 1880.35 samples/s lr: 9.64e-05 [09/21 02:21:27] lb.utils.events INFO: eta: 10:55:02 iteration: 303799/375342 consumed_samples: 311091200 total_loss: 0.3598 time: 0.5446 s/iter data_time: 0.0543 s/iter total_throughput: 1880.35 samples/s lr: 9.61e-05 [09/21 02:22:22] lb.utils.events INFO: eta: 10:53:55 iteration: 303899/375342 consumed_samples: 311193600 total_loss: 0.36 time: 0.5446 s/iter data_time: 0.0521 s/iter total_throughput: 1880.34 samples/s lr: 9.59e-05 [09/21 02:23:17] lb.utils.events INFO: eta: 10:52:58 iteration: 303999/375342 consumed_samples: 311296000 total_loss: 0.3637 time: 0.5446 s/iter data_time: 0.0543 s/iter total_throughput: 1880.34 samples/s lr: 9.57e-05 [09/21 02:24:12] lb.utils.events INFO: eta: 10:52:00 iteration: 304099/375342 consumed_samples: 311398400 total_loss: 0.3632 time: 0.5446 s/iter data_time: 0.0543 s/iter total_throughput: 1880.33 samples/s lr: 9.54e-05 [09/21 02:25:07] lb.utils.events INFO: eta: 10:50:57 iteration: 304199/375342 consumed_samples: 311500800 total_loss: 0.3561 time: 0.5446 s/iter data_time: 0.0506 s/iter total_throughput: 1880.33 samples/s lr: 9.52e-05 [09/21 02:26:01] lb.utils.events INFO: eta: 10:49:38 iteration: 304299/375342 consumed_samples: 311603200 total_loss: 0.3611 time: 0.5446 s/iter data_time: 0.0526 s/iter total_throughput: 1880.33 samples/s lr: 9.50e-05 [09/21 02:26:56] lb.utils.events INFO: eta: 10:48:38 iteration: 304399/375342 consumed_samples: 311705600 total_loss: 0.363 time: 0.5446 s/iter data_time: 0.0525 s/iter total_throughput: 1880.32 samples/s lr: 9.47e-05 [09/21 02:27:51] lb.utils.events INFO: eta: 10:47:27 iteration: 304499/375342 consumed_samples: 311808000 total_loss: 0.3592 time: 0.5446 s/iter data_time: 0.0512 s/iter total_throughput: 1880.32 samples/s lr: 9.45e-05 [09/21 02:28:46] lb.utils.events INFO: eta: 10:46:30 iteration: 304599/375342 consumed_samples: 311910400 total_loss: 0.3599 time: 0.5446 s/iter data_time: 0.0496 s/iter total_throughput: 1880.32 samples/s lr: 9.43e-05 [09/21 02:29:41] lb.utils.events INFO: eta: 10:45:35 iteration: 304699/375342 consumed_samples: 312012800 total_loss: 0.3638 time: 0.5446 s/iter data_time: 0.0529 s/iter total_throughput: 1880.31 samples/s lr: 9.40e-05 [09/21 02:30:35] lb.utils.events INFO: eta: 10:44:31 iteration: 304799/375342 consumed_samples: 312115200 total_loss: 0.3643 time: 0.5446 s/iter data_time: 0.0534 s/iter total_throughput: 1880.31 samples/s lr: 9.38e-05 [09/21 02:31:30] lb.utils.events INFO: eta: 10:43:32 iteration: 304899/375342 consumed_samples: 312217600 total_loss: 0.3594 time: 0.5446 s/iter data_time: 0.0523 s/iter total_throughput: 1880.31 samples/s lr: 9.36e-05 [09/21 02:32:25] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0304999 [09/21 02:32:25] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 02:32:25] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 02:32:30] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0487 s/iter. Inference: 0.2520 s/iter. Eval: 0.0023 s/iter. Total: 0.3030 s/iter. ETA=0:00:11 [09/21 02:32:35] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0643 s/iter. Inference: 0.2683 s/iter. Eval: 0.0023 s/iter. Total: 0.3351 s/iter. ETA=0:00:07 [09/21 02:32:41] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0648 s/iter. Inference: 0.2627 s/iter. Eval: 0.0024 s/iter. Total: 0.3302 s/iter. ETA=0:00:01 [09/21 02:32:43] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 02:32:43] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.281683 (0.000286 s / iter per device, on 8 devices) [09/21 02:32:43] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000230 s / iter per device, on 8 devices) [09/21 02:32:43] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 02:32:43] lb.evaluation.utils INFO: copypaste: Acc@1=79.23 [09/21 02:32:43] lb.evaluation.utils INFO: copypaste: Acc@5=94.432 [09/21 02:32:43] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 79.23000, better than last best score 79.09200 @ iteration 299999. [09/21 02:32:43] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 02:32:43] lb.utils.events INFO: eta: 10:42:33 iteration: 304999/375342 consumed_samples: 312320000 total_loss: 0.358 time: 0.5446 s/iter data_time: 0.0528 s/iter total_throughput: 1880.30 samples/s lr: 9.33e-05 [09/21 02:33:38] lb.utils.events INFO: eta: 10:41:25 iteration: 305099/375342 consumed_samples: 312422400 total_loss: 0.3618 time: 0.5446 s/iter data_time: 0.0554 s/iter total_throughput: 1880.30 samples/s lr: 9.31e-05 [09/21 02:34:33] lb.utils.events INFO: eta: 10:40:33 iteration: 305199/375342 consumed_samples: 312524800 total_loss: 0.3628 time: 0.5446 s/iter data_time: 0.0531 s/iter total_throughput: 1880.30 samples/s lr: 9.29e-05 [09/21 02:35:28] lb.utils.events INFO: eta: 10:39:34 iteration: 305299/375342 consumed_samples: 312627200 total_loss: 0.3612 time: 0.5446 s/iter data_time: 0.0529 s/iter total_throughput: 1880.29 samples/s lr: 9.27e-05 [09/21 02:36:23] lb.utils.events INFO: eta: 10:38:41 iteration: 305399/375342 consumed_samples: 312729600 total_loss: 0.3621 time: 0.5446 s/iter data_time: 0.0512 s/iter total_throughput: 1880.29 samples/s lr: 9.24e-05 [09/21 02:37:18] lb.utils.events INFO: eta: 10:38:11 iteration: 305499/375342 consumed_samples: 312832000 total_loss: 0.3652 time: 0.5446 s/iter data_time: 0.0541 s/iter total_throughput: 1880.28 samples/s lr: 9.22e-05 [09/21 02:38:13] lb.utils.events INFO: eta: 10:37:35 iteration: 305599/375342 consumed_samples: 312934400 total_loss: 0.3651 time: 0.5446 s/iter data_time: 0.0524 s/iter total_throughput: 1880.27 samples/s lr: 9.20e-05 [09/21 02:39:08] lb.utils.events INFO: eta: 10:37:01 iteration: 305699/375342 consumed_samples: 313036800 total_loss: 0.3653 time: 0.5446 s/iter data_time: 0.0536 s/iter total_throughput: 1880.27 samples/s lr: 9.17e-05 [09/21 02:40:03] lb.utils.events INFO: eta: 10:36:31 iteration: 305799/375342 consumed_samples: 313139200 total_loss: 0.3642 time: 0.5446 s/iter data_time: 0.0528 s/iter total_throughput: 1880.26 samples/s lr: 9.15e-05 [09/21 02:40:58] lb.utils.events INFO: eta: 10:36:17 iteration: 305899/375342 consumed_samples: 313241600 total_loss: 0.3635 time: 0.5446 s/iter data_time: 0.0538 s/iter total_throughput: 1880.25 samples/s lr: 9.13e-05 [09/21 02:41:54] lb.utils.events INFO: eta: 10:35:51 iteration: 305999/375342 consumed_samples: 313344000 total_loss: 0.36 time: 0.5446 s/iter data_time: 0.0540 s/iter total_throughput: 1880.24 samples/s lr: 9.11e-05 [09/21 02:42:49] lb.utils.events INFO: eta: 10:35:35 iteration: 306099/375342 consumed_samples: 313446400 total_loss: 0.3564 time: 0.5446 s/iter data_time: 0.0531 s/iter total_throughput: 1880.23 samples/s lr: 9.08e-05 [09/21 02:43:44] lb.utils.events INFO: eta: 10:35:09 iteration: 306199/375342 consumed_samples: 313548800 total_loss: 0.3564 time: 0.5446 s/iter data_time: 0.0526 s/iter total_throughput: 1880.22 samples/s lr: 9.06e-05 [09/21 02:44:40] lb.utils.events INFO: eta: 10:35:02 iteration: 306299/375342 consumed_samples: 313651200 total_loss: 0.3654 time: 0.5446 s/iter data_time: 0.0533 s/iter total_throughput: 1880.21 samples/s lr: 9.04e-05 [09/21 02:45:35] lb.utils.events INFO: eta: 10:34:48 iteration: 306399/375342 consumed_samples: 313753600 total_loss: 0.3706 time: 0.5446 s/iter data_time: 0.0519 s/iter total_throughput: 1880.20 samples/s lr: 9.02e-05 [09/21 02:46:31] lb.utils.events INFO: eta: 10:34:08 iteration: 306499/375342 consumed_samples: 313856000 total_loss: 0.365 time: 0.5446 s/iter data_time: 0.0540 s/iter total_throughput: 1880.19 samples/s lr: 8.99e-05 [09/21 02:47:26] lb.utils.events INFO: eta: 10:33:21 iteration: 306599/375342 consumed_samples: 313958400 total_loss: 0.3552 time: 0.5446 s/iter data_time: 0.0552 s/iter total_throughput: 1880.18 samples/s lr: 8.97e-05 [09/21 02:48:21] lb.utils.events INFO: eta: 10:32:42 iteration: 306699/375342 consumed_samples: 314060800 total_loss: 0.3566 time: 0.5446 s/iter data_time: 0.0528 s/iter total_throughput: 1880.17 samples/s lr: 8.95e-05 [09/21 02:49:17] lb.utils.events INFO: eta: 10:32:10 iteration: 306799/375342 consumed_samples: 314163200 total_loss: 0.3588 time: 0.5446 s/iter data_time: 0.0520 s/iter total_throughput: 1880.16 samples/s lr: 8.93e-05 [09/21 02:50:12] lb.utils.events INFO: eta: 10:31:25 iteration: 306899/375342 consumed_samples: 314265600 total_loss: 0.358 time: 0.5446 s/iter data_time: 0.0512 s/iter total_throughput: 1880.15 samples/s lr: 8.90e-05 [09/21 02:51:08] lb.utils.events INFO: eta: 10:30:32 iteration: 306999/375342 consumed_samples: 314368000 total_loss: 0.357 time: 0.5446 s/iter data_time: 0.0636 s/iter total_throughput: 1880.14 samples/s lr: 8.88e-05 [09/21 02:52:03] lb.utils.events INFO: eta: 10:29:04 iteration: 307099/375342 consumed_samples: 314470400 total_loss: 0.3562 time: 0.5446 s/iter data_time: 0.0533 s/iter total_throughput: 1880.13 samples/s lr: 8.86e-05 [09/21 02:52:58] lb.utils.events INFO: eta: 10:27:44 iteration: 307199/375342 consumed_samples: 314572800 total_loss: 0.3601 time: 0.5446 s/iter data_time: 0.0556 s/iter total_throughput: 1880.13 samples/s lr: 8.84e-05 [09/21 02:53:53] lb.utils.events INFO: eta: 10:26:00 iteration: 307299/375342 consumed_samples: 314675200 total_loss: 0.3617 time: 0.5446 s/iter data_time: 0.0546 s/iter total_throughput: 1880.12 samples/s lr: 8.81e-05 [09/21 02:54:48] lb.utils.events INFO: eta: 10:24:39 iteration: 307399/375342 consumed_samples: 314777600 total_loss: 0.3566 time: 0.5446 s/iter data_time: 0.0531 s/iter total_throughput: 1880.11 samples/s lr: 8.79e-05 [09/21 02:55:43] lb.utils.events INFO: eta: 10:23:25 iteration: 307499/375342 consumed_samples: 314880000 total_loss: 0.3601 time: 0.5446 s/iter data_time: 0.0556 s/iter total_throughput: 1880.11 samples/s lr: 8.77e-05 [09/21 02:56:38] lb.utils.events INFO: eta: 10:22:10 iteration: 307599/375342 consumed_samples: 314982400 total_loss: 0.3631 time: 0.5447 s/iter data_time: 0.0527 s/iter total_throughput: 1880.10 samples/s lr: 8.75e-05 [09/21 02:57:33] lb.utils.events INFO: eta: 10:20:57 iteration: 307699/375342 consumed_samples: 315084800 total_loss: 0.3573 time: 0.5447 s/iter data_time: 0.0541 s/iter total_throughput: 1880.10 samples/s lr: 8.72e-05 [09/21 02:58:28] lb.utils.events INFO: eta: 10:19:38 iteration: 307799/375342 consumed_samples: 315187200 total_loss: 0.3577 time: 0.5447 s/iter data_time: 0.0520 s/iter total_throughput: 1880.09 samples/s lr: 8.70e-05 [09/21 02:59:23] lb.utils.events INFO: eta: 10:18:18 iteration: 307899/375342 consumed_samples: 315289600 total_loss: 0.36 time: 0.5447 s/iter data_time: 0.0512 s/iter total_throughput: 1880.09 samples/s lr: 8.68e-05 [09/21 03:00:18] lb.utils.events INFO: eta: 10:17:06 iteration: 307999/375342 consumed_samples: 315392000 total_loss: 0.357 time: 0.5447 s/iter data_time: 0.0536 s/iter total_throughput: 1880.08 samples/s lr: 8.66e-05 [09/21 03:01:13] lb.utils.events INFO: eta: 10:16:03 iteration: 308099/375342 consumed_samples: 315494400 total_loss: 0.3583 time: 0.5447 s/iter data_time: 0.0516 s/iter total_throughput: 1880.08 samples/s lr: 8.64e-05 [09/21 03:02:08] lb.utils.events INFO: eta: 10:15:14 iteration: 308199/375342 consumed_samples: 315596800 total_loss: 0.3625 time: 0.5447 s/iter data_time: 0.0520 s/iter total_throughput: 1880.07 samples/s lr: 8.61e-05 [09/21 03:03:03] lb.utils.events INFO: eta: 10:14:20 iteration: 308299/375342 consumed_samples: 315699200 total_loss: 0.361 time: 0.5447 s/iter data_time: 0.0480 s/iter total_throughput: 1880.07 samples/s lr: 8.59e-05 [09/21 03:03:58] lb.utils.events INFO: eta: 10:13:29 iteration: 308399/375342 consumed_samples: 315801600 total_loss: 0.3601 time: 0.5447 s/iter data_time: 0.0551 s/iter total_throughput: 1880.06 samples/s lr: 8.57e-05 [09/21 03:04:53] lb.utils.events INFO: eta: 10:12:41 iteration: 308499/375342 consumed_samples: 315904000 total_loss: 0.3623 time: 0.5447 s/iter data_time: 0.0572 s/iter total_throughput: 1880.06 samples/s lr: 8.55e-05 [09/21 03:05:48] lb.utils.events INFO: eta: 10:11:52 iteration: 308599/375342 consumed_samples: 316006400 total_loss: 0.3644 time: 0.5447 s/iter data_time: 0.0469 s/iter total_throughput: 1880.05 samples/s lr: 8.53e-05 [09/21 03:06:43] lb.utils.events INFO: eta: 10:11:11 iteration: 308699/375342 consumed_samples: 316108800 total_loss: 0.3652 time: 0.5447 s/iter data_time: 0.0469 s/iter total_throughput: 1880.04 samples/s lr: 8.50e-05 [09/21 03:07:38] lb.utils.events INFO: eta: 10:10:46 iteration: 308799/375342 consumed_samples: 316211200 total_loss: 0.3597 time: 0.5447 s/iter data_time: 0.0506 s/iter total_throughput: 1880.03 samples/s lr: 8.48e-05 [09/21 03:08:34] lb.utils.events INFO: eta: 10:10:35 iteration: 308899/375342 consumed_samples: 316313600 total_loss: 0.353 time: 0.5447 s/iter data_time: 0.0542 s/iter total_throughput: 1880.02 samples/s lr: 8.46e-05 [09/21 03:09:30] lb.utils.events INFO: eta: 10:09:59 iteration: 308999/375342 consumed_samples: 316416000 total_loss: 0.3468 time: 0.5447 s/iter data_time: 0.0576 s/iter total_throughput: 1880.01 samples/s lr: 8.44e-05 [09/21 03:10:25] lb.utils.events INFO: eta: 10:09:37 iteration: 309099/375342 consumed_samples: 316518400 total_loss: 0.3606 time: 0.5447 s/iter data_time: 0.0505 s/iter total_throughput: 1880.00 samples/s lr: 8.42e-05 [09/21 03:11:21] lb.utils.events INFO: eta: 10:09:17 iteration: 309199/375342 consumed_samples: 316620800 total_loss: 0.362 time: 0.5447 s/iter data_time: 0.0537 s/iter total_throughput: 1879.98 samples/s lr: 8.39e-05 [09/21 03:12:16] lb.utils.events INFO: eta: 10:09:08 iteration: 309299/375342 consumed_samples: 316723200 total_loss: 0.3505 time: 0.5447 s/iter data_time: 0.0532 s/iter total_throughput: 1879.97 samples/s lr: 8.37e-05 [09/21 03:13:12] lb.utils.events INFO: eta: 10:08:42 iteration: 309399/375342 consumed_samples: 316825600 total_loss: 0.3483 time: 0.5447 s/iter data_time: 0.0501 s/iter total_throughput: 1879.96 samples/s lr: 8.35e-05 [09/21 03:14:07] lb.utils.events INFO: eta: 10:08:27 iteration: 309499/375342 consumed_samples: 316928000 total_loss: 0.3559 time: 0.5447 s/iter data_time: 0.0545 s/iter total_throughput: 1879.95 samples/s lr: 8.33e-05 [09/21 03:15:02] lb.utils.events INFO: eta: 10:07:38 iteration: 309599/375342 consumed_samples: 317030400 total_loss: 0.3621 time: 0.5447 s/iter data_time: 0.0491 s/iter total_throughput: 1879.94 samples/s lr: 8.31e-05 [09/21 03:15:58] lb.utils.events INFO: eta: 10:06:37 iteration: 309699/375342 consumed_samples: 317132800 total_loss: 0.3609 time: 0.5447 s/iter data_time: 0.0529 s/iter total_throughput: 1879.93 samples/s lr: 8.29e-05 [09/21 03:16:53] lb.utils.events INFO: eta: 10:05:42 iteration: 309799/375342 consumed_samples: 317235200 total_loss: 0.3578 time: 0.5447 s/iter data_time: 0.0540 s/iter total_throughput: 1879.92 samples/s lr: 8.26e-05 [09/21 03:17:49] lb.utils.events INFO: eta: 10:04:38 iteration: 309899/375342 consumed_samples: 317337600 total_loss: 0.3579 time: 0.5447 s/iter data_time: 0.0537 s/iter total_throughput: 1879.91 samples/s lr: 8.24e-05 [09/21 03:18:44] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0309999 [09/21 03:18:44] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 03:18:44] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 03:18:49] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0594 s/iter. Inference: 0.2606 s/iter. Eval: 0.0024 s/iter. Total: 0.3224 s/iter. ETA=0:00:11 [09/21 03:18:54] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0677 s/iter. Inference: 0.2659 s/iter. Eval: 0.0025 s/iter. Total: 0.3365 s/iter. ETA=0:00:07 [09/21 03:19:00] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0680 s/iter. Inference: 0.2615 s/iter. Eval: 0.0027 s/iter. Total: 0.3327 s/iter. ETA=0:00:01 [09/21 03:19:01] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 03:19:01] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.287993 (0.000286 s / iter per device, on 8 devices) [09/21 03:19:01] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000229 s / iter per device, on 8 devices) [09/21 03:19:01] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 03:19:01] lb.evaluation.utils INFO: copypaste: Acc@1=79.2 [09/21 03:19:01] lb.evaluation.utils INFO: copypaste: Acc@5=94.39 [09/21 03:19:02] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 79.20000, not better than best score 79.23000 @ iteration 304999. [09/21 03:19:02] lb.utils.events INFO: eta: 10:03:20 iteration: 309999/375342 consumed_samples: 317440000 total_loss: 0.3575 time: 0.5447 s/iter data_time: 0.0560 s/iter total_throughput: 1879.90 samples/s lr: 8.22e-05 [09/21 03:19:57] lb.utils.events INFO: eta: 10:01:58 iteration: 310099/375342 consumed_samples: 317542400 total_loss: 0.356 time: 0.5447 s/iter data_time: 0.0546 s/iter total_throughput: 1879.90 samples/s lr: 8.20e-05 [09/21 03:20:52] lb.utils.events INFO: eta: 10:00:49 iteration: 310199/375342 consumed_samples: 317644800 total_loss: 0.3588 time: 0.5447 s/iter data_time: 0.0522 s/iter total_throughput: 1879.89 samples/s lr: 8.18e-05 [09/21 03:21:47] lb.utils.events INFO: eta: 9:59:25 iteration: 310299/375342 consumed_samples: 317747200 total_loss: 0.3621 time: 0.5447 s/iter data_time: 0.0530 s/iter total_throughput: 1879.88 samples/s lr: 8.16e-05 [09/21 03:22:42] lb.utils.events INFO: eta: 9:57:44 iteration: 310399/375342 consumed_samples: 317849600 total_loss: 0.3588 time: 0.5447 s/iter data_time: 0.0522 s/iter total_throughput: 1879.88 samples/s lr: 8.13e-05 [09/21 03:23:37] lb.utils.events INFO: eta: 9:56:19 iteration: 310499/375342 consumed_samples: 317952000 total_loss: 0.3573 time: 0.5447 s/iter data_time: 0.0527 s/iter total_throughput: 1879.87 samples/s lr: 8.11e-05 [09/21 03:24:32] lb.utils.events INFO: eta: 9:54:57 iteration: 310599/375342 consumed_samples: 318054400 total_loss: 0.3561 time: 0.5447 s/iter data_time: 0.0523 s/iter total_throughput: 1879.86 samples/s lr: 8.09e-05 [09/21 03:25:27] lb.utils.events INFO: eta: 9:53:43 iteration: 310699/375342 consumed_samples: 318156800 total_loss: 0.352 time: 0.5447 s/iter data_time: 0.0542 s/iter total_throughput: 1879.86 samples/s lr: 8.07e-05 [09/21 03:26:22] lb.utils.events INFO: eta: 9:52:17 iteration: 310799/375342 consumed_samples: 318259200 total_loss: 0.3562 time: 0.5447 s/iter data_time: 0.0549 s/iter total_throughput: 1879.85 samples/s lr: 8.05e-05 [09/21 03:27:17] lb.utils.events INFO: eta: 9:50:43 iteration: 310899/375342 consumed_samples: 318361600 total_loss: 0.3594 time: 0.5447 s/iter data_time: 0.0551 s/iter total_throughput: 1879.85 samples/s lr: 8.03e-05 [09/21 03:28:12] lb.utils.events INFO: eta: 9:49:33 iteration: 310999/375342 consumed_samples: 318464000 total_loss: 0.358 time: 0.5447 s/iter data_time: 0.0549 s/iter total_throughput: 1879.84 samples/s lr: 8.01e-05 [09/21 03:29:07] lb.utils.events INFO: eta: 9:48:23 iteration: 311099/375342 consumed_samples: 318566400 total_loss: 0.3552 time: 0.5447 s/iter data_time: 0.0533 s/iter total_throughput: 1879.84 samples/s lr: 7.99e-05 [09/21 03:30:02] lb.utils.events INFO: eta: 9:47:20 iteration: 311199/375342 consumed_samples: 318668800 total_loss: 0.356 time: 0.5447 s/iter data_time: 0.0525 s/iter total_throughput: 1879.83 samples/s lr: 7.96e-05 [09/21 03:30:57] lb.utils.events INFO: eta: 9:46:34 iteration: 311299/375342 consumed_samples: 318771200 total_loss: 0.3559 time: 0.5447 s/iter data_time: 0.0558 s/iter total_throughput: 1879.82 samples/s lr: 7.94e-05 [09/21 03:31:52] lb.utils.events INFO: eta: 9:45:54 iteration: 311399/375342 consumed_samples: 318873600 total_loss: 0.3547 time: 0.5447 s/iter data_time: 0.0529 s/iter total_throughput: 1879.82 samples/s lr: 7.92e-05 [09/21 03:32:48] lb.utils.events INFO: eta: 9:45:01 iteration: 311499/375342 consumed_samples: 318976000 total_loss: 0.3598 time: 0.5447 s/iter data_time: 0.0525 s/iter total_throughput: 1879.81 samples/s lr: 7.90e-05 [09/21 03:33:42] lb.utils.events INFO: eta: 9:44:25 iteration: 311599/375342 consumed_samples: 319078400 total_loss: 0.3634 time: 0.5447 s/iter data_time: 0.0529 s/iter total_throughput: 1879.81 samples/s lr: 7.88e-05 [09/21 03:34:37] lb.utils.events INFO: eta: 9:43:37 iteration: 311699/375342 consumed_samples: 319180800 total_loss: 0.3651 time: 0.5447 s/iter data_time: 0.0527 s/iter total_throughput: 1879.80 samples/s lr: 7.86e-05 [09/21 03:35:32] lb.utils.events INFO: eta: 9:42:46 iteration: 311799/375342 consumed_samples: 319283200 total_loss: 0.3601 time: 0.5447 s/iter data_time: 0.0500 s/iter total_throughput: 1879.80 samples/s lr: 7.84e-05 [09/21 03:36:27] lb.utils.events INFO: eta: 9:41:55 iteration: 311899/375342 consumed_samples: 319385600 total_loss: 0.3569 time: 0.5447 s/iter data_time: 0.0435 s/iter total_throughput: 1879.79 samples/s lr: 7.82e-05 [09/21 03:37:23] lb.utils.events INFO: eta: 9:41:16 iteration: 311999/375342 consumed_samples: 319488000 total_loss: 0.3635 time: 0.5447 s/iter data_time: 0.0465 s/iter total_throughput: 1879.78 samples/s lr: 7.80e-05 [09/21 03:38:18] lb.utils.events INFO: eta: 9:40:51 iteration: 312099/375342 consumed_samples: 319590400 total_loss: 0.3591 time: 0.5447 s/iter data_time: 0.0497 s/iter total_throughput: 1879.78 samples/s lr: 7.77e-05 [09/21 03:39:13] lb.utils.events INFO: eta: 9:40:11 iteration: 312199/375342 consumed_samples: 319692800 total_loss: 0.3601 time: 0.5447 s/iter data_time: 0.0508 s/iter total_throughput: 1879.76 samples/s lr: 7.75e-05 [09/21 03:40:09] lb.utils.events INFO: eta: 9:39:35 iteration: 312299/375342 consumed_samples: 319795200 total_loss: 0.3628 time: 0.5448 s/iter data_time: 0.0510 s/iter total_throughput: 1879.75 samples/s lr: 7.73e-05 [09/21 03:41:04] lb.utils.events INFO: eta: 9:38:55 iteration: 312399/375342 consumed_samples: 319897600 total_loss: 0.3576 time: 0.5448 s/iter data_time: 0.0518 s/iter total_throughput: 1879.74 samples/s lr: 7.71e-05 [09/21 03:42:00] lb.utils.events INFO: eta: 9:38:14 iteration: 312499/375342 consumed_samples: 320000000 total_loss: 0.3579 time: 0.5448 s/iter data_time: 0.0530 s/iter total_throughput: 1879.73 samples/s lr: 7.69e-05 [09/21 03:42:55] lb.utils.events INFO: eta: 9:37:37 iteration: 312599/375342 consumed_samples: 320102400 total_loss: 0.3588 time: 0.5448 s/iter data_time: 0.0510 s/iter total_throughput: 1879.73 samples/s lr: 7.67e-05 [09/21 03:43:50] lb.utils.events INFO: eta: 9:36:55 iteration: 312699/375342 consumed_samples: 320204800 total_loss: 0.3577 time: 0.5448 s/iter data_time: 0.0502 s/iter total_throughput: 1879.72 samples/s lr: 7.65e-05 [09/21 03:44:45] lb.utils.events INFO: eta: 9:36:15 iteration: 312799/375342 consumed_samples: 320307200 total_loss: 0.3575 time: 0.5448 s/iter data_time: 0.0524 s/iter total_throughput: 1879.71 samples/s lr: 7.63e-05 [09/21 03:45:40] lb.utils.events INFO: eta: 9:35:27 iteration: 312899/375342 consumed_samples: 320409600 total_loss: 0.356 time: 0.5448 s/iter data_time: 0.0540 s/iter total_throughput: 1879.70 samples/s lr: 7.61e-05 [09/21 03:46:36] lb.utils.events INFO: eta: 9:34:31 iteration: 312999/375342 consumed_samples: 320512000 total_loss: 0.3525 time: 0.5448 s/iter data_time: 0.0519 s/iter total_throughput: 1879.70 samples/s lr: 7.59e-05 [09/21 03:47:31] lb.utils.events INFO: eta: 9:33:17 iteration: 313099/375342 consumed_samples: 320614400 total_loss: 0.3553 time: 0.5448 s/iter data_time: 0.0519 s/iter total_throughput: 1879.69 samples/s lr: 7.57e-05 [09/21 03:48:26] lb.utils.events INFO: eta: 9:31:54 iteration: 313199/375342 consumed_samples: 320716800 total_loss: 0.3556 time: 0.5448 s/iter data_time: 0.0504 s/iter total_throughput: 1879.68 samples/s lr: 7.55e-05 [09/21 03:49:21] lb.utils.events INFO: eta: 9:30:18 iteration: 313299/375342 consumed_samples: 320819200 total_loss: 0.3507 time: 0.5448 s/iter data_time: 0.0525 s/iter total_throughput: 1879.68 samples/s lr: 7.53e-05 [09/21 03:50:15] lb.utils.events INFO: eta: 9:28:41 iteration: 313399/375342 consumed_samples: 320921600 total_loss: 0.354 time: 0.5448 s/iter data_time: 0.0527 s/iter total_throughput: 1879.68 samples/s lr: 7.51e-05 [09/21 03:51:10] lb.utils.events INFO: eta: 9:27:10 iteration: 313499/375342 consumed_samples: 321024000 total_loss: 0.3541 time: 0.5448 s/iter data_time: 0.0520 s/iter total_throughput: 1879.67 samples/s lr: 7.48e-05 [09/21 03:52:05] lb.utils.events INFO: eta: 9:25:20 iteration: 313599/375342 consumed_samples: 321126400 total_loss: 0.3492 time: 0.5448 s/iter data_time: 0.0506 s/iter total_throughput: 1879.67 samples/s lr: 7.46e-05 [09/21 03:52:59] lb.utils.events INFO: eta: 9:23:47 iteration: 313699/375342 consumed_samples: 321228800 total_loss: 0.3492 time: 0.5448 s/iter data_time: 0.0523 s/iter total_throughput: 1879.67 samples/s lr: 7.44e-05 [09/21 03:53:54] lb.utils.events INFO: eta: 9:22:10 iteration: 313799/375342 consumed_samples: 321331200 total_loss: 0.3459 time: 0.5448 s/iter data_time: 0.0518 s/iter total_throughput: 1879.67 samples/s lr: 7.42e-05 [09/21 03:54:49] lb.utils.events INFO: eta: 9:20:45 iteration: 313899/375342 consumed_samples: 321433600 total_loss: 0.3537 time: 0.5448 s/iter data_time: 0.0571 s/iter total_throughput: 1879.67 samples/s lr: 7.40e-05 [09/21 03:55:43] lb.utils.events INFO: eta: 9:19:28 iteration: 313999/375342 consumed_samples: 321536000 total_loss: 0.358 time: 0.5448 s/iter data_time: 0.0490 s/iter total_throughput: 1879.66 samples/s lr: 7.38e-05 [09/21 03:56:38] lb.utils.events INFO: eta: 9:18:13 iteration: 314099/375342 consumed_samples: 321638400 total_loss: 0.359 time: 0.5448 s/iter data_time: 0.0539 s/iter total_throughput: 1879.66 samples/s lr: 7.36e-05 [09/21 03:57:33] lb.utils.events INFO: eta: 9:17:16 iteration: 314199/375342 consumed_samples: 321740800 total_loss: 0.3615 time: 0.5448 s/iter data_time: 0.0505 s/iter total_throughput: 1879.66 samples/s lr: 7.34e-05 [09/21 03:58:28] lb.utils.events INFO: eta: 9:16:22 iteration: 314299/375342 consumed_samples: 321843200 total_loss: 0.3566 time: 0.5448 s/iter data_time: 0.0527 s/iter total_throughput: 1879.65 samples/s lr: 7.32e-05 [09/21 03:59:23] lb.utils.events INFO: eta: 9:15:28 iteration: 314399/375342 consumed_samples: 321945600 total_loss: 0.3565 time: 0.5448 s/iter data_time: 0.0525 s/iter total_throughput: 1879.65 samples/s lr: 7.30e-05 [09/21 04:00:18] lb.utils.events INFO: eta: 9:14:50 iteration: 314499/375342 consumed_samples: 322048000 total_loss: 0.3538 time: 0.5448 s/iter data_time: 0.0554 s/iter total_throughput: 1879.64 samples/s lr: 7.28e-05 [09/21 04:01:13] lb.utils.events INFO: eta: 9:14:15 iteration: 314599/375342 consumed_samples: 322150400 total_loss: 0.3523 time: 0.5448 s/iter data_time: 0.0550 s/iter total_throughput: 1879.64 samples/s lr: 7.26e-05 [09/21 04:02:08] lb.utils.events INFO: eta: 9:13:59 iteration: 314699/375342 consumed_samples: 322252800 total_loss: 0.3563 time: 0.5448 s/iter data_time: 0.0518 s/iter total_throughput: 1879.63 samples/s lr: 7.24e-05 [09/21 04:03:03] lb.utils.events INFO: eta: 9:13:45 iteration: 314799/375342 consumed_samples: 322355200 total_loss: 0.3555 time: 0.5448 s/iter data_time: 0.0521 s/iter total_throughput: 1879.63 samples/s lr: 7.22e-05 [09/21 04:03:58] lb.utils.events INFO: eta: 9:13:24 iteration: 314899/375342 consumed_samples: 322457600 total_loss: 0.3547 time: 0.5448 s/iter data_time: 0.0549 s/iter total_throughput: 1879.62 samples/s lr: 7.20e-05 [09/21 04:04:53] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0314999 [09/21 04:04:54] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 04:04:54] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 04:04:59] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0647 s/iter. Inference: 0.2503 s/iter. Eval: 0.0021 s/iter. Total: 0.3171 s/iter. ETA=0:00:11 [09/21 04:05:04] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0844 s/iter. Inference: 0.2511 s/iter. Eval: 0.0023 s/iter. Total: 0.3382 s/iter. ETA=0:00:07 [09/21 04:05:09] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0804 s/iter. Inference: 0.2505 s/iter. Eval: 0.0024 s/iter. Total: 0.3337 s/iter. ETA=0:00:02 [09/21 04:05:11] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 04:05:11] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.319469 (0.000286 s / iter per device, on 8 devices) [09/21 04:05:11] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000219 s / iter per device, on 8 devices) [09/21 04:05:11] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 04:05:11] lb.evaluation.utils INFO: copypaste: Acc@1=79.39 [09/21 04:05:11] lb.evaluation.utils INFO: copypaste: Acc@5=94.394 [09/21 04:05:11] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 79.39000, better than last best score 79.23000 @ iteration 304999. [09/21 04:05:11] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 04:05:12] lb.utils.events INFO: eta: 9:12:46 iteration: 314999/375342 consumed_samples: 322560000 total_loss: 0.3496 time: 0.5448 s/iter data_time: 0.0485 s/iter total_throughput: 1879.61 samples/s lr: 7.18e-05 [09/21 04:06:07] lb.utils.events INFO: eta: 9:12:08 iteration: 315099/375342 consumed_samples: 322662400 total_loss: 0.3546 time: 0.5448 s/iter data_time: 0.0548 s/iter total_throughput: 1879.61 samples/s lr: 7.16e-05 [09/21 04:07:02] lb.utils.events INFO: eta: 9:11:27 iteration: 315199/375342 consumed_samples: 322764800 total_loss: 0.356 time: 0.5448 s/iter data_time: 0.0498 s/iter total_throughput: 1879.60 samples/s lr: 7.14e-05 [09/21 04:07:57] lb.utils.events INFO: eta: 9:10:50 iteration: 315299/375342 consumed_samples: 322867200 total_loss: 0.3567 time: 0.5448 s/iter data_time: 0.0448 s/iter total_throughput: 1879.60 samples/s lr: 7.12e-05 [09/21 04:08:52] lb.utils.events INFO: eta: 9:10:08 iteration: 315399/375342 consumed_samples: 322969600 total_loss: 0.3577 time: 0.5448 s/iter data_time: 0.0475 s/iter total_throughput: 1879.59 samples/s lr: 7.10e-05 [09/21 04:09:47] lb.utils.events INFO: eta: 9:09:36 iteration: 315499/375342 consumed_samples: 323072000 total_loss: 0.354 time: 0.5448 s/iter data_time: 0.0507 s/iter total_throughput: 1879.58 samples/s lr: 7.08e-05 [09/21 04:10:43] lb.utils.events INFO: eta: 9:08:56 iteration: 315599/375342 consumed_samples: 323174400 total_loss: 0.3503 time: 0.5448 s/iter data_time: 0.0529 s/iter total_throughput: 1879.57 samples/s lr: 7.06e-05 [09/21 04:11:38] lb.utils.events INFO: eta: 9:08:07 iteration: 315699/375342 consumed_samples: 323276800 total_loss: 0.3486 time: 0.5448 s/iter data_time: 0.0530 s/iter total_throughput: 1879.56 samples/s lr: 7.04e-05 [09/21 04:12:33] lb.utils.events INFO: eta: 9:07:15 iteration: 315799/375342 consumed_samples: 323379200 total_loss: 0.3565 time: 0.5448 s/iter data_time: 0.0498 s/iter total_throughput: 1879.56 samples/s lr: 7.02e-05 [09/21 04:13:28] lb.utils.events INFO: eta: 9:06:36 iteration: 315899/375342 consumed_samples: 323481600 total_loss: 0.3554 time: 0.5448 s/iter data_time: 0.0486 s/iter total_throughput: 1879.55 samples/s lr: 7.00e-05 [09/21 04:14:23] lb.utils.events INFO: eta: 9:05:41 iteration: 315999/375342 consumed_samples: 323584000 total_loss: 0.353 time: 0.5448 s/iter data_time: 0.0499 s/iter total_throughput: 1879.54 samples/s lr: 6.98e-05 [09/21 04:15:18] lb.utils.events INFO: eta: 9:04:44 iteration: 316099/375342 consumed_samples: 323686400 total_loss: 0.3525 time: 0.5448 s/iter data_time: 0.0533 s/iter total_throughput: 1879.54 samples/s lr: 6.96e-05 [09/21 04:16:13] lb.utils.events INFO: eta: 9:03:35 iteration: 316199/375342 consumed_samples: 323788800 total_loss: 0.3564 time: 0.5448 s/iter data_time: 0.0527 s/iter total_throughput: 1879.53 samples/s lr: 6.94e-05 [09/21 04:17:08] lb.utils.events INFO: eta: 9:02:33 iteration: 316299/375342 consumed_samples: 323891200 total_loss: 0.3577 time: 0.5448 s/iter data_time: 0.0517 s/iter total_throughput: 1879.53 samples/s lr: 6.92e-05 [09/21 04:18:03] lb.utils.events INFO: eta: 9:01:26 iteration: 316399/375342 consumed_samples: 323993600 total_loss: 0.3539 time: 0.5448 s/iter data_time: 0.0517 s/iter total_throughput: 1879.52 samples/s lr: 6.90e-05 [09/21 04:18:58] lb.utils.events INFO: eta: 8:59:58 iteration: 316499/375342 consumed_samples: 324096000 total_loss: 0.353 time: 0.5448 s/iter data_time: 0.0543 s/iter total_throughput: 1879.52 samples/s lr: 6.88e-05 [09/21 04:19:53] lb.utils.events INFO: eta: 8:58:35 iteration: 316599/375342 consumed_samples: 324198400 total_loss: 0.3615 time: 0.5448 s/iter data_time: 0.0523 s/iter total_throughput: 1879.52 samples/s lr: 6.86e-05 [09/21 04:20:47] lb.utils.events INFO: eta: 8:57:07 iteration: 316699/375342 consumed_samples: 324300800 total_loss: 0.3607 time: 0.5448 s/iter data_time: 0.0523 s/iter total_throughput: 1879.52 samples/s lr: 6.84e-05 [09/21 04:21:42] lb.utils.events INFO: eta: 8:55:30 iteration: 316799/375342 consumed_samples: 324403200 total_loss: 0.3558 time: 0.5448 s/iter data_time: 0.0519 s/iter total_throughput: 1879.51 samples/s lr: 6.82e-05 [09/21 04:22:37] lb.utils.events INFO: eta: 8:53:54 iteration: 316899/375342 consumed_samples: 324505600 total_loss: 0.3577 time: 0.5448 s/iter data_time: 0.0507 s/iter total_throughput: 1879.51 samples/s lr: 6.81e-05 [09/21 04:23:31] lb.utils.events INFO: eta: 8:52:16 iteration: 316999/375342 consumed_samples: 324608000 total_loss: 0.3525 time: 0.5448 s/iter data_time: 0.0526 s/iter total_throughput: 1879.51 samples/s lr: 6.79e-05 [09/21 04:24:26] lb.utils.events INFO: eta: 8:50:54 iteration: 317099/375342 consumed_samples: 324710400 total_loss: 0.3484 time: 0.5448 s/iter data_time: 0.0472 s/iter total_throughput: 1879.51 samples/s lr: 6.77e-05 [09/21 04:25:20] lb.utils.events INFO: eta: 8:49:31 iteration: 317199/375342 consumed_samples: 324812800 total_loss: 0.3494 time: 0.5448 s/iter data_time: 0.0482 s/iter total_throughput: 1879.51 samples/s lr: 6.75e-05 [09/21 04:26:15] lb.utils.events INFO: eta: 8:48:21 iteration: 317299/375342 consumed_samples: 324915200 total_loss: 0.3566 time: 0.5448 s/iter data_time: 0.0483 s/iter total_throughput: 1879.50 samples/s lr: 6.73e-05 [09/21 04:27:11] lb.utils.events INFO: eta: 8:47:13 iteration: 317399/375342 consumed_samples: 325017600 total_loss: 0.3549 time: 0.5448 s/iter data_time: 0.0569 s/iter total_throughput: 1879.49 samples/s lr: 6.71e-05 [09/21 04:28:06] lb.utils.events INFO: eta: 8:46:23 iteration: 317499/375342 consumed_samples: 325120000 total_loss: 0.3599 time: 0.5448 s/iter data_time: 0.0531 s/iter total_throughput: 1879.49 samples/s lr: 6.69e-05 [09/21 04:29:01] lb.utils.events INFO: eta: 8:45:42 iteration: 317599/375342 consumed_samples: 325222400 total_loss: 0.3567 time: 0.5448 s/iter data_time: 0.0558 s/iter total_throughput: 1879.48 samples/s lr: 6.67e-05 [09/21 04:29:56] lb.utils.events INFO: eta: 8:45:04 iteration: 317699/375342 consumed_samples: 325324800 total_loss: 0.3482 time: 0.5448 s/iter data_time: 0.0531 s/iter total_throughput: 1879.48 samples/s lr: 6.65e-05 [09/21 04:30:51] lb.utils.events INFO: eta: 8:44:19 iteration: 317799/375342 consumed_samples: 325427200 total_loss: 0.3499 time: 0.5448 s/iter data_time: 0.0521 s/iter total_throughput: 1879.47 samples/s lr: 6.63e-05 [09/21 04:31:46] lb.utils.events INFO: eta: 8:43:47 iteration: 317899/375342 consumed_samples: 325529600 total_loss: 0.3568 time: 0.5448 s/iter data_time: 0.0530 s/iter total_throughput: 1879.47 samples/s lr: 6.61e-05 [09/21 04:32:41] lb.utils.events INFO: eta: 8:43:23 iteration: 317999/375342 consumed_samples: 325632000 total_loss: 0.3595 time: 0.5448 s/iter data_time: 0.0512 s/iter total_throughput: 1879.46 samples/s lr: 6.59e-05 [09/21 04:33:36] lb.utils.events INFO: eta: 8:42:57 iteration: 318099/375342 consumed_samples: 325734400 total_loss: 0.3579 time: 0.5448 s/iter data_time: 0.0495 s/iter total_throughput: 1879.46 samples/s lr: 6.57e-05 [09/21 04:34:31] lb.utils.events INFO: eta: 8:42:25 iteration: 318199/375342 consumed_samples: 325836800 total_loss: 0.353 time: 0.5448 s/iter data_time: 0.0522 s/iter total_throughput: 1879.45 samples/s lr: 6.55e-05 [09/21 04:35:26] lb.utils.events INFO: eta: 8:41:51 iteration: 318299/375342 consumed_samples: 325939200 total_loss: 0.344 time: 0.5448 s/iter data_time: 0.0481 s/iter total_throughput: 1879.45 samples/s lr: 6.54e-05 [09/21 04:36:21] lb.utils.events INFO: eta: 8:41:08 iteration: 318399/375342 consumed_samples: 326041600 total_loss: 0.3415 time: 0.5448 s/iter data_time: 0.0547 s/iter total_throughput: 1879.44 samples/s lr: 6.52e-05 [09/21 04:37:16] lb.utils.events INFO: eta: 8:40:15 iteration: 318499/375342 consumed_samples: 326144000 total_loss: 0.3495 time: 0.5448 s/iter data_time: 0.0545 s/iter total_throughput: 1879.44 samples/s lr: 6.50e-05 [09/21 04:38:11] lb.utils.events INFO: eta: 8:39:23 iteration: 318599/375342 consumed_samples: 326246400 total_loss: 0.358 time: 0.5448 s/iter data_time: 0.0542 s/iter total_throughput: 1879.43 samples/s lr: 6.48e-05 [09/21 04:39:06] lb.utils.events INFO: eta: 8:38:27 iteration: 318699/375342 consumed_samples: 326348800 total_loss: 0.3581 time: 0.5448 s/iter data_time: 0.0538 s/iter total_throughput: 1879.43 samples/s lr: 6.46e-05 [09/21 04:40:01] lb.utils.events INFO: eta: 8:37:32 iteration: 318799/375342 consumed_samples: 326451200 total_loss: 0.3552 time: 0.5448 s/iter data_time: 0.0507 s/iter total_throughput: 1879.42 samples/s lr: 6.44e-05 [09/21 04:40:56] lb.utils.events INFO: eta: 8:36:40 iteration: 318899/375342 consumed_samples: 326553600 total_loss: 0.3543 time: 0.5448 s/iter data_time: 0.0545 s/iter total_throughput: 1879.42 samples/s lr: 6.42e-05 [09/21 04:41:50] lb.utils.events INFO: eta: 8:35:30 iteration: 318999/375342 consumed_samples: 326656000 total_loss: 0.3575 time: 0.5449 s/iter data_time: 0.0528 s/iter total_throughput: 1879.42 samples/s lr: 6.40e-05 [09/21 04:42:45] lb.utils.events INFO: eta: 8:34:34 iteration: 319099/375342 consumed_samples: 326758400 total_loss: 0.3592 time: 0.5449 s/iter data_time: 0.0540 s/iter total_throughput: 1879.41 samples/s lr: 6.38e-05 [09/21 04:43:40] lb.utils.events INFO: eta: 8:33:38 iteration: 319199/375342 consumed_samples: 326860800 total_loss: 0.3588 time: 0.5449 s/iter data_time: 0.0528 s/iter total_throughput: 1879.41 samples/s lr: 6.37e-05 [09/21 04:44:35] lb.utils.events INFO: eta: 8:32:43 iteration: 319299/375342 consumed_samples: 326963200 total_loss: 0.3577 time: 0.5449 s/iter data_time: 0.0503 s/iter total_throughput: 1879.40 samples/s lr: 6.35e-05 [09/21 04:45:30] lb.utils.events INFO: eta: 8:31:38 iteration: 319399/375342 consumed_samples: 327065600 total_loss: 0.3518 time: 0.5449 s/iter data_time: 0.0557 s/iter total_throughput: 1879.40 samples/s lr: 6.33e-05 [09/21 04:46:25] lb.utils.events INFO: eta: 8:30:44 iteration: 319499/375342 consumed_samples: 327168000 total_loss: 0.3519 time: 0.5449 s/iter data_time: 0.0520 s/iter total_throughput: 1879.40 samples/s lr: 6.31e-05 [09/21 04:47:20] lb.utils.events INFO: eta: 8:29:47 iteration: 319599/375342 consumed_samples: 327270400 total_loss: 0.3524 time: 0.5449 s/iter data_time: 0.0533 s/iter total_throughput: 1879.39 samples/s lr: 6.29e-05 [09/21 04:48:15] lb.utils.events INFO: eta: 8:29:04 iteration: 319699/375342 consumed_samples: 327372800 total_loss: 0.3524 time: 0.5449 s/iter data_time: 0.0559 s/iter total_throughput: 1879.39 samples/s lr: 6.27e-05 [09/21 04:49:09] lb.utils.events INFO: eta: 8:28:10 iteration: 319799/375342 consumed_samples: 327475200 total_loss: 0.3643 time: 0.5449 s/iter data_time: 0.0551 s/iter total_throughput: 1879.38 samples/s lr: 6.25e-05 [09/21 04:50:04] lb.utils.events INFO: eta: 8:27:14 iteration: 319899/375342 consumed_samples: 327577600 total_loss: 0.3535 time: 0.5449 s/iter data_time: 0.0538 s/iter total_throughput: 1879.38 samples/s lr: 6.23e-05 [09/21 04:50:59] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0319999 [09/21 04:51:00] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 04:51:00] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 04:51:05] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0639 s/iter. Inference: 0.2481 s/iter. Eval: 0.0021 s/iter. Total: 0.3141 s/iter. ETA=0:00:11 [09/21 04:51:10] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0827 s/iter. Inference: 0.2513 s/iter. Eval: 0.0025 s/iter. Total: 0.3368 s/iter. ETA=0:00:07 [09/21 04:51:15] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0768 s/iter. Inference: 0.2529 s/iter. Eval: 0.0025 s/iter. Total: 0.3327 s/iter. ETA=0:00:01 [09/21 04:51:17] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 04:51:17] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.298728 (0.000286 s / iter per device, on 8 devices) [09/21 04:51:17] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000221 s / iter per device, on 8 devices) [09/21 04:51:17] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 04:51:17] lb.evaluation.utils INFO: copypaste: Acc@1=79.508 [09/21 04:51:17] lb.evaluation.utils INFO: copypaste: Acc@5=94.568 [09/21 04:51:17] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 79.50800, better than last best score 79.39000 @ iteration 314999. [09/21 04:51:17] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 04:51:18] lb.utils.events INFO: eta: 8:26:31 iteration: 319999/375342 consumed_samples: 327680000 total_loss: 0.3497 time: 0.5449 s/iter data_time: 0.0524 s/iter total_throughput: 1879.37 samples/s lr: 6.22e-05 [09/21 04:52:13] lb.utils.events INFO: eta: 8:25:37 iteration: 320099/375342 consumed_samples: 327782400 total_loss: 0.352 time: 0.5449 s/iter data_time: 0.0540 s/iter total_throughput: 1879.37 samples/s lr: 6.20e-05 [09/21 04:53:08] lb.utils.events INFO: eta: 8:24:49 iteration: 320199/375342 consumed_samples: 327884800 total_loss: 0.3536 time: 0.5449 s/iter data_time: 0.0549 s/iter total_throughput: 1879.36 samples/s lr: 6.18e-05 [09/21 04:54:03] lb.utils.events INFO: eta: 8:24:14 iteration: 320299/375342 consumed_samples: 327987200 total_loss: 0.3595 time: 0.5449 s/iter data_time: 0.0518 s/iter total_throughput: 1879.36 samples/s lr: 6.16e-05 [09/21 04:54:58] lb.utils.events INFO: eta: 8:23:38 iteration: 320399/375342 consumed_samples: 328089600 total_loss: 0.3609 time: 0.5449 s/iter data_time: 0.0519 s/iter total_throughput: 1879.35 samples/s lr: 6.14e-05 [09/21 04:55:53] lb.utils.events INFO: eta: 8:22:54 iteration: 320499/375342 consumed_samples: 328192000 total_loss: 0.3577 time: 0.5449 s/iter data_time: 0.0513 s/iter total_throughput: 1879.34 samples/s lr: 6.12e-05 [09/21 04:56:48] lb.utils.events INFO: eta: 8:22:11 iteration: 320599/375342 consumed_samples: 328294400 total_loss: 0.36 time: 0.5449 s/iter data_time: 0.0485 s/iter total_throughput: 1879.34 samples/s lr: 6.11e-05 [09/21 04:57:44] lb.utils.events INFO: eta: 8:21:18 iteration: 320699/375342 consumed_samples: 328396800 total_loss: 0.3615 time: 0.5449 s/iter data_time: 0.0529 s/iter total_throughput: 1879.33 samples/s lr: 6.09e-05 [09/21 04:58:39] lb.utils.events INFO: eta: 8:20:37 iteration: 320799/375342 consumed_samples: 328499200 total_loss: 0.3602 time: 0.5449 s/iter data_time: 0.0582 s/iter total_throughput: 1879.32 samples/s lr: 6.07e-05 [09/21 04:59:34] lb.utils.events INFO: eta: 8:19:47 iteration: 320899/375342 consumed_samples: 328601600 total_loss: 0.3534 time: 0.5449 s/iter data_time: 0.0541 s/iter total_throughput: 1879.31 samples/s lr: 6.05e-05 [09/21 05:00:29] lb.utils.events INFO: eta: 8:18:41 iteration: 320999/375342 consumed_samples: 328704000 total_loss: 0.345 time: 0.5449 s/iter data_time: 0.0536 s/iter total_throughput: 1879.31 samples/s lr: 6.03e-05 [09/21 05:01:24] lb.utils.events INFO: eta: 8:17:34 iteration: 321099/375342 consumed_samples: 328806400 total_loss: 0.3505 time: 0.5449 s/iter data_time: 0.0524 s/iter total_throughput: 1879.31 samples/s lr: 6.01e-05 [09/21 05:02:19] lb.utils.events INFO: eta: 8:16:35 iteration: 321199/375342 consumed_samples: 328908800 total_loss: 0.3557 time: 0.5449 s/iter data_time: 0.0537 s/iter total_throughput: 1879.30 samples/s lr: 6.00e-05 [09/21 05:03:14] lb.utils.events INFO: eta: 8:15:34 iteration: 321299/375342 consumed_samples: 329011200 total_loss: 0.3553 time: 0.5449 s/iter data_time: 0.0510 s/iter total_throughput: 1879.30 samples/s lr: 5.98e-05 [09/21 05:04:09] lb.utils.events INFO: eta: 8:14:29 iteration: 321399/375342 consumed_samples: 329113600 total_loss: 0.3558 time: 0.5449 s/iter data_time: 0.0534 s/iter total_throughput: 1879.29 samples/s lr: 5.96e-05 [09/21 05:05:03] lb.utils.events INFO: eta: 8:13:24 iteration: 321499/375342 consumed_samples: 329216000 total_loss: 0.358 time: 0.5449 s/iter data_time: 0.0525 s/iter total_throughput: 1879.29 samples/s lr: 5.94e-05 [09/21 05:05:58] lb.utils.events INFO: eta: 8:12:27 iteration: 321599/375342 consumed_samples: 329318400 total_loss: 0.3508 time: 0.5449 s/iter data_time: 0.0510 s/iter total_throughput: 1879.28 samples/s lr: 5.92e-05 [09/21 05:06:53] lb.utils.events INFO: eta: 8:11:18 iteration: 321699/375342 consumed_samples: 329420800 total_loss: 0.3484 time: 0.5449 s/iter data_time: 0.0543 s/iter total_throughput: 1879.28 samples/s lr: 5.91e-05 [09/21 05:07:48] lb.utils.events INFO: eta: 8:10:06 iteration: 321799/375342 consumed_samples: 329523200 total_loss: 0.3548 time: 0.5449 s/iter data_time: 0.0510 s/iter total_throughput: 1879.28 samples/s lr: 5.89e-05 [09/21 05:08:43] lb.utils.events INFO: eta: 8:09:09 iteration: 321899/375342 consumed_samples: 329625600 total_loss: 0.3477 time: 0.5449 s/iter data_time: 0.0527 s/iter total_throughput: 1879.27 samples/s lr: 5.87e-05 [09/21 05:09:38] lb.utils.events INFO: eta: 8:08:18 iteration: 321999/375342 consumed_samples: 329728000 total_loss: 0.3457 time: 0.5449 s/iter data_time: 0.0514 s/iter total_throughput: 1879.27 samples/s lr: 5.85e-05 [09/21 05:10:33] lb.utils.events INFO: eta: 8:07:37 iteration: 322099/375342 consumed_samples: 329830400 total_loss: 0.3524 time: 0.5449 s/iter data_time: 0.0489 s/iter total_throughput: 1879.26 samples/s lr: 5.83e-05 [09/21 05:11:28] lb.utils.events INFO: eta: 8:06:36 iteration: 322199/375342 consumed_samples: 329932800 total_loss: 0.3516 time: 0.5449 s/iter data_time: 0.0494 s/iter total_throughput: 1879.26 samples/s lr: 5.82e-05 [09/21 05:12:23] lb.utils.events INFO: eta: 8:05:33 iteration: 322299/375342 consumed_samples: 330035200 total_loss: 0.3535 time: 0.5449 s/iter data_time: 0.0546 s/iter total_throughput: 1879.26 samples/s lr: 5.80e-05 [09/21 05:13:18] lb.utils.events INFO: eta: 8:04:32 iteration: 322399/375342 consumed_samples: 330137600 total_loss: 0.3514 time: 0.5449 s/iter data_time: 0.0519 s/iter total_throughput: 1879.25 samples/s lr: 5.78e-05 [09/21 05:14:12] lb.utils.events INFO: eta: 8:03:25 iteration: 322499/375342 consumed_samples: 330240000 total_loss: 0.3516 time: 0.5449 s/iter data_time: 0.0522 s/iter total_throughput: 1879.25 samples/s lr: 5.76e-05 [09/21 05:15:07] lb.utils.events INFO: eta: 8:02:20 iteration: 322599/375342 consumed_samples: 330342400 total_loss: 0.3538 time: 0.5449 s/iter data_time: 0.0545 s/iter total_throughput: 1879.25 samples/s lr: 5.75e-05 [09/21 05:16:02] lb.utils.events INFO: eta: 8:01:24 iteration: 322699/375342 consumed_samples: 330444800 total_loss: 0.3538 time: 0.5449 s/iter data_time: 0.0550 s/iter total_throughput: 1879.24 samples/s lr: 5.73e-05 [09/21 05:16:57] lb.utils.events INFO: eta: 8:00:32 iteration: 322799/375342 consumed_samples: 330547200 total_loss: 0.3597 time: 0.5449 s/iter data_time: 0.0538 s/iter total_throughput: 1879.24 samples/s lr: 5.71e-05 [09/21 05:17:52] lb.utils.events INFO: eta: 7:59:43 iteration: 322899/375342 consumed_samples: 330649600 total_loss: 0.3575 time: 0.5449 s/iter data_time: 0.0531 s/iter total_throughput: 1879.23 samples/s lr: 5.69e-05 [09/21 05:18:47] lb.utils.events INFO: eta: 7:58:48 iteration: 322999/375342 consumed_samples: 330752000 total_loss: 0.3534 time: 0.5449 s/iter data_time: 0.0528 s/iter total_throughput: 1879.23 samples/s lr: 5.67e-05 [09/21 05:19:42] lb.utils.events INFO: eta: 7:58:07 iteration: 323099/375342 consumed_samples: 330854400 total_loss: 0.3521 time: 0.5449 s/iter data_time: 0.0472 s/iter total_throughput: 1879.22 samples/s lr: 5.66e-05 [09/21 05:20:37] lb.utils.events INFO: eta: 7:57:17 iteration: 323199/375342 consumed_samples: 330956800 total_loss: 0.351 time: 0.5449 s/iter data_time: 0.0529 s/iter total_throughput: 1879.21 samples/s lr: 5.64e-05 [09/21 05:21:32] lb.utils.events INFO: eta: 7:56:40 iteration: 323299/375342 consumed_samples: 331059200 total_loss: 0.3551 time: 0.5449 s/iter data_time: 0.0542 s/iter total_throughput: 1879.21 samples/s lr: 5.62e-05 [09/21 05:22:28] lb.utils.events INFO: eta: 7:55:51 iteration: 323399/375342 consumed_samples: 331161600 total_loss: 0.3569 time: 0.5449 s/iter data_time: 0.0533 s/iter total_throughput: 1879.20 samples/s lr: 5.60e-05 [09/21 05:23:23] lb.utils.events INFO: eta: 7:55:19 iteration: 323499/375342 consumed_samples: 331264000 total_loss: 0.3532 time: 0.5449 s/iter data_time: 0.0541 s/iter total_throughput: 1879.20 samples/s lr: 5.59e-05 [09/21 05:24:18] lb.utils.events INFO: eta: 7:54:48 iteration: 323599/375342 consumed_samples: 331366400 total_loss: 0.3535 time: 0.5449 s/iter data_time: 0.0511 s/iter total_throughput: 1879.19 samples/s lr: 5.57e-05 [09/21 05:25:14] lb.utils.events INFO: eta: 7:54:28 iteration: 323699/375342 consumed_samples: 331468800 total_loss: 0.3543 time: 0.5449 s/iter data_time: 0.0532 s/iter total_throughput: 1879.18 samples/s lr: 5.55e-05 [09/21 05:26:09] lb.utils.events INFO: eta: 7:53:56 iteration: 323799/375342 consumed_samples: 331571200 total_loss: 0.3597 time: 0.5449 s/iter data_time: 0.0549 s/iter total_throughput: 1879.17 samples/s lr: 5.54e-05 [09/21 05:27:04] lb.utils.events INFO: eta: 7:53:21 iteration: 323899/375342 consumed_samples: 331673600 total_loss: 0.3585 time: 0.5449 s/iter data_time: 0.0538 s/iter total_throughput: 1879.16 samples/s lr: 5.52e-05 [09/21 05:28:00] lb.utils.events INFO: eta: 7:52:44 iteration: 323999/375342 consumed_samples: 331776000 total_loss: 0.3552 time: 0.5449 s/iter data_time: 0.0523 s/iter total_throughput: 1879.15 samples/s lr: 5.50e-05 [09/21 05:28:55] lb.utils.events INFO: eta: 7:51:57 iteration: 324099/375342 consumed_samples: 331878400 total_loss: 0.3526 time: 0.5449 s/iter data_time: 0.0524 s/iter total_throughput: 1879.14 samples/s lr: 5.48e-05 [09/21 05:29:51] lb.utils.events INFO: eta: 7:51:20 iteration: 324199/375342 consumed_samples: 331980800 total_loss: 0.3546 time: 0.5449 s/iter data_time: 0.0522 s/iter total_throughput: 1879.13 samples/s lr: 5.47e-05 [09/21 05:30:46] lb.utils.events INFO: eta: 7:50:18 iteration: 324299/375342 consumed_samples: 332083200 total_loss: 0.3535 time: 0.5449 s/iter data_time: 0.0552 s/iter total_throughput: 1879.12 samples/s lr: 5.45e-05 [09/21 05:31:41] lb.utils.events INFO: eta: 7:49:18 iteration: 324399/375342 consumed_samples: 332185600 total_loss: 0.3531 time: 0.5449 s/iter data_time: 0.0501 s/iter total_throughput: 1879.11 samples/s lr: 5.43e-05 [09/21 05:32:36] lb.utils.events INFO: eta: 7:48:20 iteration: 324499/375342 consumed_samples: 332288000 total_loss: 0.3534 time: 0.5449 s/iter data_time: 0.0515 s/iter total_throughput: 1879.11 samples/s lr: 5.41e-05 [09/21 05:33:31] lb.utils.events INFO: eta: 7:46:50 iteration: 324599/375342 consumed_samples: 332390400 total_loss: 0.3527 time: 0.5449 s/iter data_time: 0.0554 s/iter total_throughput: 1879.11 samples/s lr: 5.40e-05 [09/21 05:34:26] lb.utils.events INFO: eta: 7:45:32 iteration: 324699/375342 consumed_samples: 332492800 total_loss: 0.3529 time: 0.5449 s/iter data_time: 0.0495 s/iter total_throughput: 1879.10 samples/s lr: 5.38e-05 [09/21 05:35:21] lb.utils.events INFO: eta: 7:44:03 iteration: 324799/375342 consumed_samples: 332595200 total_loss: 0.3514 time: 0.5449 s/iter data_time: 0.0531 s/iter total_throughput: 1879.10 samples/s lr: 5.36e-05 [09/21 05:36:16] lb.utils.events INFO: eta: 7:42:43 iteration: 324899/375342 consumed_samples: 332697600 total_loss: 0.3475 time: 0.5449 s/iter data_time: 0.0553 s/iter total_throughput: 1879.09 samples/s lr: 5.35e-05 [09/21 05:37:11] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0324999 [09/21 05:37:11] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 05:37:11] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 05:37:16] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0435 s/iter. Inference: 0.2572 s/iter. Eval: 0.0022 s/iter. Total: 0.3030 s/iter. ETA=0:00:11 [09/21 05:37:21] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0629 s/iter. Inference: 0.2674 s/iter. Eval: 0.0025 s/iter. Total: 0.3333 s/iter. ETA=0:00:07 [09/21 05:37:26] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0626 s/iter. Inference: 0.2625 s/iter. Eval: 0.0024 s/iter. Total: 0.3283 s/iter. ETA=0:00:01 [09/21 05:37:28] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 05:37:28] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.181341 (0.000284 s / iter per device, on 8 devices) [09/21 05:37:28] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000229 s / iter per device, on 8 devices) [09/21 05:37:28] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 05:37:28] lb.evaluation.utils INFO: copypaste: Acc@1=79.464 [09/21 05:37:28] lb.evaluation.utils INFO: copypaste: Acc@5=94.61 [09/21 05:37:28] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 79.46400, not better than best score 79.50800 @ iteration 319999. [09/21 05:37:28] lb.utils.events INFO: eta: 7:41:33 iteration: 324999/375342 consumed_samples: 332800000 total_loss: 0.3506 time: 0.5449 s/iter data_time: 0.0537 s/iter total_throughput: 1879.09 samples/s lr: 5.33e-05 [09/21 05:38:23] lb.utils.events INFO: eta: 7:40:12 iteration: 325099/375342 consumed_samples: 332902400 total_loss: 0.3534 time: 0.5449 s/iter data_time: 0.0523 s/iter total_throughput: 1879.09 samples/s lr: 5.31e-05 [09/21 05:39:18] lb.utils.events INFO: eta: 7:38:58 iteration: 325199/375342 consumed_samples: 333004800 total_loss: 0.3528 time: 0.5449 s/iter data_time: 0.0561 s/iter total_throughput: 1879.08 samples/s lr: 5.30e-05 [09/21 05:40:13] lb.utils.events INFO: eta: 7:38:00 iteration: 325299/375342 consumed_samples: 333107200 total_loss: 0.3554 time: 0.5449 s/iter data_time: 0.0580 s/iter total_throughput: 1879.08 samples/s lr: 5.28e-05 [09/21 05:41:08] lb.utils.events INFO: eta: 7:37:15 iteration: 325399/375342 consumed_samples: 333209600 total_loss: 0.3514 time: 0.5450 s/iter data_time: 0.0540 s/iter total_throughput: 1879.07 samples/s lr: 5.26e-05 [09/21 05:42:03] lb.utils.events INFO: eta: 7:36:19 iteration: 325499/375342 consumed_samples: 333312000 total_loss: 0.3513 time: 0.5450 s/iter data_time: 0.0531 s/iter total_throughput: 1879.07 samples/s lr: 5.25e-05 [09/21 05:42:58] lb.utils.events INFO: eta: 7:35:28 iteration: 325599/375342 consumed_samples: 333414400 total_loss: 0.3535 time: 0.5450 s/iter data_time: 0.0540 s/iter total_throughput: 1879.06 samples/s lr: 5.23e-05 [09/21 05:43:53] lb.utils.events INFO: eta: 7:34:34 iteration: 325699/375342 consumed_samples: 333516800 total_loss: 0.3528 time: 0.5450 s/iter data_time: 0.0538 s/iter total_throughput: 1879.06 samples/s lr: 5.21e-05 [09/21 05:44:48] lb.utils.events INFO: eta: 7:33:43 iteration: 325799/375342 consumed_samples: 333619200 total_loss: 0.3495 time: 0.5450 s/iter data_time: 0.0579 s/iter total_throughput: 1879.05 samples/s lr: 5.20e-05 [09/21 05:45:43] lb.utils.events INFO: eta: 7:32:51 iteration: 325899/375342 consumed_samples: 333721600 total_loss: 0.3509 time: 0.5450 s/iter data_time: 0.0543 s/iter total_throughput: 1879.05 samples/s lr: 5.18e-05 [09/21 05:46:38] lb.utils.events INFO: eta: 7:32:06 iteration: 325999/375342 consumed_samples: 333824000 total_loss: 0.3554 time: 0.5450 s/iter data_time: 0.0534 s/iter total_throughput: 1879.04 samples/s lr: 5.16e-05 [09/21 05:47:34] lb.utils.events INFO: eta: 7:31:28 iteration: 326099/375342 consumed_samples: 333926400 total_loss: 0.353 time: 0.5450 s/iter data_time: 0.0515 s/iter total_throughput: 1879.03 samples/s lr: 5.15e-05 [09/21 05:48:29] lb.utils.events INFO: eta: 7:30:49 iteration: 326199/375342 consumed_samples: 334028800 total_loss: 0.3529 time: 0.5450 s/iter data_time: 0.0563 s/iter total_throughput: 1879.02 samples/s lr: 5.13e-05 [09/21 05:49:25] lb.utils.events INFO: eta: 7:30:27 iteration: 326299/375342 consumed_samples: 334131200 total_loss: 0.3516 time: 0.5450 s/iter data_time: 0.0574 s/iter total_throughput: 1879.01 samples/s lr: 5.11e-05 [09/21 05:50:20] lb.utils.events INFO: eta: 7:30:09 iteration: 326399/375342 consumed_samples: 334233600 total_loss: 0.3536 time: 0.5450 s/iter data_time: 0.0501 s/iter total_throughput: 1879.00 samples/s lr: 5.10e-05 [09/21 05:51:16] lb.utils.events INFO: eta: 7:29:48 iteration: 326499/375342 consumed_samples: 334336000 total_loss: 0.3538 time: 0.5450 s/iter data_time: 0.0519 s/iter total_throughput: 1878.99 samples/s lr: 5.08e-05 [09/21 05:52:12] lb.utils.events INFO: eta: 7:29:18 iteration: 326599/375342 consumed_samples: 334438400 total_loss: 0.3469 time: 0.5450 s/iter data_time: 0.0524 s/iter total_throughput: 1878.98 samples/s lr: 5.06e-05 [09/21 05:53:07] lb.utils.events INFO: eta: 7:29:01 iteration: 326699/375342 consumed_samples: 334540800 total_loss: 0.3503 time: 0.5450 s/iter data_time: 0.0535 s/iter total_throughput: 1878.96 samples/s lr: 5.05e-05 [09/21 05:54:03] lb.utils.events INFO: eta: 7:28:53 iteration: 326799/375342 consumed_samples: 334643200 total_loss: 0.3519 time: 0.5450 s/iter data_time: 0.0541 s/iter total_throughput: 1878.95 samples/s lr: 5.03e-05 [09/21 05:54:59] lb.utils.events INFO: eta: 7:28:28 iteration: 326899/375342 consumed_samples: 334745600 total_loss: 0.3507 time: 0.5450 s/iter data_time: 0.0557 s/iter total_throughput: 1878.94 samples/s lr: 5.01e-05 [09/21 05:55:54] lb.utils.events INFO: eta: 7:27:43 iteration: 326999/375342 consumed_samples: 334848000 total_loss: 0.356 time: 0.5450 s/iter data_time: 0.0505 s/iter total_throughput: 1878.93 samples/s lr: 5.00e-05 [09/21 05:56:49] lb.utils.events INFO: eta: 7:26:43 iteration: 327099/375342 consumed_samples: 334950400 total_loss: 0.3568 time: 0.5450 s/iter data_time: 0.0446 s/iter total_throughput: 1878.92 samples/s lr: 4.98e-05 [09/21 05:57:44] lb.utils.events INFO: eta: 7:25:32 iteration: 327199/375342 consumed_samples: 335052800 total_loss: 0.3499 time: 0.5450 s/iter data_time: 0.0517 s/iter total_throughput: 1878.92 samples/s lr: 4.96e-05 [09/21 05:58:40] lb.utils.events INFO: eta: 7:24:22 iteration: 327299/375342 consumed_samples: 335155200 total_loss: 0.3476 time: 0.5450 s/iter data_time: 0.0514 s/iter total_throughput: 1878.91 samples/s lr: 4.95e-05 [09/21 05:59:35] lb.utils.events INFO: eta: 7:23:15 iteration: 327399/375342 consumed_samples: 335257600 total_loss: 0.3538 time: 0.5450 s/iter data_time: 0.0516 s/iter total_throughput: 1878.90 samples/s lr: 4.93e-05 [09/21 06:00:31] lb.utils.events INFO: eta: 7:22:08 iteration: 327499/375342 consumed_samples: 335360000 total_loss: 0.3597 time: 0.5450 s/iter data_time: 0.0525 s/iter total_throughput: 1878.89 samples/s lr: 4.92e-05 [09/21 06:01:26] lb.utils.events INFO: eta: 7:20:51 iteration: 327599/375342 consumed_samples: 335462400 total_loss: 0.3564 time: 0.5450 s/iter data_time: 0.0519 s/iter total_throughput: 1878.88 samples/s lr: 4.90e-05 [09/21 06:02:22] lb.utils.events INFO: eta: 7:19:26 iteration: 327699/375342 consumed_samples: 335564800 total_loss: 0.3504 time: 0.5450 s/iter data_time: 0.0559 s/iter total_throughput: 1878.87 samples/s lr: 4.88e-05 [09/21 06:03:17] lb.utils.events INFO: eta: 7:17:53 iteration: 327799/375342 consumed_samples: 335667200 total_loss: 0.3476 time: 0.5450 s/iter data_time: 0.0571 s/iter total_throughput: 1878.86 samples/s lr: 4.87e-05 [09/21 06:04:11] lb.utils.events INFO: eta: 7:16:30 iteration: 327899/375342 consumed_samples: 335769600 total_loss: 0.3539 time: 0.5450 s/iter data_time: 0.0514 s/iter total_throughput: 1878.86 samples/s lr: 4.85e-05 [09/21 06:05:06] lb.utils.events INFO: eta: 7:15:09 iteration: 327999/375342 consumed_samples: 335872000 total_loss: 0.3589 time: 0.5450 s/iter data_time: 0.0545 s/iter total_throughput: 1878.86 samples/s lr: 4.84e-05 [09/21 06:06:01] lb.utils.events INFO: eta: 7:14:00 iteration: 328099/375342 consumed_samples: 335974400 total_loss: 0.3565 time: 0.5450 s/iter data_time: 0.0550 s/iter total_throughput: 1878.85 samples/s lr: 4.82e-05 [09/21 06:06:56] lb.utils.events INFO: eta: 7:13:01 iteration: 328199/375342 consumed_samples: 336076800 total_loss: 0.3583 time: 0.5450 s/iter data_time: 0.0489 s/iter total_throughput: 1878.85 samples/s lr: 4.80e-05 [09/21 06:07:51] lb.utils.events INFO: eta: 7:11:42 iteration: 328299/375342 consumed_samples: 336179200 total_loss: 0.3574 time: 0.5450 s/iter data_time: 0.0525 s/iter total_throughput: 1878.84 samples/s lr: 4.79e-05 [09/21 06:08:46] lb.utils.events INFO: eta: 7:10:37 iteration: 328399/375342 consumed_samples: 336281600 total_loss: 0.3542 time: 0.5450 s/iter data_time: 0.0547 s/iter total_throughput: 1878.84 samples/s lr: 4.77e-05 [09/21 06:09:41] lb.utils.events INFO: eta: 7:09:33 iteration: 328499/375342 consumed_samples: 336384000 total_loss: 0.3528 time: 0.5450 s/iter data_time: 0.0537 s/iter total_throughput: 1878.83 samples/s lr: 4.76e-05 [09/21 06:10:37] lb.utils.events INFO: eta: 7:08:33 iteration: 328599/375342 consumed_samples: 336486400 total_loss: 0.3511 time: 0.5450 s/iter data_time: 0.0530 s/iter total_throughput: 1878.82 samples/s lr: 4.74e-05 [09/21 06:11:32] lb.utils.events INFO: eta: 7:07:40 iteration: 328699/375342 consumed_samples: 336588800 total_loss: 0.3568 time: 0.5450 s/iter data_time: 0.0529 s/iter total_throughput: 1878.82 samples/s lr: 4.72e-05 [09/21 06:12:27] lb.utils.events INFO: eta: 7:06:50 iteration: 328799/375342 consumed_samples: 336691200 total_loss: 0.3583 time: 0.5450 s/iter data_time: 0.0526 s/iter total_throughput: 1878.81 samples/s lr: 4.71e-05 [09/21 06:13:22] lb.utils.events INFO: eta: 7:06:03 iteration: 328899/375342 consumed_samples: 336793600 total_loss: 0.3546 time: 0.5450 s/iter data_time: 0.0523 s/iter total_throughput: 1878.81 samples/s lr: 4.69e-05 [09/21 06:14:17] lb.utils.events INFO: eta: 7:05:16 iteration: 328999/375342 consumed_samples: 336896000 total_loss: 0.3574 time: 0.5450 s/iter data_time: 0.0526 s/iter total_throughput: 1878.80 samples/s lr: 4.68e-05 [09/21 06:15:12] lb.utils.events INFO: eta: 7:04:29 iteration: 329099/375342 consumed_samples: 336998400 total_loss: 0.3593 time: 0.5450 s/iter data_time: 0.0522 s/iter total_throughput: 1878.80 samples/s lr: 4.66e-05 [09/21 06:16:07] lb.utils.events INFO: eta: 7:03:42 iteration: 329199/375342 consumed_samples: 337100800 total_loss: 0.3524 time: 0.5450 s/iter data_time: 0.0536 s/iter total_throughput: 1878.79 samples/s lr: 4.65e-05 [09/21 06:17:03] lb.utils.events INFO: eta: 7:03:13 iteration: 329299/375342 consumed_samples: 337203200 total_loss: 0.3538 time: 0.5450 s/iter data_time: 0.0588 s/iter total_throughput: 1878.78 samples/s lr: 4.63e-05 [09/21 06:17:58] lb.utils.events INFO: eta: 7:02:26 iteration: 329399/375342 consumed_samples: 337305600 total_loss: 0.3532 time: 0.5450 s/iter data_time: 0.0478 s/iter total_throughput: 1878.77 samples/s lr: 4.61e-05 [09/21 06:18:53] lb.utils.events INFO: eta: 7:01:46 iteration: 329499/375342 consumed_samples: 337408000 total_loss: 0.3491 time: 0.5450 s/iter data_time: 0.0491 s/iter total_throughput: 1878.76 samples/s lr: 4.60e-05 [09/21 06:19:49] lb.utils.events INFO: eta: 7:01:04 iteration: 329599/375342 consumed_samples: 337510400 total_loss: 0.3521 time: 0.5450 s/iter data_time: 0.0550 s/iter total_throughput: 1878.75 samples/s lr: 4.58e-05 [09/21 06:20:45] lb.utils.events INFO: eta: 7:00:25 iteration: 329699/375342 consumed_samples: 337612800 total_loss: 0.3604 time: 0.5450 s/iter data_time: 0.0529 s/iter total_throughput: 1878.74 samples/s lr: 4.57e-05 [09/21 06:21:40] lb.utils.events INFO: eta: 6:59:54 iteration: 329799/375342 consumed_samples: 337715200 total_loss: 0.3558 time: 0.5450 s/iter data_time: 0.0563 s/iter total_throughput: 1878.73 samples/s lr: 4.55e-05 [09/21 06:22:36] lb.utils.events INFO: eta: 6:59:23 iteration: 329899/375342 consumed_samples: 337817600 total_loss: 0.3482 time: 0.5451 s/iter data_time: 0.0548 s/iter total_throughput: 1878.72 samples/s lr: 4.54e-05 [09/21 06:23:31] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0329999 [09/21 06:23:32] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 06:23:32] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 06:23:37] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0502 s/iter. Inference: 0.2602 s/iter. Eval: 0.0031 s/iter. Total: 0.3135 s/iter. ETA=0:00:11 [09/21 06:23:42] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0631 s/iter. Inference: 0.2663 s/iter. Eval: 0.0032 s/iter. Total: 0.3333 s/iter. ETA=0:00:07 [09/21 06:23:47] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0698 s/iter. Inference: 0.2600 s/iter. Eval: 0.0033 s/iter. Total: 0.3340 s/iter. ETA=0:00:02 [09/21 06:23:49] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 06:23:49] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.360516 (0.000287 s / iter per device, on 8 devices) [09/21 06:23:49] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000226 s / iter per device, on 8 devices) [09/21 06:23:49] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 06:23:49] lb.evaluation.utils INFO: copypaste: Acc@1=79.686 [09/21 06:23:49] lb.evaluation.utils INFO: copypaste: Acc@5=94.592 [09/21 06:23:49] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 79.68600, better than last best score 79.50800 @ iteration 319999. [09/21 06:23:49] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 06:23:50] lb.utils.events INFO: eta: 6:58:47 iteration: 329999/375342 consumed_samples: 337920000 total_loss: 0.3522 time: 0.5451 s/iter data_time: 0.0501 s/iter total_throughput: 1878.71 samples/s lr: 4.52e-05 [09/21 06:24:45] lb.utils.events INFO: eta: 6:58:26 iteration: 330099/375342 consumed_samples: 338022400 total_loss: 0.3562 time: 0.5451 s/iter data_time: 0.0491 s/iter total_throughput: 1878.70 samples/s lr: 4.51e-05 [09/21 06:25:41] lb.utils.events INFO: eta: 6:57:33 iteration: 330199/375342 consumed_samples: 338124800 total_loss: 0.3503 time: 0.5451 s/iter data_time: 0.0442 s/iter total_throughput: 1878.69 samples/s lr: 4.49e-05 [09/21 06:26:36] lb.utils.events INFO: eta: 6:56:40 iteration: 330299/375342 consumed_samples: 338227200 total_loss: 0.351 time: 0.5451 s/iter data_time: 0.0487 s/iter total_throughput: 1878.68 samples/s lr: 4.48e-05 [09/21 06:27:32] lb.utils.events INFO: eta: 6:55:44 iteration: 330399/375342 consumed_samples: 338329600 total_loss: 0.3517 time: 0.5451 s/iter data_time: 0.0524 s/iter total_throughput: 1878.67 samples/s lr: 4.46e-05 [09/21 06:28:27] lb.utils.events INFO: eta: 6:54:48 iteration: 330499/375342 consumed_samples: 338432000 total_loss: 0.3482 time: 0.5451 s/iter data_time: 0.0499 s/iter total_throughput: 1878.66 samples/s lr: 4.45e-05 [09/21 06:29:22] lb.utils.events INFO: eta: 6:53:37 iteration: 330599/375342 consumed_samples: 338534400 total_loss: 0.3514 time: 0.5451 s/iter data_time: 0.0524 s/iter total_throughput: 1878.65 samples/s lr: 4.43e-05 [09/21 06:30:18] lb.utils.events INFO: eta: 6:52:28 iteration: 330699/375342 consumed_samples: 338636800 total_loss: 0.3553 time: 0.5451 s/iter data_time: 0.0548 s/iter total_throughput: 1878.65 samples/s lr: 4.42e-05 [09/21 06:31:13] lb.utils.events INFO: eta: 6:51:25 iteration: 330799/375342 consumed_samples: 338739200 total_loss: 0.3521 time: 0.5451 s/iter data_time: 0.0504 s/iter total_throughput: 1878.64 samples/s lr: 4.40e-05 [09/21 06:32:08] lb.utils.events INFO: eta: 6:50:10 iteration: 330899/375342 consumed_samples: 338841600 total_loss: 0.3456 time: 0.5451 s/iter data_time: 0.0535 s/iter total_throughput: 1878.63 samples/s lr: 4.39e-05 [09/21 06:33:04] lb.utils.events INFO: eta: 6:49:08 iteration: 330999/375342 consumed_samples: 338944000 total_loss: 0.3485 time: 0.5451 s/iter data_time: 0.0540 s/iter total_throughput: 1878.62 samples/s lr: 4.37e-05 [09/21 06:33:59] lb.utils.events INFO: eta: 6:47:47 iteration: 331099/375342 consumed_samples: 339046400 total_loss: 0.3516 time: 0.5451 s/iter data_time: 0.0520 s/iter total_throughput: 1878.62 samples/s lr: 4.36e-05 [09/21 06:34:54] lb.utils.events INFO: eta: 6:46:41 iteration: 331199/375342 consumed_samples: 339148800 total_loss: 0.347 time: 0.5451 s/iter data_time: 0.0538 s/iter total_throughput: 1878.61 samples/s lr: 4.34e-05 [09/21 06:35:49] lb.utils.events INFO: eta: 6:45:19 iteration: 331299/375342 consumed_samples: 339251200 total_loss: 0.3499 time: 0.5451 s/iter data_time: 0.0528 s/iter total_throughput: 1878.60 samples/s lr: 4.33e-05 [09/21 06:36:44] lb.utils.events INFO: eta: 6:44:07 iteration: 331399/375342 consumed_samples: 339353600 total_loss: 0.3504 time: 0.5451 s/iter data_time: 0.0565 s/iter total_throughput: 1878.60 samples/s lr: 4.31e-05 [09/21 06:37:39] lb.utils.events INFO: eta: 6:42:38 iteration: 331499/375342 consumed_samples: 339456000 total_loss: 0.3493 time: 0.5451 s/iter data_time: 0.0506 s/iter total_throughput: 1878.59 samples/s lr: 4.30e-05 [09/21 06:38:34] lb.utils.events INFO: eta: 6:41:35 iteration: 331599/375342 consumed_samples: 339558400 total_loss: 0.3421 time: 0.5451 s/iter data_time: 0.0521 s/iter total_throughput: 1878.59 samples/s lr: 4.28e-05 [09/21 06:39:29] lb.utils.events INFO: eta: 6:40:33 iteration: 331699/375342 consumed_samples: 339660800 total_loss: 0.3449 time: 0.5451 s/iter data_time: 0.0522 s/iter total_throughput: 1878.58 samples/s lr: 4.27e-05 [09/21 06:40:24] lb.utils.events INFO: eta: 6:39:34 iteration: 331799/375342 consumed_samples: 339763200 total_loss: 0.3567 time: 0.5451 s/iter data_time: 0.0555 s/iter total_throughput: 1878.58 samples/s lr: 4.25e-05 [09/21 06:41:19] lb.utils.events INFO: eta: 6:38:39 iteration: 331899/375342 consumed_samples: 339865600 total_loss: 0.3502 time: 0.5451 s/iter data_time: 0.0551 s/iter total_throughput: 1878.57 samples/s lr: 4.24e-05 [09/21 06:42:15] lb.utils.events INFO: eta: 6:37:49 iteration: 331999/375342 consumed_samples: 339968000 total_loss: 0.3485 time: 0.5451 s/iter data_time: 0.0506 s/iter total_throughput: 1878.56 samples/s lr: 4.22e-05 [09/21 06:43:10] lb.utils.events INFO: eta: 6:37:02 iteration: 332099/375342 consumed_samples: 340070400 total_loss: 0.3553 time: 0.5451 s/iter data_time: 0.0510 s/iter total_throughput: 1878.55 samples/s lr: 4.21e-05 [09/21 06:44:05] lb.utils.events INFO: eta: 6:36:13 iteration: 332199/375342 consumed_samples: 340172800 total_loss: 0.3597 time: 0.5451 s/iter data_time: 0.0534 s/iter total_throughput: 1878.55 samples/s lr: 4.19e-05 [09/21 06:45:00] lb.utils.events INFO: eta: 6:35:17 iteration: 332299/375342 consumed_samples: 340275200 total_loss: 0.3534 time: 0.5451 s/iter data_time: 0.0494 s/iter total_throughput: 1878.54 samples/s lr: 4.18e-05 [09/21 06:45:55] lb.utils.events INFO: eta: 6:34:23 iteration: 332399/375342 consumed_samples: 340377600 total_loss: 0.3507 time: 0.5451 s/iter data_time: 0.0494 s/iter total_throughput: 1878.54 samples/s lr: 4.16e-05 [09/21 06:46:50] lb.utils.events INFO: eta: 6:33:36 iteration: 332499/375342 consumed_samples: 340480000 total_loss: 0.3513 time: 0.5451 s/iter data_time: 0.0532 s/iter total_throughput: 1878.53 samples/s lr: 4.15e-05 [09/21 06:47:45] lb.utils.events INFO: eta: 6:32:52 iteration: 332599/375342 consumed_samples: 340582400 total_loss: 0.3505 time: 0.5451 s/iter data_time: 0.0481 s/iter total_throughput: 1878.53 samples/s lr: 4.13e-05 [09/21 06:48:41] lb.utils.events INFO: eta: 6:32:04 iteration: 332699/375342 consumed_samples: 340684800 total_loss: 0.3444 time: 0.5451 s/iter data_time: 0.0420 s/iter total_throughput: 1878.52 samples/s lr: 4.12e-05 [09/21 06:49:36] lb.utils.events INFO: eta: 6:31:23 iteration: 332799/375342 consumed_samples: 340787200 total_loss: 0.3447 time: 0.5451 s/iter data_time: 0.0491 s/iter total_throughput: 1878.51 samples/s lr: 4.11e-05 [09/21 06:50:32] lb.utils.events INFO: eta: 6:30:41 iteration: 332899/375342 consumed_samples: 340889600 total_loss: 0.3476 time: 0.5451 s/iter data_time: 0.0435 s/iter total_throughput: 1878.50 samples/s lr: 4.09e-05 [09/21 06:51:27] lb.utils.events INFO: eta: 6:29:47 iteration: 332999/375342 consumed_samples: 340992000 total_loss: 0.3511 time: 0.5451 s/iter data_time: 0.0550 s/iter total_throughput: 1878.49 samples/s lr: 4.08e-05 [09/21 06:52:23] lb.utils.events INFO: eta: 6:28:58 iteration: 333099/375342 consumed_samples: 341094400 total_loss: 0.3545 time: 0.5451 s/iter data_time: 0.0553 s/iter total_throughput: 1878.48 samples/s lr: 4.06e-05 [09/21 06:53:18] lb.utils.events INFO: eta: 6:28:24 iteration: 333199/375342 consumed_samples: 341196800 total_loss: 0.3493 time: 0.5451 s/iter data_time: 0.0504 s/iter total_throughput: 1878.47 samples/s lr: 4.05e-05 [09/21 06:54:14] lb.utils.events INFO: eta: 6:27:45 iteration: 333299/375342 consumed_samples: 341299200 total_loss: 0.3534 time: 0.5451 s/iter data_time: 0.0461 s/iter total_throughput: 1878.46 samples/s lr: 4.03e-05 [09/21 06:55:09] lb.utils.events INFO: eta: 6:27:09 iteration: 333399/375342 consumed_samples: 341401600 total_loss: 0.3547 time: 0.5451 s/iter data_time: 0.0490 s/iter total_throughput: 1878.45 samples/s lr: 4.02e-05 [09/21 06:56:05] lb.utils.events INFO: eta: 6:26:47 iteration: 333499/375342 consumed_samples: 341504000 total_loss: 0.3465 time: 0.5451 s/iter data_time: 0.0516 s/iter total_throughput: 1878.44 samples/s lr: 4.00e-05 [09/21 06:57:00] lb.utils.events INFO: eta: 6:26:15 iteration: 333599/375342 consumed_samples: 341606400 total_loss: 0.3444 time: 0.5451 s/iter data_time: 0.0496 s/iter total_throughput: 1878.43 samples/s lr: 3.99e-05 [09/21 06:57:56] lb.utils.events INFO: eta: 6:25:21 iteration: 333699/375342 consumed_samples: 341708800 total_loss: 0.3467 time: 0.5451 s/iter data_time: 0.0539 s/iter total_throughput: 1878.42 samples/s lr: 3.98e-05 [09/21 06:58:51] lb.utils.events INFO: eta: 6:24:27 iteration: 333799/375342 consumed_samples: 341811200 total_loss: 0.3495 time: 0.5451 s/iter data_time: 0.0491 s/iter total_throughput: 1878.41 samples/s lr: 3.96e-05 [09/21 06:59:47] lb.utils.events INFO: eta: 6:23:25 iteration: 333899/375342 consumed_samples: 341913600 total_loss: 0.3477 time: 0.5451 s/iter data_time: 0.0552 s/iter total_throughput: 1878.40 samples/s lr: 3.95e-05 [09/21 07:00:42] lb.utils.events INFO: eta: 6:22:09 iteration: 333999/375342 consumed_samples: 342016000 total_loss: 0.3564 time: 0.5451 s/iter data_time: 0.0532 s/iter total_throughput: 1878.39 samples/s lr: 3.93e-05 [09/21 07:01:37] lb.utils.events INFO: eta: 6:20:56 iteration: 334099/375342 consumed_samples: 342118400 total_loss: 0.3587 time: 0.5451 s/iter data_time: 0.0541 s/iter total_throughput: 1878.39 samples/s lr: 3.92e-05 [09/21 07:02:32] lb.utils.events INFO: eta: 6:19:39 iteration: 334199/375342 consumed_samples: 342220800 total_loss: 0.3496 time: 0.5452 s/iter data_time: 0.0541 s/iter total_throughput: 1878.38 samples/s lr: 3.91e-05 [09/21 07:03:27] lb.utils.events INFO: eta: 6:18:23 iteration: 334299/375342 consumed_samples: 342323200 total_loss: 0.3425 time: 0.5452 s/iter data_time: 0.0536 s/iter total_throughput: 1878.38 samples/s lr: 3.89e-05 [09/21 07:04:23] lb.utils.events INFO: eta: 6:17:10 iteration: 334399/375342 consumed_samples: 342425600 total_loss: 0.3395 time: 0.5452 s/iter data_time: 0.0551 s/iter total_throughput: 1878.37 samples/s lr: 3.88e-05 [09/21 07:05:18] lb.utils.events INFO: eta: 6:16:08 iteration: 334499/375342 consumed_samples: 342528000 total_loss: 0.3462 time: 0.5452 s/iter data_time: 0.0504 s/iter total_throughput: 1878.36 samples/s lr: 3.86e-05 [09/21 07:06:14] lb.utils.events INFO: eta: 6:15:00 iteration: 334599/375342 consumed_samples: 342630400 total_loss: 0.3493 time: 0.5452 s/iter data_time: 0.0521 s/iter total_throughput: 1878.35 samples/s lr: 3.85e-05 [09/21 07:07:09] lb.utils.events INFO: eta: 6:13:35 iteration: 334699/375342 consumed_samples: 342732800 total_loss: 0.35 time: 0.5452 s/iter data_time: 0.0497 s/iter total_throughput: 1878.35 samples/s lr: 3.84e-05 [09/21 07:08:04] lb.utils.events INFO: eta: 6:12:28 iteration: 334799/375342 consumed_samples: 342835200 total_loss: 0.3514 time: 0.5452 s/iter data_time: 0.0532 s/iter total_throughput: 1878.34 samples/s lr: 3.82e-05 [09/21 07:08:59] lb.utils.events INFO: eta: 6:11:16 iteration: 334899/375342 consumed_samples: 342937600 total_loss: 0.347 time: 0.5452 s/iter data_time: 0.0535 s/iter total_throughput: 1878.34 samples/s lr: 3.81e-05 [09/21 07:09:54] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0334999 [09/21 07:09:54] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 07:09:54] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 07:09:59] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0587 s/iter. Inference: 0.2491 s/iter. Eval: 0.0024 s/iter. Total: 0.3102 s/iter. ETA=0:00:11 [09/21 07:10:04] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0835 s/iter. Inference: 0.2494 s/iter. Eval: 0.0026 s/iter. Total: 0.3359 s/iter. ETA=0:00:07 [09/21 07:10:09] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0782 s/iter. Inference: 0.2505 s/iter. Eval: 0.0024 s/iter. Total: 0.3317 s/iter. ETA=0:00:01 [09/21 07:10:11] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 07:10:11] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.282501 (0.000286 s / iter per device, on 8 devices) [09/21 07:10:11] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:10 (0.000219 s / iter per device, on 8 devices) [09/21 07:10:11] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 07:10:11] lb.evaluation.utils INFO: copypaste: Acc@1=79.794 [09/21 07:10:11] lb.evaluation.utils INFO: copypaste: Acc@5=94.664 [09/21 07:10:11] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 79.79400, better than last best score 79.68600 @ iteration 329999. [09/21 07:10:11] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 07:10:12] lb.utils.events INFO: eta: 6:10:14 iteration: 334999/375342 consumed_samples: 343040000 total_loss: 0.3466 time: 0.5452 s/iter data_time: 0.0534 s/iter total_throughput: 1878.33 samples/s lr: 3.80e-05 [09/21 07:11:07] lb.utils.events INFO: eta: 6:09:20 iteration: 335099/375342 consumed_samples: 343142400 total_loss: 0.3415 time: 0.5452 s/iter data_time: 0.0556 s/iter total_throughput: 1878.33 samples/s lr: 3.78e-05 [09/21 07:12:02] lb.utils.events INFO: eta: 6:08:30 iteration: 335199/375342 consumed_samples: 343244800 total_loss: 0.3483 time: 0.5452 s/iter data_time: 0.0541 s/iter total_throughput: 1878.32 samples/s lr: 3.77e-05 [09/21 07:12:58] lb.utils.events INFO: eta: 6:07:52 iteration: 335299/375342 consumed_samples: 343347200 total_loss: 0.3464 time: 0.5452 s/iter data_time: 0.0595 s/iter total_throughput: 1878.31 samples/s lr: 3.75e-05 [09/21 07:13:53] lb.utils.events INFO: eta: 6:07:06 iteration: 335399/375342 consumed_samples: 343449600 total_loss: 0.3453 time: 0.5452 s/iter data_time: 0.0551 s/iter total_throughput: 1878.30 samples/s lr: 3.74e-05 [09/21 07:14:48] lb.utils.events INFO: eta: 6:06:11 iteration: 335499/375342 consumed_samples: 343552000 total_loss: 0.3458 time: 0.5452 s/iter data_time: 0.0544 s/iter total_throughput: 1878.30 samples/s lr: 3.73e-05 [09/21 07:15:43] lb.utils.events INFO: eta: 6:05:14 iteration: 335599/375342 consumed_samples: 343654400 total_loss: 0.3461 time: 0.5452 s/iter data_time: 0.0518 s/iter total_throughput: 1878.29 samples/s lr: 3.71e-05 [09/21 07:16:39] lb.utils.events INFO: eta: 6:04:30 iteration: 335699/375342 consumed_samples: 343756800 total_loss: 0.3479 time: 0.5452 s/iter data_time: 0.0511 s/iter total_throughput: 1878.29 samples/s lr: 3.70e-05 [09/21 07:17:34] lb.utils.events INFO: eta: 6:03:38 iteration: 335799/375342 consumed_samples: 343859200 total_loss: 0.3438 time: 0.5452 s/iter data_time: 0.0545 s/iter total_throughput: 1878.28 samples/s lr: 3.69e-05 [09/21 07:18:29] lb.utils.events INFO: eta: 6:02:37 iteration: 335899/375342 consumed_samples: 343961600 total_loss: 0.3476 time: 0.5452 s/iter data_time: 0.0433 s/iter total_throughput: 1878.28 samples/s lr: 3.67e-05 [09/21 07:19:24] lb.utils.events INFO: eta: 6:01:41 iteration: 335999/375342 consumed_samples: 344064000 total_loss: 0.3507 time: 0.5452 s/iter data_time: 0.0420 s/iter total_throughput: 1878.27 samples/s lr: 3.66e-05 [09/21 07:20:19] lb.utils.events INFO: eta: 6:00:47 iteration: 336099/375342 consumed_samples: 344166400 total_loss: 0.3498 time: 0.5452 s/iter data_time: 0.0507 s/iter total_throughput: 1878.26 samples/s lr: 3.65e-05 [09/21 07:21:14] lb.utils.events INFO: eta: 6:00:02 iteration: 336199/375342 consumed_samples: 344268800 total_loss: 0.3441 time: 0.5452 s/iter data_time: 0.0509 s/iter total_throughput: 1878.25 samples/s lr: 3.63e-05 [09/21 07:22:10] lb.utils.events INFO: eta: 5:59:11 iteration: 336299/375342 consumed_samples: 344371200 total_loss: 0.3501 time: 0.5452 s/iter data_time: 0.0515 s/iter total_throughput: 1878.24 samples/s lr: 3.62e-05 [09/21 07:23:05] lb.utils.events INFO: eta: 5:58:16 iteration: 336399/375342 consumed_samples: 344473600 total_loss: 0.3471 time: 0.5452 s/iter data_time: 0.0519 s/iter total_throughput: 1878.24 samples/s lr: 3.61e-05 [09/21 07:24:01] lb.utils.events INFO: eta: 5:57:32 iteration: 336499/375342 consumed_samples: 344576000 total_loss: 0.3492 time: 0.5452 s/iter data_time: 0.0501 s/iter total_throughput: 1878.23 samples/s lr: 3.59e-05 [09/21 07:24:56] lb.utils.events INFO: eta: 5:56:47 iteration: 336599/375342 consumed_samples: 344678400 total_loss: 0.3543 time: 0.5452 s/iter data_time: 0.0404 s/iter total_throughput: 1878.22 samples/s lr: 3.58e-05 [09/21 07:25:51] lb.utils.events INFO: eta: 5:56:01 iteration: 336699/375342 consumed_samples: 344780800 total_loss: 0.3507 time: 0.5452 s/iter data_time: 0.0535 s/iter total_throughput: 1878.21 samples/s lr: 3.57e-05 [09/21 07:26:47] lb.utils.events INFO: eta: 5:55:10 iteration: 336799/375342 consumed_samples: 344883200 total_loss: 0.3466 time: 0.5452 s/iter data_time: 0.0566 s/iter total_throughput: 1878.20 samples/s lr: 3.55e-05 [09/21 07:27:42] lb.utils.events INFO: eta: 5:54:29 iteration: 336899/375342 consumed_samples: 344985600 total_loss: 0.3447 time: 0.5452 s/iter data_time: 0.0551 s/iter total_throughput: 1878.19 samples/s lr: 3.54e-05 [09/21 07:28:37] lb.utils.events INFO: eta: 5:53:36 iteration: 336999/375342 consumed_samples: 345088000 total_loss: 0.3474 time: 0.5452 s/iter data_time: 0.0528 s/iter total_throughput: 1878.19 samples/s lr: 3.53e-05 [09/21 07:29:33] lb.utils.events INFO: eta: 5:52:37 iteration: 337099/375342 consumed_samples: 345190400 total_loss: 0.3495 time: 0.5452 s/iter data_time: 0.0541 s/iter total_throughput: 1878.18 samples/s lr: 3.51e-05 [09/21 07:30:28] lb.utils.events INFO: eta: 5:51:26 iteration: 337199/375342 consumed_samples: 345292800 total_loss: 0.3422 time: 0.5452 s/iter data_time: 0.0526 s/iter total_throughput: 1878.18 samples/s lr: 3.50e-05 [09/21 07:31:22] lb.utils.events INFO: eta: 5:50:08 iteration: 337299/375342 consumed_samples: 345395200 total_loss: 0.3427 time: 0.5452 s/iter data_time: 0.0533 s/iter total_throughput: 1878.17 samples/s lr: 3.49e-05 [09/21 07:32:17] lb.utils.events INFO: eta: 5:48:42 iteration: 337399/375342 consumed_samples: 345497600 total_loss: 0.3446 time: 0.5452 s/iter data_time: 0.0516 s/iter total_throughput: 1878.17 samples/s lr: 3.48e-05 [09/21 07:33:12] lb.utils.events INFO: eta: 5:47:27 iteration: 337499/375342 consumed_samples: 345600000 total_loss: 0.3473 time: 0.5452 s/iter data_time: 0.0535 s/iter total_throughput: 1878.17 samples/s lr: 3.46e-05 [09/21 07:34:07] lb.utils.events INFO: eta: 5:46:06 iteration: 337599/375342 consumed_samples: 345702400 total_loss: 0.3533 time: 0.5452 s/iter data_time: 0.0511 s/iter total_throughput: 1878.17 samples/s lr: 3.45e-05 [09/21 07:35:01] lb.utils.events INFO: eta: 5:44:44 iteration: 337699/375342 consumed_samples: 345804800 total_loss: 0.3494 time: 0.5452 s/iter data_time: 0.0497 s/iter total_throughput: 1878.16 samples/s lr: 3.44e-05 [09/21 07:35:56] lb.utils.events INFO: eta: 5:43:31 iteration: 337799/375342 consumed_samples: 345907200 total_loss: 0.3432 time: 0.5452 s/iter data_time: 0.0498 s/iter total_throughput: 1878.16 samples/s lr: 3.42e-05 [09/21 07:36:51] lb.utils.events INFO: eta: 5:42:23 iteration: 337899/375342 consumed_samples: 346009600 total_loss: 0.3482 time: 0.5452 s/iter data_time: 0.0470 s/iter total_throughput: 1878.16 samples/s lr: 3.41e-05 [09/21 07:37:46] lb.utils.events INFO: eta: 5:41:25 iteration: 337999/375342 consumed_samples: 346112000 total_loss: 0.3475 time: 0.5452 s/iter data_time: 0.0472 s/iter total_throughput: 1878.15 samples/s lr: 3.40e-05 [09/21 07:38:42] lb.utils.events INFO: eta: 5:40:16 iteration: 338099/375342 consumed_samples: 346214400 total_loss: 0.3483 time: 0.5452 s/iter data_time: 0.0458 s/iter total_throughput: 1878.14 samples/s lr: 3.39e-05 [09/21 07:39:37] lb.utils.events INFO: eta: 5:39:17 iteration: 338199/375342 consumed_samples: 346316800 total_loss: 0.3501 time: 0.5452 s/iter data_time: 0.0431 s/iter total_throughput: 1878.14 samples/s lr: 3.37e-05 [09/21 07:40:32] lb.utils.events INFO: eta: 5:38:26 iteration: 338299/375342 consumed_samples: 346419200 total_loss: 0.3413 time: 0.5452 s/iter data_time: 0.0431 s/iter total_throughput: 1878.13 samples/s lr: 3.36e-05 [09/21 07:41:27] lb.utils.events INFO: eta: 5:37:51 iteration: 338399/375342 consumed_samples: 346521600 total_loss: 0.343 time: 0.5452 s/iter data_time: 0.0448 s/iter total_throughput: 1878.12 samples/s lr: 3.35e-05 [09/21 07:42:22] lb.utils.events INFO: eta: 5:37:19 iteration: 338499/375342 consumed_samples: 346624000 total_loss: 0.348 time: 0.5452 s/iter data_time: 0.0461 s/iter total_throughput: 1878.12 samples/s lr: 3.33e-05 [09/21 07:43:18] lb.utils.events INFO: eta: 5:36:52 iteration: 338599/375342 consumed_samples: 346726400 total_loss: 0.3468 time: 0.5452 s/iter data_time: 0.0491 s/iter total_throughput: 1878.11 samples/s lr: 3.32e-05 [09/21 07:44:13] lb.utils.events INFO: eta: 5:36:20 iteration: 338699/375342 consumed_samples: 346828800 total_loss: 0.3513 time: 0.5452 s/iter data_time: 0.0429 s/iter total_throughput: 1878.10 samples/s lr: 3.31e-05 [09/21 07:45:08] lb.utils.events INFO: eta: 5:35:35 iteration: 338799/375342 consumed_samples: 346931200 total_loss: 0.3519 time: 0.5452 s/iter data_time: 0.0495 s/iter total_throughput: 1878.10 samples/s lr: 3.30e-05 [09/21 07:46:03] lb.utils.events INFO: eta: 5:34:49 iteration: 338899/375342 consumed_samples: 347033600 total_loss: 0.3497 time: 0.5452 s/iter data_time: 0.0417 s/iter total_throughput: 1878.09 samples/s lr: 3.28e-05 [09/21 07:46:58] lb.utils.events INFO: eta: 5:33:56 iteration: 338999/375342 consumed_samples: 347136000 total_loss: 0.3458 time: 0.5452 s/iter data_time: 0.0482 s/iter total_throughput: 1878.09 samples/s lr: 3.27e-05 [09/21 07:47:53] lb.utils.events INFO: eta: 5:33:11 iteration: 339099/375342 consumed_samples: 347238400 total_loss: 0.3399 time: 0.5452 s/iter data_time: 0.0442 s/iter total_throughput: 1878.08 samples/s lr: 3.26e-05 [09/21 07:48:48] lb.utils.events INFO: eta: 5:32:17 iteration: 339199/375342 consumed_samples: 347340800 total_loss: 0.3453 time: 0.5452 s/iter data_time: 0.0472 s/iter total_throughput: 1878.07 samples/s lr: 3.25e-05 [09/21 07:49:43] lb.utils.events INFO: eta: 5:31:23 iteration: 339299/375342 consumed_samples: 347443200 total_loss: 0.3445 time: 0.5452 s/iter data_time: 0.0414 s/iter total_throughput: 1878.07 samples/s lr: 3.24e-05 [09/21 07:50:38] lb.utils.events INFO: eta: 5:30:19 iteration: 339399/375342 consumed_samples: 347545600 total_loss: 0.3456 time: 0.5452 s/iter data_time: 0.0425 s/iter total_throughput: 1878.06 samples/s lr: 3.22e-05 [09/21 07:51:34] lb.utils.events INFO: eta: 5:29:28 iteration: 339499/375342 consumed_samples: 347648000 total_loss: 0.3415 time: 0.5452 s/iter data_time: 0.0486 s/iter total_throughput: 1878.06 samples/s lr: 3.21e-05 [09/21 07:52:29] lb.utils.events INFO: eta: 5:28:37 iteration: 339599/375342 consumed_samples: 347750400 total_loss: 0.342 time: 0.5452 s/iter data_time: 0.0488 s/iter total_throughput: 1878.05 samples/s lr: 3.20e-05 [09/21 07:53:24] lb.utils.events INFO: eta: 5:27:46 iteration: 339699/375342 consumed_samples: 347852800 total_loss: 0.3483 time: 0.5452 s/iter data_time: 0.0537 s/iter total_throughput: 1878.04 samples/s lr: 3.19e-05 [09/21 07:54:20] lb.utils.events INFO: eta: 5:26:59 iteration: 339799/375342 consumed_samples: 347955200 total_loss: 0.3497 time: 0.5453 s/iter data_time: 0.0529 s/iter total_throughput: 1878.03 samples/s lr: 3.17e-05 [09/21 07:55:15] lb.utils.events INFO: eta: 5:26:10 iteration: 339899/375342 consumed_samples: 348057600 total_loss: 0.3499 time: 0.5453 s/iter data_time: 0.0513 s/iter total_throughput: 1878.03 samples/s lr: 3.16e-05 [09/21 07:56:10] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0339999 [09/21 07:56:11] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 07:56:11] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 07:56:15] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0634 s/iter. Inference: 0.2499 s/iter. Eval: 0.0021 s/iter. Total: 0.3155 s/iter. ETA=0:00:11 [09/21 07:56:21] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0737 s/iter. Inference: 0.2617 s/iter. Eval: 0.0027 s/iter. Total: 0.3385 s/iter. ETA=0:00:07 [09/21 07:56:26] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0739 s/iter. Inference: 0.2570 s/iter. Eval: 0.0025 s/iter. Total: 0.3339 s/iter. ETA=0:00:02 [09/21 07:56:28] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 07:56:28] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.318974 (0.000286 s / iter per device, on 8 devices) [09/21 07:56:28] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/21 07:56:28] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 07:56:28] lb.evaluation.utils INFO: copypaste: Acc@1=79.78 [09/21 07:56:28] lb.evaluation.utils INFO: copypaste: Acc@5=94.694 [09/21 07:56:28] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 79.78000, not better than best score 79.79400 @ iteration 334999. [09/21 07:56:28] lb.utils.events INFO: eta: 5:25:19 iteration: 339999/375342 consumed_samples: 348160000 total_loss: 0.3488 time: 0.5453 s/iter data_time: 0.0519 s/iter total_throughput: 1878.02 samples/s lr: 3.15e-05 [09/21 07:57:23] lb.utils.events INFO: eta: 5:24:18 iteration: 340099/375342 consumed_samples: 348262400 total_loss: 0.3485 time: 0.5453 s/iter data_time: 0.0520 s/iter total_throughput: 1878.01 samples/s lr: 3.14e-05 [09/21 07:58:18] lb.utils.events INFO: eta: 5:23:23 iteration: 340199/375342 consumed_samples: 348364800 total_loss: 0.3437 time: 0.5453 s/iter data_time: 0.0543 s/iter total_throughput: 1878.01 samples/s lr: 3.13e-05 [09/21 07:59:13] lb.utils.events INFO: eta: 5:22:28 iteration: 340299/375342 consumed_samples: 348467200 total_loss: 0.3466 time: 0.5453 s/iter data_time: 0.0532 s/iter total_throughput: 1878.00 samples/s lr: 3.11e-05 [09/21 08:00:08] lb.utils.events INFO: eta: 5:21:32 iteration: 340399/375342 consumed_samples: 348569600 total_loss: 0.3475 time: 0.5453 s/iter data_time: 0.0532 s/iter total_throughput: 1878.00 samples/s lr: 3.10e-05 [09/21 08:01:03] lb.utils.events INFO: eta: 5:20:22 iteration: 340499/375342 consumed_samples: 348672000 total_loss: 0.3476 time: 0.5453 s/iter data_time: 0.0539 s/iter total_throughput: 1877.99 samples/s lr: 3.09e-05 [09/21 08:01:58] lb.utils.events INFO: eta: 5:19:10 iteration: 340599/375342 consumed_samples: 348774400 total_loss: 0.348 time: 0.5453 s/iter data_time: 0.0503 s/iter total_throughput: 1877.99 samples/s lr: 3.08e-05 [09/21 08:02:53] lb.utils.events INFO: eta: 5:17:55 iteration: 340699/375342 consumed_samples: 348876800 total_loss: 0.3506 time: 0.5453 s/iter data_time: 0.0523 s/iter total_throughput: 1877.99 samples/s lr: 3.07e-05 [09/21 08:03:48] lb.utils.events INFO: eta: 5:16:40 iteration: 340799/375342 consumed_samples: 348979200 total_loss: 0.3563 time: 0.5453 s/iter data_time: 0.0525 s/iter total_throughput: 1877.99 samples/s lr: 3.05e-05 [09/21 08:04:42] lb.utils.events INFO: eta: 5:15:18 iteration: 340899/375342 consumed_samples: 349081600 total_loss: 0.3511 time: 0.5453 s/iter data_time: 0.0531 s/iter total_throughput: 1877.98 samples/s lr: 3.04e-05 [09/21 08:05:37] lb.utils.events INFO: eta: 5:13:57 iteration: 340999/375342 consumed_samples: 349184000 total_loss: 0.3498 time: 0.5453 s/iter data_time: 0.0536 s/iter total_throughput: 1877.98 samples/s lr: 3.03e-05 [09/21 08:06:32] lb.utils.events INFO: eta: 5:12:49 iteration: 341099/375342 consumed_samples: 349286400 total_loss: 0.3502 time: 0.5453 s/iter data_time: 0.0529 s/iter total_throughput: 1877.98 samples/s lr: 3.02e-05 [09/21 08:07:26] lb.utils.events INFO: eta: 5:11:43 iteration: 341199/375342 consumed_samples: 349388800 total_loss: 0.3459 time: 0.5453 s/iter data_time: 0.0468 s/iter total_throughput: 1877.98 samples/s lr: 3.01e-05 [09/21 08:08:21] lb.utils.events INFO: eta: 5:10:45 iteration: 341299/375342 consumed_samples: 349491200 total_loss: 0.3469 time: 0.5453 s/iter data_time: 0.0485 s/iter total_throughput: 1877.98 samples/s lr: 3.00e-05 [09/21 08:09:16] lb.utils.events INFO: eta: 5:09:42 iteration: 341399/375342 consumed_samples: 349593600 total_loss: 0.3493 time: 0.5453 s/iter data_time: 0.0474 s/iter total_throughput: 1877.97 samples/s lr: 2.98e-05 [09/21 08:10:12] lb.utils.events INFO: eta: 5:08:48 iteration: 341499/375342 consumed_samples: 349696000 total_loss: 0.3508 time: 0.5453 s/iter data_time: 0.0452 s/iter total_throughput: 1877.96 samples/s lr: 2.97e-05 [09/21 08:11:07] lb.utils.events INFO: eta: 5:07:56 iteration: 341599/375342 consumed_samples: 349798400 total_loss: 0.347 time: 0.5453 s/iter data_time: 0.0472 s/iter total_throughput: 1877.96 samples/s lr: 2.96e-05 [09/21 08:12:02] lb.utils.events INFO: eta: 5:07:03 iteration: 341699/375342 consumed_samples: 349900800 total_loss: 0.3476 time: 0.5453 s/iter data_time: 0.0432 s/iter total_throughput: 1877.96 samples/s lr: 2.95e-05 [09/21 08:12:57] lb.utils.events INFO: eta: 5:06:15 iteration: 341799/375342 consumed_samples: 350003200 total_loss: 0.3513 time: 0.5453 s/iter data_time: 0.0460 s/iter total_throughput: 1877.95 samples/s lr: 2.94e-05 [09/21 08:13:52] lb.utils.events INFO: eta: 5:05:36 iteration: 341899/375342 consumed_samples: 350105600 total_loss: 0.3511 time: 0.5453 s/iter data_time: 0.0441 s/iter total_throughput: 1877.95 samples/s lr: 2.93e-05 [09/21 08:14:47] lb.utils.events INFO: eta: 5:04:59 iteration: 341999/375342 consumed_samples: 350208000 total_loss: 0.344 time: 0.5453 s/iter data_time: 0.0436 s/iter total_throughput: 1877.94 samples/s lr: 2.92e-05 [09/21 08:15:42] lb.utils.events INFO: eta: 5:04:27 iteration: 342099/375342 consumed_samples: 350310400 total_loss: 0.3455 time: 0.5453 s/iter data_time: 0.0465 s/iter total_throughput: 1877.94 samples/s lr: 2.90e-05 [09/21 08:16:37] lb.utils.events INFO: eta: 5:03:55 iteration: 342199/375342 consumed_samples: 350412800 total_loss: 0.3464 time: 0.5453 s/iter data_time: 0.0513 s/iter total_throughput: 1877.93 samples/s lr: 2.89e-05 [09/21 08:17:32] lb.utils.events INFO: eta: 5:03:13 iteration: 342299/375342 consumed_samples: 350515200 total_loss: 0.3451 time: 0.5453 s/iter data_time: 0.0442 s/iter total_throughput: 1877.92 samples/s lr: 2.88e-05 [09/21 08:18:27] lb.utils.events INFO: eta: 5:02:30 iteration: 342399/375342 consumed_samples: 350617600 total_loss: 0.3432 time: 0.5453 s/iter data_time: 0.0460 s/iter total_throughput: 1877.92 samples/s lr: 2.87e-05 [09/21 08:19:23] lb.utils.events INFO: eta: 5:01:42 iteration: 342499/375342 consumed_samples: 350720000 total_loss: 0.3414 time: 0.5453 s/iter data_time: 0.0408 s/iter total_throughput: 1877.91 samples/s lr: 2.86e-05 [09/21 08:20:17] lb.utils.events INFO: eta: 5:00:45 iteration: 342599/375342 consumed_samples: 350822400 total_loss: 0.3431 time: 0.5453 s/iter data_time: 0.0460 s/iter total_throughput: 1877.91 samples/s lr: 2.85e-05 [09/21 08:21:13] lb.utils.events INFO: eta: 4:59:54 iteration: 342699/375342 consumed_samples: 350924800 total_loss: 0.3507 time: 0.5453 s/iter data_time: 0.0439 s/iter total_throughput: 1877.90 samples/s lr: 2.84e-05 [09/21 08:22:08] lb.utils.events INFO: eta: 4:58:57 iteration: 342799/375342 consumed_samples: 351027200 total_loss: 0.3478 time: 0.5453 s/iter data_time: 0.0437 s/iter total_throughput: 1877.90 samples/s lr: 2.82e-05 [09/21 08:23:03] lb.utils.events INFO: eta: 4:58:10 iteration: 342899/375342 consumed_samples: 351129600 total_loss: 0.3455 time: 0.5453 s/iter data_time: 0.0492 s/iter total_throughput: 1877.89 samples/s lr: 2.81e-05 [09/21 08:23:58] lb.utils.events INFO: eta: 4:57:31 iteration: 342999/375342 consumed_samples: 351232000 total_loss: 0.3477 time: 0.5453 s/iter data_time: 0.0534 s/iter total_throughput: 1877.88 samples/s lr: 2.80e-05 [09/21 08:24:54] lb.utils.events INFO: eta: 4:56:40 iteration: 343099/375342 consumed_samples: 351334400 total_loss: 0.3451 time: 0.5453 s/iter data_time: 0.0527 s/iter total_throughput: 1877.87 samples/s lr: 2.79e-05 [09/21 08:25:49] lb.utils.events INFO: eta: 4:55:41 iteration: 343199/375342 consumed_samples: 351436800 total_loss: 0.3446 time: 0.5453 s/iter data_time: 0.0484 s/iter total_throughput: 1877.87 samples/s lr: 2.78e-05 [09/21 08:26:44] lb.utils.events INFO: eta: 4:54:44 iteration: 343299/375342 consumed_samples: 351539200 total_loss: 0.3468 time: 0.5453 s/iter data_time: 0.0518 s/iter total_throughput: 1877.86 samples/s lr: 2.77e-05 [09/21 08:27:39] lb.utils.events INFO: eta: 4:53:48 iteration: 343399/375342 consumed_samples: 351641600 total_loss: 0.349 time: 0.5453 s/iter data_time: 0.0534 s/iter total_throughput: 1877.85 samples/s lr: 2.76e-05 [09/21 08:28:34] lb.utils.events INFO: eta: 4:52:53 iteration: 343499/375342 consumed_samples: 351744000 total_loss: 0.3472 time: 0.5453 s/iter data_time: 0.0529 s/iter total_throughput: 1877.85 samples/s lr: 2.75e-05 [09/21 08:29:29] lb.utils.events INFO: eta: 4:51:54 iteration: 343599/375342 consumed_samples: 351846400 total_loss: 0.3447 time: 0.5453 s/iter data_time: 0.0553 s/iter total_throughput: 1877.84 samples/s lr: 2.74e-05 [09/21 08:30:24] lb.utils.events INFO: eta: 4:50:57 iteration: 343699/375342 consumed_samples: 351948800 total_loss: 0.3501 time: 0.5453 s/iter data_time: 0.0508 s/iter total_throughput: 1877.84 samples/s lr: 2.73e-05 [09/21 08:31:19] lb.utils.events INFO: eta: 4:49:56 iteration: 343799/375342 consumed_samples: 352051200 total_loss: 0.3511 time: 0.5453 s/iter data_time: 0.0504 s/iter total_throughput: 1877.83 samples/s lr: 2.72e-05 [09/21 08:32:14] lb.utils.events INFO: eta: 4:48:43 iteration: 343899/375342 consumed_samples: 352153600 total_loss: 0.3494 time: 0.5453 s/iter data_time: 0.0520 s/iter total_throughput: 1877.83 samples/s lr: 2.70e-05 [09/21 08:33:09] lb.utils.events INFO: eta: 4:47:28 iteration: 343999/375342 consumed_samples: 352256000 total_loss: 0.343 time: 0.5453 s/iter data_time: 0.0524 s/iter total_throughput: 1877.83 samples/s lr: 2.69e-05 [09/21 08:34:04] lb.utils.events INFO: eta: 4:46:15 iteration: 344099/375342 consumed_samples: 352358400 total_loss: 0.3419 time: 0.5453 s/iter data_time: 0.0523 s/iter total_throughput: 1877.83 samples/s lr: 2.68e-05 [09/21 08:34:58] lb.utils.events INFO: eta: 4:45:05 iteration: 344199/375342 consumed_samples: 352460800 total_loss: 0.3434 time: 0.5453 s/iter data_time: 0.0500 s/iter total_throughput: 1877.83 samples/s lr: 2.67e-05 [09/21 08:35:53] lb.utils.events INFO: eta: 4:43:52 iteration: 344299/375342 consumed_samples: 352563200 total_loss: 0.345 time: 0.5453 s/iter data_time: 0.0513 s/iter total_throughput: 1877.83 samples/s lr: 2.66e-05 [09/21 08:36:48] lb.utils.events INFO: eta: 4:42:38 iteration: 344399/375342 consumed_samples: 352665600 total_loss: 0.3456 time: 0.5453 s/iter data_time: 0.0508 s/iter total_throughput: 1877.82 samples/s lr: 2.65e-05 [09/21 08:37:42] lb.utils.events INFO: eta: 4:41:29 iteration: 344499/375342 consumed_samples: 352768000 total_loss: 0.3353 time: 0.5453 s/iter data_time: 0.0516 s/iter total_throughput: 1877.82 samples/s lr: 2.64e-05 [09/21 08:38:37] lb.utils.events INFO: eta: 4:40:22 iteration: 344599/375342 consumed_samples: 352870400 total_loss: 0.3385 time: 0.5453 s/iter data_time: 0.0460 s/iter total_throughput: 1877.82 samples/s lr: 2.63e-05 [09/21 08:39:32] lb.utils.events INFO: eta: 4:39:26 iteration: 344699/375342 consumed_samples: 352972800 total_loss: 0.3445 time: 0.5453 s/iter data_time: 0.0465 s/iter total_throughput: 1877.82 samples/s lr: 2.62e-05 [09/21 08:40:27] lb.utils.events INFO: eta: 4:38:23 iteration: 344799/375342 consumed_samples: 353075200 total_loss: 0.3403 time: 0.5453 s/iter data_time: 0.0499 s/iter total_throughput: 1877.82 samples/s lr: 2.61e-05 [09/21 08:41:22] lb.utils.events INFO: eta: 4:37:25 iteration: 344899/375342 consumed_samples: 353177600 total_loss: 0.3473 time: 0.5453 s/iter data_time: 0.0480 s/iter total_throughput: 1877.81 samples/s lr: 2.60e-05 [09/21 08:42:17] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0344999 [09/21 08:42:18] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 08:42:18] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 08:42:22] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0548 s/iter. Inference: 0.2510 s/iter. Eval: 0.0022 s/iter. Total: 0.3081 s/iter. ETA=0:00:11 [09/21 08:42:27] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0605 s/iter. Inference: 0.2757 s/iter. Eval: 0.0022 s/iter. Total: 0.3388 s/iter. ETA=0:00:07 [09/21 08:42:33] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0618 s/iter. Inference: 0.2662 s/iter. Eval: 0.0022 s/iter. Total: 0.3307 s/iter. ETA=0:00:01 [09/21 08:42:35] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 08:42:35] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.308110 (0.000286 s / iter per device, on 8 devices) [09/21 08:42:35] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000231 s / iter per device, on 8 devices) [09/21 08:42:35] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 08:42:35] lb.evaluation.utils INFO: copypaste: Acc@1=79.95 [09/21 08:42:35] lb.evaluation.utils INFO: copypaste: Acc@5=94.664 [09/21 08:42:35] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 79.95000, better than last best score 79.79400 @ iteration 334999. [09/21 08:42:35] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 08:42:35] lb.utils.events INFO: eta: 4:36:32 iteration: 344999/375342 consumed_samples: 353280000 total_loss: 0.3519 time: 0.5453 s/iter data_time: 0.0443 s/iter total_throughput: 1877.80 samples/s lr: 2.59e-05 [09/21 08:43:30] lb.utils.events INFO: eta: 4:35:47 iteration: 345099/375342 consumed_samples: 353382400 total_loss: 0.3482 time: 0.5453 s/iter data_time: 0.0503 s/iter total_throughput: 1877.80 samples/s lr: 2.58e-05 [09/21 08:44:25] lb.utils.events INFO: eta: 4:35:08 iteration: 345199/375342 consumed_samples: 353484800 total_loss: 0.3515 time: 0.5453 s/iter data_time: 0.0455 s/iter total_throughput: 1877.80 samples/s lr: 2.57e-05 [09/21 08:45:20] lb.utils.events INFO: eta: 4:34:21 iteration: 345299/375342 consumed_samples: 353587200 total_loss: 0.3479 time: 0.5453 s/iter data_time: 0.0454 s/iter total_throughput: 1877.79 samples/s lr: 2.56e-05 [09/21 08:46:15] lb.utils.events INFO: eta: 4:33:39 iteration: 345399/375342 consumed_samples: 353689600 total_loss: 0.3491 time: 0.5453 s/iter data_time: 0.0514 s/iter total_throughput: 1877.79 samples/s lr: 2.55e-05 [09/21 08:47:10] lb.utils.events INFO: eta: 4:33:06 iteration: 345499/375342 consumed_samples: 353792000 total_loss: 0.3493 time: 0.5453 s/iter data_time: 0.0494 s/iter total_throughput: 1877.78 samples/s lr: 2.54e-05 [09/21 08:48:05] lb.utils.events INFO: eta: 4:32:28 iteration: 345599/375342 consumed_samples: 353894400 total_loss: 0.3439 time: 0.5453 s/iter data_time: 0.0502 s/iter total_throughput: 1877.78 samples/s lr: 2.53e-05 [09/21 08:49:01] lb.utils.events INFO: eta: 4:31:40 iteration: 345699/375342 consumed_samples: 353996800 total_loss: 0.3497 time: 0.5453 s/iter data_time: 0.0477 s/iter total_throughput: 1877.77 samples/s lr: 2.52e-05 [09/21 08:49:56] lb.utils.events INFO: eta: 4:30:52 iteration: 345799/375342 consumed_samples: 354099200 total_loss: 0.3468 time: 0.5453 s/iter data_time: 0.0463 s/iter total_throughput: 1877.77 samples/s lr: 2.51e-05 [09/21 08:50:51] lb.utils.events INFO: eta: 4:30:05 iteration: 345899/375342 consumed_samples: 354201600 total_loss: 0.3458 time: 0.5453 s/iter data_time: 0.0515 s/iter total_throughput: 1877.76 samples/s lr: 2.50e-05 [09/21 08:51:46] lb.utils.events INFO: eta: 4:29:09 iteration: 345999/375342 consumed_samples: 354304000 total_loss: 0.3489 time: 0.5453 s/iter data_time: 0.0492 s/iter total_throughput: 1877.76 samples/s lr: 2.49e-05 [09/21 08:52:41] lb.utils.events INFO: eta: 4:28:15 iteration: 346099/375342 consumed_samples: 354406400 total_loss: 0.3427 time: 0.5453 s/iter data_time: 0.0485 s/iter total_throughput: 1877.75 samples/s lr: 2.48e-05 [09/21 08:53:36] lb.utils.events INFO: eta: 4:27:24 iteration: 346199/375342 consumed_samples: 354508800 total_loss: 0.3397 time: 0.5453 s/iter data_time: 0.0451 s/iter total_throughput: 1877.75 samples/s lr: 2.47e-05 [09/21 08:54:31] lb.utils.events INFO: eta: 4:26:40 iteration: 346299/375342 consumed_samples: 354611200 total_loss: 0.3447 time: 0.5453 s/iter data_time: 0.0487 s/iter total_throughput: 1877.74 samples/s lr: 2.46e-05 [09/21 08:55:26] lb.utils.events INFO: eta: 4:25:52 iteration: 346399/375342 consumed_samples: 354713600 total_loss: 0.3483 time: 0.5453 s/iter data_time: 0.0531 s/iter total_throughput: 1877.73 samples/s lr: 2.45e-05 [09/21 08:56:22] lb.utils.events INFO: eta: 4:24:59 iteration: 346499/375342 consumed_samples: 354816000 total_loss: 0.3505 time: 0.5453 s/iter data_time: 0.0500 s/iter total_throughput: 1877.73 samples/s lr: 2.44e-05 [09/21 08:57:17] lb.utils.events INFO: eta: 4:24:07 iteration: 346599/375342 consumed_samples: 354918400 total_loss: 0.3501 time: 0.5453 s/iter data_time: 0.0503 s/iter total_throughput: 1877.72 samples/s lr: 2.43e-05 [09/21 08:58:12] lb.utils.events INFO: eta: 4:23:07 iteration: 346699/375342 consumed_samples: 355020800 total_loss: 0.3475 time: 0.5453 s/iter data_time: 0.0526 s/iter total_throughput: 1877.71 samples/s lr: 2.42e-05 [09/21 08:59:07] lb.utils.events INFO: eta: 4:22:12 iteration: 346799/375342 consumed_samples: 355123200 total_loss: 0.3476 time: 0.5453 s/iter data_time: 0.0511 s/iter total_throughput: 1877.71 samples/s lr: 2.41e-05 [09/21 09:00:02] lb.utils.events INFO: eta: 4:21:22 iteration: 346899/375342 consumed_samples: 355225600 total_loss: 0.3461 time: 0.5453 s/iter data_time: 0.0540 s/iter total_throughput: 1877.70 samples/s lr: 2.40e-05 [09/21 09:00:57] lb.utils.events INFO: eta: 4:20:30 iteration: 346999/375342 consumed_samples: 355328000 total_loss: 0.3468 time: 0.5453 s/iter data_time: 0.0551 s/iter total_throughput: 1877.70 samples/s lr: 2.39e-05 [09/21 09:01:52] lb.utils.events INFO: eta: 4:19:36 iteration: 347099/375342 consumed_samples: 355430400 total_loss: 0.3454 time: 0.5454 s/iter data_time: 0.0536 s/iter total_throughput: 1877.69 samples/s lr: 2.38e-05 [09/21 09:02:48] lb.utils.events INFO: eta: 4:18:37 iteration: 347199/375342 consumed_samples: 355532800 total_loss: 0.3446 time: 0.5454 s/iter data_time: 0.0545 s/iter total_throughput: 1877.69 samples/s lr: 2.37e-05 [09/21 09:03:42] lb.utils.events INFO: eta: 4:17:33 iteration: 347299/375342 consumed_samples: 355635200 total_loss: 0.3493 time: 0.5454 s/iter data_time: 0.0531 s/iter total_throughput: 1877.68 samples/s lr: 2.36e-05 [09/21 09:04:37] lb.utils.events INFO: eta: 4:16:30 iteration: 347399/375342 consumed_samples: 355737600 total_loss: 0.3491 time: 0.5454 s/iter data_time: 0.0535 s/iter total_throughput: 1877.68 samples/s lr: 2.35e-05 [09/21 09:05:32] lb.utils.events INFO: eta: 4:15:20 iteration: 347499/375342 consumed_samples: 355840000 total_loss: 0.3479 time: 0.5454 s/iter data_time: 0.0519 s/iter total_throughput: 1877.68 samples/s lr: 2.34e-05 [09/21 09:06:27] lb.utils.events INFO: eta: 4:14:12 iteration: 347599/375342 consumed_samples: 355942400 total_loss: 0.346 time: 0.5454 s/iter data_time: 0.0532 s/iter total_throughput: 1877.68 samples/s lr: 2.33e-05 [09/21 09:07:22] lb.utils.events INFO: eta: 4:13:07 iteration: 347699/375342 consumed_samples: 356044800 total_loss: 0.3431 time: 0.5454 s/iter data_time: 0.0522 s/iter total_throughput: 1877.68 samples/s lr: 2.32e-05 [09/21 09:08:16] lb.utils.events INFO: eta: 4:12:01 iteration: 347799/375342 consumed_samples: 356147200 total_loss: 0.3431 time: 0.5454 s/iter data_time: 0.0524 s/iter total_throughput: 1877.68 samples/s lr: 2.31e-05 [09/21 09:09:11] lb.utils.events INFO: eta: 4:10:48 iteration: 347899/375342 consumed_samples: 356249600 total_loss: 0.3473 time: 0.5454 s/iter data_time: 0.0520 s/iter total_throughput: 1877.67 samples/s lr: 2.30e-05 [09/21 09:10:05] lb.utils.events INFO: eta: 4:09:42 iteration: 347999/375342 consumed_samples: 356352000 total_loss: 0.3468 time: 0.5454 s/iter data_time: 0.0532 s/iter total_throughput: 1877.67 samples/s lr: 2.29e-05 [09/21 09:11:00] lb.utils.events INFO: eta: 4:08:37 iteration: 348099/375342 consumed_samples: 356454400 total_loss: 0.3452 time: 0.5454 s/iter data_time: 0.0473 s/iter total_throughput: 1877.67 samples/s lr: 2.28e-05 [09/21 09:11:55] lb.utils.events INFO: eta: 4:07:34 iteration: 348199/375342 consumed_samples: 356556800 total_loss: 0.3469 time: 0.5454 s/iter data_time: 0.0459 s/iter total_throughput: 1877.67 samples/s lr: 2.27e-05 [09/21 09:12:50] lb.utils.events INFO: eta: 4:06:37 iteration: 348299/375342 consumed_samples: 356659200 total_loss: 0.3484 time: 0.5454 s/iter data_time: 0.0489 s/iter total_throughput: 1877.66 samples/s lr: 2.26e-05 [09/21 09:13:46] lb.utils.events INFO: eta: 4:05:42 iteration: 348399/375342 consumed_samples: 356761600 total_loss: 0.3471 time: 0.5454 s/iter data_time: 0.0470 s/iter total_throughput: 1877.65 samples/s lr: 2.25e-05 [09/21 09:14:41] lb.utils.events INFO: eta: 4:04:52 iteration: 348499/375342 consumed_samples: 356864000 total_loss: 0.3456 time: 0.5454 s/iter data_time: 0.0466 s/iter total_throughput: 1877.65 samples/s lr: 2.24e-05 [09/21 09:15:36] lb.utils.events INFO: eta: 4:04:05 iteration: 348599/375342 consumed_samples: 356966400 total_loss: 0.3471 time: 0.5454 s/iter data_time: 0.0451 s/iter total_throughput: 1877.65 samples/s lr: 2.23e-05 [09/21 09:16:31] lb.utils.events INFO: eta: 4:03:17 iteration: 348699/375342 consumed_samples: 357068800 total_loss: 0.3506 time: 0.5454 s/iter data_time: 0.0388 s/iter total_throughput: 1877.64 samples/s lr: 2.23e-05 [09/21 09:17:26] lb.utils.events INFO: eta: 4:02:41 iteration: 348799/375342 consumed_samples: 357171200 total_loss: 0.351 time: 0.5454 s/iter data_time: 0.0460 s/iter total_throughput: 1877.64 samples/s lr: 2.22e-05 [09/21 09:18:20] lb.utils.events INFO: eta: 4:02:02 iteration: 348899/375342 consumed_samples: 357273600 total_loss: 0.3454 time: 0.5454 s/iter data_time: 0.0422 s/iter total_throughput: 1877.63 samples/s lr: 2.21e-05 [09/21 09:19:15] lb.utils.events INFO: eta: 4:01:15 iteration: 348999/375342 consumed_samples: 357376000 total_loss: 0.342 time: 0.5454 s/iter data_time: 0.0437 s/iter total_throughput: 1877.63 samples/s lr: 2.20e-05 [09/21 09:20:10] lb.utils.events INFO: eta: 4:00:23 iteration: 349099/375342 consumed_samples: 357478400 total_loss: 0.3413 time: 0.5454 s/iter data_time: 0.0425 s/iter total_throughput: 1877.63 samples/s lr: 2.19e-05 [09/21 09:21:06] lb.utils.events INFO: eta: 3:59:35 iteration: 349199/375342 consumed_samples: 357580800 total_loss: 0.3465 time: 0.5454 s/iter data_time: 0.0435 s/iter total_throughput: 1877.62 samples/s lr: 2.18e-05 [09/21 09:22:01] lb.utils.events INFO: eta: 3:58:47 iteration: 349299/375342 consumed_samples: 357683200 total_loss: 0.3483 time: 0.5454 s/iter data_time: 0.0446 s/iter total_throughput: 1877.62 samples/s lr: 2.17e-05 [09/21 09:22:56] lb.utils.events INFO: eta: 3:57:57 iteration: 349399/375342 consumed_samples: 357785600 total_loss: 0.3408 time: 0.5454 s/iter data_time: 0.0497 s/iter total_throughput: 1877.61 samples/s lr: 2.16e-05 [09/21 09:23:51] lb.utils.events INFO: eta: 3:57:03 iteration: 349499/375342 consumed_samples: 357888000 total_loss: 0.3486 time: 0.5454 s/iter data_time: 0.0453 s/iter total_throughput: 1877.61 samples/s lr: 2.15e-05 [09/21 09:24:46] lb.utils.events INFO: eta: 3:56:10 iteration: 349599/375342 consumed_samples: 357990400 total_loss: 0.3545 time: 0.5454 s/iter data_time: 0.0492 s/iter total_throughput: 1877.60 samples/s lr: 2.14e-05 [09/21 09:25:41] lb.utils.events INFO: eta: 3:55:24 iteration: 349699/375342 consumed_samples: 358092800 total_loss: 0.3473 time: 0.5454 s/iter data_time: 0.0495 s/iter total_throughput: 1877.59 samples/s lr: 2.14e-05 [09/21 09:26:36] lb.utils.events INFO: eta: 3:54:38 iteration: 349799/375342 consumed_samples: 358195200 total_loss: 0.3444 time: 0.5454 s/iter data_time: 0.0529 s/iter total_throughput: 1877.59 samples/s lr: 2.13e-05 [09/21 09:27:32] lb.utils.events INFO: eta: 3:53:48 iteration: 349899/375342 consumed_samples: 358297600 total_loss: 0.3377 time: 0.5454 s/iter data_time: 0.0522 s/iter total_throughput: 1877.58 samples/s lr: 2.12e-05 [09/21 09:28:27] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0349999 [09/21 09:28:27] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 09:28:27] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 09:28:32] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0606 s/iter. Inference: 0.2475 s/iter. Eval: 0.0025 s/iter. Total: 0.3105 s/iter. ETA=0:00:11 [09/21 09:28:37] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0631 s/iter. Inference: 0.2683 s/iter. Eval: 0.0026 s/iter. Total: 0.3343 s/iter. ETA=0:00:07 [09/21 09:28:42] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0641 s/iter. Inference: 0.2633 s/iter. Eval: 0.0025 s/iter. Total: 0.3303 s/iter. ETA=0:00:01 [09/21 09:28:44] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 09:28:44] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.276346 (0.000286 s / iter per device, on 8 devices) [09/21 09:28:44] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000231 s / iter per device, on 8 devices) [09/21 09:28:44] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 09:28:44] lb.evaluation.utils INFO: copypaste: Acc@1=79.92 [09/21 09:28:44] lb.evaluation.utils INFO: copypaste: Acc@5=94.718 [09/21 09:28:44] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 79.92000, not better than best score 79.95000 @ iteration 344999. [09/21 09:28:44] lb.utils.events INFO: eta: 3:53:00 iteration: 349999/375342 consumed_samples: 358400000 total_loss: 0.3526 time: 0.5454 s/iter data_time: 0.0505 s/iter total_throughput: 1877.57 samples/s lr: 2.11e-05 [09/21 09:29:39] lb.utils.events INFO: eta: 3:52:03 iteration: 350099/375342 consumed_samples: 358502400 total_loss: 0.3492 time: 0.5454 s/iter data_time: 0.0528 s/iter total_throughput: 1877.57 samples/s lr: 2.10e-05 [09/21 09:30:34] lb.utils.events INFO: eta: 3:51:06 iteration: 350199/375342 consumed_samples: 358604800 total_loss: 0.345 time: 0.5454 s/iter data_time: 0.0519 s/iter total_throughput: 1877.57 samples/s lr: 2.09e-05 [09/21 09:31:30] lb.utils.events INFO: eta: 3:50:08 iteration: 350299/375342 consumed_samples: 358707200 total_loss: 0.3491 time: 0.5454 s/iter data_time: 0.0521 s/iter total_throughput: 1877.56 samples/s lr: 2.08e-05 [09/21 09:32:25] lb.utils.events INFO: eta: 3:49:08 iteration: 350399/375342 consumed_samples: 358809600 total_loss: 0.3481 time: 0.5454 s/iter data_time: 0.0514 s/iter total_throughput: 1877.56 samples/s lr: 2.07e-05 [09/21 09:33:20] lb.utils.events INFO: eta: 3:48:11 iteration: 350499/375342 consumed_samples: 358912000 total_loss: 0.3399 time: 0.5454 s/iter data_time: 0.0486 s/iter total_throughput: 1877.55 samples/s lr: 2.07e-05 [09/21 09:34:14] lb.utils.events INFO: eta: 3:47:05 iteration: 350599/375342 consumed_samples: 359014400 total_loss: 0.3395 time: 0.5454 s/iter data_time: 0.0529 s/iter total_throughput: 1877.55 samples/s lr: 2.06e-05 [09/21 09:35:09] lb.utils.events INFO: eta: 3:45:56 iteration: 350699/375342 consumed_samples: 359116800 total_loss: 0.344 time: 0.5454 s/iter data_time: 0.0523 s/iter total_throughput: 1877.55 samples/s lr: 2.05e-05 [09/21 09:36:04] lb.utils.events INFO: eta: 3:44:54 iteration: 350799/375342 consumed_samples: 359219200 total_loss: 0.349 time: 0.5454 s/iter data_time: 0.0527 s/iter total_throughput: 1877.54 samples/s lr: 2.04e-05 [09/21 09:36:59] lb.utils.events INFO: eta: 3:43:48 iteration: 350899/375342 consumed_samples: 359321600 total_loss: 0.3469 time: 0.5454 s/iter data_time: 0.0517 s/iter total_throughput: 1877.54 samples/s lr: 2.03e-05 [09/21 09:37:54] lb.utils.events INFO: eta: 3:42:40 iteration: 350999/375342 consumed_samples: 359424000 total_loss: 0.3463 time: 0.5454 s/iter data_time: 0.0527 s/iter total_throughput: 1877.54 samples/s lr: 2.02e-05 [09/21 09:38:48] lb.utils.events INFO: eta: 3:41:37 iteration: 351099/375342 consumed_samples: 359526400 total_loss: 0.3476 time: 0.5454 s/iter data_time: 0.0514 s/iter total_throughput: 1877.54 samples/s lr: 2.02e-05 [09/21 09:39:43] lb.utils.events INFO: eta: 3:40:32 iteration: 351199/375342 consumed_samples: 359628800 total_loss: 0.3521 time: 0.5454 s/iter data_time: 0.0514 s/iter total_throughput: 1877.54 samples/s lr: 2.01e-05 [09/21 09:40:38] lb.utils.events INFO: eta: 3:39:32 iteration: 351299/375342 consumed_samples: 359731200 total_loss: 0.3489 time: 0.5454 s/iter data_time: 0.0528 s/iter total_throughput: 1877.54 samples/s lr: 2.00e-05 [09/21 09:41:32] lb.utils.events INFO: eta: 3:38:30 iteration: 351399/375342 consumed_samples: 359833600 total_loss: 0.3457 time: 0.5454 s/iter data_time: 0.0525 s/iter total_throughput: 1877.54 samples/s lr: 1.99e-05 [09/21 09:42:27] lb.utils.events INFO: eta: 3:37:32 iteration: 351499/375342 consumed_samples: 359936000 total_loss: 0.345 time: 0.5454 s/iter data_time: 0.0463 s/iter total_throughput: 1877.53 samples/s lr: 1.98e-05 [09/21 09:43:22] lb.utils.events INFO: eta: 3:36:37 iteration: 351599/375342 consumed_samples: 360038400 total_loss: 0.3453 time: 0.5454 s/iter data_time: 0.0477 s/iter total_throughput: 1877.53 samples/s lr: 1.97e-05 [09/21 09:44:17] lb.utils.events INFO: eta: 3:35:41 iteration: 351699/375342 consumed_samples: 360140800 total_loss: 0.3449 time: 0.5454 s/iter data_time: 0.0478 s/iter total_throughput: 1877.53 samples/s lr: 1.97e-05 [09/21 09:45:12] lb.utils.events INFO: eta: 3:34:42 iteration: 351799/375342 consumed_samples: 360243200 total_loss: 0.3429 time: 0.5454 s/iter data_time: 0.0473 s/iter total_throughput: 1877.52 samples/s lr: 1.96e-05 [09/21 09:46:07] lb.utils.events INFO: eta: 3:33:48 iteration: 351899/375342 consumed_samples: 360345600 total_loss: 0.3409 time: 0.5454 s/iter data_time: 0.0423 s/iter total_throughput: 1877.51 samples/s lr: 1.95e-05 [09/21 09:47:02] lb.utils.events INFO: eta: 3:33:00 iteration: 351999/375342 consumed_samples: 360448000 total_loss: 0.3449 time: 0.5454 s/iter data_time: 0.0455 s/iter total_throughput: 1877.51 samples/s lr: 1.94e-05 [09/21 09:47:57] lb.utils.events INFO: eta: 3:32:12 iteration: 352099/375342 consumed_samples: 360550400 total_loss: 0.3493 time: 0.5454 s/iter data_time: 0.0464 s/iter total_throughput: 1877.50 samples/s lr: 1.93e-05 [09/21 09:48:52] lb.utils.events INFO: eta: 3:31:23 iteration: 352199/375342 consumed_samples: 360652800 total_loss: 0.3509 time: 0.5454 s/iter data_time: 0.0471 s/iter total_throughput: 1877.50 samples/s lr: 1.93e-05 [09/21 09:49:47] lb.utils.events INFO: eta: 3:30:39 iteration: 352299/375342 consumed_samples: 360755200 total_loss: 0.3501 time: 0.5454 s/iter data_time: 0.0416 s/iter total_throughput: 1877.50 samples/s lr: 1.92e-05 [09/21 09:50:42] lb.utils.events INFO: eta: 3:29:53 iteration: 352399/375342 consumed_samples: 360857600 total_loss: 0.3433 time: 0.5454 s/iter data_time: 0.0490 s/iter total_throughput: 1877.49 samples/s lr: 1.91e-05 [09/21 09:51:37] lb.utils.events INFO: eta: 3:29:06 iteration: 352499/375342 consumed_samples: 360960000 total_loss: 0.3435 time: 0.5454 s/iter data_time: 0.0508 s/iter total_throughput: 1877.49 samples/s lr: 1.90e-05 [09/21 09:52:32] lb.utils.events INFO: eta: 3:28:17 iteration: 352599/375342 consumed_samples: 361062400 total_loss: 0.3494 time: 0.5454 s/iter data_time: 0.0470 s/iter total_throughput: 1877.49 samples/s lr: 1.89e-05 [09/21 09:53:27] lb.utils.events INFO: eta: 3:27:27 iteration: 352699/375342 consumed_samples: 361164800 total_loss: 0.3467 time: 0.5454 s/iter data_time: 0.0455 s/iter total_throughput: 1877.48 samples/s lr: 1.89e-05 [09/21 09:54:22] lb.utils.events INFO: eta: 3:26:37 iteration: 352799/375342 consumed_samples: 361267200 total_loss: 0.3476 time: 0.5454 s/iter data_time: 0.0473 s/iter total_throughput: 1877.48 samples/s lr: 1.88e-05 [09/21 09:55:17] lb.utils.events INFO: eta: 3:25:45 iteration: 352899/375342 consumed_samples: 361369600 total_loss: 0.3448 time: 0.5454 s/iter data_time: 0.0443 s/iter total_throughput: 1877.47 samples/s lr: 1.87e-05 [09/21 09:56:12] lb.utils.events INFO: eta: 3:24:48 iteration: 352999/375342 consumed_samples: 361472000 total_loss: 0.3364 time: 0.5454 s/iter data_time: 0.0458 s/iter total_throughput: 1877.47 samples/s lr: 1.86e-05 [09/21 09:57:07] lb.utils.events INFO: eta: 3:23:54 iteration: 353099/375342 consumed_samples: 361574400 total_loss: 0.348 time: 0.5454 s/iter data_time: 0.0507 s/iter total_throughput: 1877.46 samples/s lr: 1.86e-05 [09/21 09:58:03] lb.utils.events INFO: eta: 3:23:06 iteration: 353199/375342 consumed_samples: 361676800 total_loss: 0.3525 time: 0.5454 s/iter data_time: 0.0535 s/iter total_throughput: 1877.46 samples/s lr: 1.85e-05 [09/21 09:58:58] lb.utils.events INFO: eta: 3:22:12 iteration: 353299/375342 consumed_samples: 361779200 total_loss: 0.3501 time: 0.5454 s/iter data_time: 0.0511 s/iter total_throughput: 1877.45 samples/s lr: 1.84e-05 [09/21 09:59:53] lb.utils.events INFO: eta: 3:21:18 iteration: 353399/375342 consumed_samples: 361881600 total_loss: 0.3494 time: 0.5454 s/iter data_time: 0.0496 s/iter total_throughput: 1877.45 samples/s lr: 1.83e-05 [09/21 10:00:48] lb.utils.events INFO: eta: 3:20:25 iteration: 353499/375342 consumed_samples: 361984000 total_loss: 0.3467 time: 0.5454 s/iter data_time: 0.0505 s/iter total_throughput: 1877.44 samples/s lr: 1.82e-05 [09/21 10:01:43] lb.utils.events INFO: eta: 3:19:31 iteration: 353599/375342 consumed_samples: 362086400 total_loss: 0.3436 time: 0.5454 s/iter data_time: 0.0499 s/iter total_throughput: 1877.44 samples/s lr: 1.82e-05 [09/21 10:02:38] lb.utils.events INFO: eta: 3:18:35 iteration: 353699/375342 consumed_samples: 362188800 total_loss: 0.3432 time: 0.5454 s/iter data_time: 0.0521 s/iter total_throughput: 1877.43 samples/s lr: 1.81e-05 [09/21 10:03:33] lb.utils.events INFO: eta: 3:17:40 iteration: 353799/375342 consumed_samples: 362291200 total_loss: 0.3441 time: 0.5454 s/iter data_time: 0.0552 s/iter total_throughput: 1877.43 samples/s lr: 1.80e-05 [09/21 10:04:28] lb.utils.events INFO: eta: 3:16:43 iteration: 353899/375342 consumed_samples: 362393600 total_loss: 0.3489 time: 0.5454 s/iter data_time: 0.0541 s/iter total_throughput: 1877.43 samples/s lr: 1.80e-05 [09/21 10:05:23] lb.utils.events INFO: eta: 3:15:45 iteration: 353999/375342 consumed_samples: 362496000 total_loss: 0.3471 time: 0.5454 s/iter data_time: 0.0515 s/iter total_throughput: 1877.42 samples/s lr: 1.79e-05 [09/21 10:06:17] lb.utils.events INFO: eta: 3:14:40 iteration: 354099/375342 consumed_samples: 362598400 total_loss: 0.3436 time: 0.5454 s/iter data_time: 0.0519 s/iter total_throughput: 1877.42 samples/s lr: 1.78e-05 [09/21 10:07:12] lb.utils.events INFO: eta: 3:13:37 iteration: 354199/375342 consumed_samples: 362700800 total_loss: 0.3466 time: 0.5454 s/iter data_time: 0.0503 s/iter total_throughput: 1877.42 samples/s lr: 1.77e-05 [09/21 10:08:07] lb.utils.events INFO: eta: 3:12:33 iteration: 354299/375342 consumed_samples: 362803200 total_loss: 0.3506 time: 0.5454 s/iter data_time: 0.0536 s/iter total_throughput: 1877.41 samples/s lr: 1.77e-05 [09/21 10:09:02] lb.utils.events INFO: eta: 3:11:29 iteration: 354399/375342 consumed_samples: 362905600 total_loss: 0.3522 time: 0.5454 s/iter data_time: 0.0511 s/iter total_throughput: 1877.41 samples/s lr: 1.76e-05 [09/21 10:09:56] lb.utils.events INFO: eta: 3:10:23 iteration: 354499/375342 consumed_samples: 363008000 total_loss: 0.3495 time: 0.5454 s/iter data_time: 0.0506 s/iter total_throughput: 1877.41 samples/s lr: 1.75e-05 [09/21 10:10:51] lb.utils.events INFO: eta: 3:09:23 iteration: 354599/375342 consumed_samples: 363110400 total_loss: 0.3451 time: 0.5454 s/iter data_time: 0.0519 s/iter total_throughput: 1877.41 samples/s lr: 1.74e-05 [09/21 10:11:46] lb.utils.events INFO: eta: 3:08:24 iteration: 354699/375342 consumed_samples: 363212800 total_loss: 0.3491 time: 0.5454 s/iter data_time: 0.0529 s/iter total_throughput: 1877.41 samples/s lr: 1.74e-05 [09/21 10:12:40] lb.utils.events INFO: eta: 3:07:23 iteration: 354799/375342 consumed_samples: 363315200 total_loss: 0.348 time: 0.5454 s/iter data_time: 0.0520 s/iter total_throughput: 1877.41 samples/s lr: 1.73e-05 [09/21 10:13:35] lb.utils.events INFO: eta: 3:06:22 iteration: 354899/375342 consumed_samples: 363417600 total_loss: 0.3441 time: 0.5454 s/iter data_time: 0.0531 s/iter total_throughput: 1877.41 samples/s lr: 1.72e-05 [09/21 10:14:30] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0354999 [09/21 10:14:30] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 10:14:30] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 10:14:35] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0647 s/iter. Inference: 0.2503 s/iter. Eval: 0.0024 s/iter. Total: 0.3174 s/iter. ETA=0:00:11 [09/21 10:14:40] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0734 s/iter. Inference: 0.2610 s/iter. Eval: 0.0026 s/iter. Total: 0.3373 s/iter. ETA=0:00:07 [09/21 10:14:45] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0738 s/iter. Inference: 0.2566 s/iter. Eval: 0.0026 s/iter. Total: 0.3334 s/iter. ETA=0:00:02 [09/21 10:14:47] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 10:14:47] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.293641 (0.000286 s / iter per device, on 8 devices) [09/21 10:14:47] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000224 s / iter per device, on 8 devices) [09/21 10:14:47] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 10:14:47] lb.evaluation.utils INFO: copypaste: Acc@1=79.932 [09/21 10:14:47] lb.evaluation.utils INFO: copypaste: Acc@5=94.688 [09/21 10:14:47] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 79.93200, not better than best score 79.95000 @ iteration 344999. [09/21 10:14:47] lb.utils.events INFO: eta: 3:05:25 iteration: 354999/375342 consumed_samples: 363520000 total_loss: 0.3424 time: 0.5454 s/iter data_time: 0.0470 s/iter total_throughput: 1877.41 samples/s lr: 1.72e-05 [09/21 10:15:42] lb.utils.events INFO: eta: 3:04:31 iteration: 355099/375342 consumed_samples: 363622400 total_loss: 0.337 time: 0.5454 s/iter data_time: 0.0496 s/iter total_throughput: 1877.40 samples/s lr: 1.71e-05 [09/21 10:16:37] lb.utils.events INFO: eta: 3:03:35 iteration: 355199/375342 consumed_samples: 363724800 total_loss: 0.3406 time: 0.5454 s/iter data_time: 0.0479 s/iter total_throughput: 1877.40 samples/s lr: 1.70e-05 [09/21 10:17:33] lb.utils.events INFO: eta: 3:02:40 iteration: 355299/375342 consumed_samples: 363827200 total_loss: 0.3482 time: 0.5454 s/iter data_time: 0.0427 s/iter total_throughput: 1877.39 samples/s lr: 1.69e-05 [09/21 10:18:28] lb.utils.events INFO: eta: 3:01:48 iteration: 355399/375342 consumed_samples: 363929600 total_loss: 0.3474 time: 0.5454 s/iter data_time: 0.0483 s/iter total_throughput: 1877.39 samples/s lr: 1.69e-05 [09/21 10:19:22] lb.utils.events INFO: eta: 3:00:58 iteration: 355499/375342 consumed_samples: 364032000 total_loss: 0.3434 time: 0.5454 s/iter data_time: 0.0472 s/iter total_throughput: 1877.39 samples/s lr: 1.68e-05 [09/21 10:20:17] lb.utils.events INFO: eta: 3:00:11 iteration: 355599/375342 consumed_samples: 364134400 total_loss: 0.3468 time: 0.5454 s/iter data_time: 0.0483 s/iter total_throughput: 1877.38 samples/s lr: 1.67e-05 [09/21 10:21:12] lb.utils.events INFO: eta: 2:59:25 iteration: 355699/375342 consumed_samples: 364236800 total_loss: 0.345 time: 0.5454 s/iter data_time: 0.0452 s/iter total_throughput: 1877.38 samples/s lr: 1.67e-05 [09/21 10:22:07] lb.utils.events INFO: eta: 2:58:40 iteration: 355799/375342 consumed_samples: 364339200 total_loss: 0.3334 time: 0.5454 s/iter data_time: 0.0452 s/iter total_throughput: 1877.37 samples/s lr: 1.66e-05 [09/21 10:23:02] lb.utils.events INFO: eta: 2:57:56 iteration: 355899/375342 consumed_samples: 364441600 total_loss: 0.3394 time: 0.5454 s/iter data_time: 0.0457 s/iter total_throughput: 1877.37 samples/s lr: 1.65e-05 [09/21 10:23:57] lb.utils.events INFO: eta: 2:57:06 iteration: 355999/375342 consumed_samples: 364544000 total_loss: 0.3429 time: 0.5454 s/iter data_time: 0.0460 s/iter total_throughput: 1877.37 samples/s lr: 1.65e-05 [09/21 10:24:53] lb.utils.events INFO: eta: 2:56:14 iteration: 356099/375342 consumed_samples: 364646400 total_loss: 0.3447 time: 0.5454 s/iter data_time: 0.0451 s/iter total_throughput: 1877.36 samples/s lr: 1.64e-05 [09/21 10:25:48] lb.utils.events INFO: eta: 2:55:26 iteration: 356199/375342 consumed_samples: 364748800 total_loss: 0.3466 time: 0.5454 s/iter data_time: 0.0419 s/iter total_throughput: 1877.36 samples/s lr: 1.63e-05 [09/21 10:26:42] lb.utils.events INFO: eta: 2:54:31 iteration: 356299/375342 consumed_samples: 364851200 total_loss: 0.3436 time: 0.5454 s/iter data_time: 0.0472 s/iter total_throughput: 1877.35 samples/s lr: 1.63e-05 [09/21 10:27:37] lb.utils.events INFO: eta: 2:53:36 iteration: 356399/375342 consumed_samples: 364953600 total_loss: 0.3399 time: 0.5454 s/iter data_time: 0.0462 s/iter total_throughput: 1877.35 samples/s lr: 1.62e-05 [09/21 10:28:32] lb.utils.events INFO: eta: 2:52:43 iteration: 356499/375342 consumed_samples: 365056000 total_loss: 0.3404 time: 0.5455 s/iter data_time: 0.0495 s/iter total_throughput: 1877.35 samples/s lr: 1.61e-05 [09/21 10:29:28] lb.utils.events INFO: eta: 2:51:53 iteration: 356599/375342 consumed_samples: 365158400 total_loss: 0.3418 time: 0.5455 s/iter data_time: 0.0509 s/iter total_throughput: 1877.34 samples/s lr: 1.61e-05 [09/21 10:30:23] lb.utils.events INFO: eta: 2:51:03 iteration: 356699/375342 consumed_samples: 365260800 total_loss: 0.3405 time: 0.5455 s/iter data_time: 0.0522 s/iter total_throughput: 1877.33 samples/s lr: 1.60e-05 [09/21 10:31:18] lb.utils.events INFO: eta: 2:50:13 iteration: 356799/375342 consumed_samples: 365363200 total_loss: 0.3417 time: 0.5455 s/iter data_time: 0.0494 s/iter total_throughput: 1877.33 samples/s lr: 1.59e-05 [09/21 10:32:13] lb.utils.events INFO: eta: 2:49:14 iteration: 356899/375342 consumed_samples: 365465600 total_loss: 0.3442 time: 0.5455 s/iter data_time: 0.0512 s/iter total_throughput: 1877.32 samples/s lr: 1.59e-05 [09/21 10:33:08] lb.utils.events INFO: eta: 2:48:17 iteration: 356999/375342 consumed_samples: 365568000 total_loss: 0.3464 time: 0.5455 s/iter data_time: 0.0496 s/iter total_throughput: 1877.32 samples/s lr: 1.58e-05 [09/21 10:34:03] lb.utils.events INFO: eta: 2:47:18 iteration: 357099/375342 consumed_samples: 365670400 total_loss: 0.3443 time: 0.5455 s/iter data_time: 0.0522 s/iter total_throughput: 1877.31 samples/s lr: 1.58e-05 [09/21 10:34:58] lb.utils.events INFO: eta: 2:46:22 iteration: 357199/375342 consumed_samples: 365772800 total_loss: 0.3421 time: 0.5455 s/iter data_time: 0.0539 s/iter total_throughput: 1877.31 samples/s lr: 1.57e-05 [09/21 10:35:53] lb.utils.events INFO: eta: 2:45:29 iteration: 357299/375342 consumed_samples: 365875200 total_loss: 0.3393 time: 0.5455 s/iter data_time: 0.0511 s/iter total_throughput: 1877.31 samples/s lr: 1.56e-05 [09/21 10:36:48] lb.utils.events INFO: eta: 2:44:28 iteration: 357399/375342 consumed_samples: 365977600 total_loss: 0.3426 time: 0.5455 s/iter data_time: 0.0500 s/iter total_throughput: 1877.30 samples/s lr: 1.56e-05 [09/21 10:37:43] lb.utils.events INFO: eta: 2:43:31 iteration: 357499/375342 consumed_samples: 366080000 total_loss: 0.3465 time: 0.5455 s/iter data_time: 0.0523 s/iter total_throughput: 1877.30 samples/s lr: 1.55e-05 [09/21 10:38:38] lb.utils.events INFO: eta: 2:42:30 iteration: 357599/375342 consumed_samples: 366182400 total_loss: 0.3462 time: 0.5455 s/iter data_time: 0.0523 s/iter total_throughput: 1877.30 samples/s lr: 1.54e-05 [09/21 10:39:32] lb.utils.events INFO: eta: 2:41:30 iteration: 357699/375342 consumed_samples: 366284800 total_loss: 0.3401 time: 0.5455 s/iter data_time: 0.0525 s/iter total_throughput: 1877.30 samples/s lr: 1.54e-05 [09/21 10:40:27] lb.utils.events INFO: eta: 2:40:25 iteration: 357799/375342 consumed_samples: 366387200 total_loss: 0.344 time: 0.5455 s/iter data_time: 0.0515 s/iter total_throughput: 1877.29 samples/s lr: 1.53e-05 [09/21 10:41:22] lb.utils.events INFO: eta: 2:39:26 iteration: 357899/375342 consumed_samples: 366489600 total_loss: 0.3438 time: 0.5455 s/iter data_time: 0.0525 s/iter total_throughput: 1877.29 samples/s lr: 1.53e-05 [09/21 10:42:16] lb.utils.events INFO: eta: 2:38:26 iteration: 357999/375342 consumed_samples: 366592000 total_loss: 0.3342 time: 0.5455 s/iter data_time: 0.0522 s/iter total_throughput: 1877.29 samples/s lr: 1.52e-05 [09/21 10:43:11] lb.utils.events INFO: eta: 2:37:23 iteration: 358099/375342 consumed_samples: 366694400 total_loss: 0.3449 time: 0.5455 s/iter data_time: 0.0517 s/iter total_throughput: 1877.29 samples/s lr: 1.51e-05 [09/21 10:44:06] lb.utils.events INFO: eta: 2:36:23 iteration: 358199/375342 consumed_samples: 366796800 total_loss: 0.3534 time: 0.5455 s/iter data_time: 0.0515 s/iter total_throughput: 1877.29 samples/s lr: 1.51e-05 [09/21 10:45:00] lb.utils.events INFO: eta: 2:35:22 iteration: 358299/375342 consumed_samples: 366899200 total_loss: 0.3442 time: 0.5455 s/iter data_time: 0.0517 s/iter total_throughput: 1877.29 samples/s lr: 1.50e-05 [09/21 10:45:55] lb.utils.events INFO: eta: 2:34:24 iteration: 358399/375342 consumed_samples: 367001600 total_loss: 0.3428 time: 0.5455 s/iter data_time: 0.0477 s/iter total_throughput: 1877.29 samples/s lr: 1.50e-05 [09/21 10:46:50] lb.utils.events INFO: eta: 2:33:27 iteration: 358499/375342 consumed_samples: 367104000 total_loss: 0.3498 time: 0.5455 s/iter data_time: 0.0482 s/iter total_throughput: 1877.29 samples/s lr: 1.49e-05 [09/21 10:47:45] lb.utils.events INFO: eta: 2:32:32 iteration: 358599/375342 consumed_samples: 367206400 total_loss: 0.3526 time: 0.5455 s/iter data_time: 0.0503 s/iter total_throughput: 1877.29 samples/s lr: 1.49e-05 [09/21 10:48:39] lb.utils.events INFO: eta: 2:31:37 iteration: 358699/375342 consumed_samples: 367308800 total_loss: 0.3473 time: 0.5455 s/iter data_time: 0.0508 s/iter total_throughput: 1877.28 samples/s lr: 1.48e-05 [09/21 10:49:35] lb.utils.events INFO: eta: 2:30:46 iteration: 358799/375342 consumed_samples: 367411200 total_loss: 0.3403 time: 0.5455 s/iter data_time: 0.0479 s/iter total_throughput: 1877.27 samples/s lr: 1.47e-05 [09/21 10:50:30] lb.utils.events INFO: eta: 2:29:54 iteration: 358899/375342 consumed_samples: 367513600 total_loss: 0.3461 time: 0.5455 s/iter data_time: 0.0440 s/iter total_throughput: 1877.27 samples/s lr: 1.47e-05 [09/21 10:51:25] lb.utils.events INFO: eta: 2:29:04 iteration: 358999/375342 consumed_samples: 367616000 total_loss: 0.3487 time: 0.5455 s/iter data_time: 0.0425 s/iter total_throughput: 1877.27 samples/s lr: 1.46e-05 [09/21 10:52:20] lb.utils.events INFO: eta: 2:28:14 iteration: 359099/375342 consumed_samples: 367718400 total_loss: 0.3474 time: 0.5455 s/iter data_time: 0.0435 s/iter total_throughput: 1877.26 samples/s lr: 1.46e-05 [09/21 10:53:15] lb.utils.events INFO: eta: 2:27:27 iteration: 359199/375342 consumed_samples: 367820800 total_loss: 0.3441 time: 0.5455 s/iter data_time: 0.0457 s/iter total_throughput: 1877.26 samples/s lr: 1.45e-05 [09/21 10:54:09] lb.utils.events INFO: eta: 2:26:39 iteration: 359299/375342 consumed_samples: 367923200 total_loss: 0.3435 time: 0.5455 s/iter data_time: 0.0446 s/iter total_throughput: 1877.26 samples/s lr: 1.45e-05 [09/21 10:55:04] lb.utils.events INFO: eta: 2:25:47 iteration: 359399/375342 consumed_samples: 368025600 total_loss: 0.3443 time: 0.5455 s/iter data_time: 0.0438 s/iter total_throughput: 1877.26 samples/s lr: 1.44e-05 [09/21 10:55:59] lb.utils.events INFO: eta: 2:24:54 iteration: 359499/375342 consumed_samples: 368128000 total_loss: 0.3485 time: 0.5455 s/iter data_time: 0.0487 s/iter total_throughput: 1877.25 samples/s lr: 1.43e-05 [09/21 10:56:54] lb.utils.events INFO: eta: 2:24:03 iteration: 359599/375342 consumed_samples: 368230400 total_loss: 0.3509 time: 0.5455 s/iter data_time: 0.0472 s/iter total_throughput: 1877.25 samples/s lr: 1.43e-05 [09/21 10:57:49] lb.utils.events INFO: eta: 2:23:09 iteration: 359699/375342 consumed_samples: 368332800 total_loss: 0.3462 time: 0.5455 s/iter data_time: 0.0453 s/iter total_throughput: 1877.25 samples/s lr: 1.42e-05 [09/21 10:58:44] lb.utils.events INFO: eta: 2:22:16 iteration: 359799/375342 consumed_samples: 368435200 total_loss: 0.3431 time: 0.5455 s/iter data_time: 0.0481 s/iter total_throughput: 1877.24 samples/s lr: 1.42e-05 [09/21 10:59:39] lb.utils.events INFO: eta: 2:21:22 iteration: 359899/375342 consumed_samples: 368537600 total_loss: 0.3387 time: 0.5455 s/iter data_time: 0.0461 s/iter total_throughput: 1877.24 samples/s lr: 1.41e-05 [09/21 11:00:34] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0359999 [09/21 11:00:35] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 11:00:35] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 11:00:39] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0196 s/iter. Inference: 0.2615 s/iter. Eval: 0.0022 s/iter. Total: 0.2833 s/iter. ETA=0:00:10 [09/21 11:00:45] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0668 s/iter. Inference: 0.2587 s/iter. Eval: 0.0022 s/iter. Total: 0.3279 s/iter. ETA=0:00:07 [09/21 11:00:50] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0694 s/iter. Inference: 0.2556 s/iter. Eval: 0.0022 s/iter. Total: 0.3275 s/iter. ETA=0:00:01 [09/21 11:00:52] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 11:00:52] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.190373 (0.000284 s / iter per device, on 8 devices) [09/21 11:00:52] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000223 s / iter per device, on 8 devices) [09/21 11:00:52] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 11:00:52] lb.evaluation.utils INFO: copypaste: Acc@1=80.036 [09/21 11:00:52] lb.evaluation.utils INFO: copypaste: Acc@5=94.71199999999999 [09/21 11:00:52] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 80.03600, better than last best score 79.95000 @ iteration 344999. [09/21 11:00:52] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 11:00:53] lb.utils.events INFO: eta: 2:20:33 iteration: 359999/375342 consumed_samples: 368640000 total_loss: 0.341 time: 0.5455 s/iter data_time: 0.0480 s/iter total_throughput: 1877.23 samples/s lr: 1.41e-05 [09/21 11:01:48] lb.utils.events INFO: eta: 2:19:48 iteration: 360099/375342 consumed_samples: 368742400 total_loss: 0.3451 time: 0.5455 s/iter data_time: 0.0538 s/iter total_throughput: 1877.23 samples/s lr: 1.40e-05 [09/21 11:02:43] lb.utils.events INFO: eta: 2:18:57 iteration: 360199/375342 consumed_samples: 368844800 total_loss: 0.3479 time: 0.5455 s/iter data_time: 0.0477 s/iter total_throughput: 1877.22 samples/s lr: 1.40e-05 [09/21 11:03:38] lb.utils.events INFO: eta: 2:18:05 iteration: 360299/375342 consumed_samples: 368947200 total_loss: 0.3477 time: 0.5455 s/iter data_time: 0.0518 s/iter total_throughput: 1877.21 samples/s lr: 1.39e-05 [09/21 11:04:34] lb.utils.events INFO: eta: 2:17:16 iteration: 360399/375342 consumed_samples: 369049600 total_loss: 0.3437 time: 0.5455 s/iter data_time: 0.0554 s/iter total_throughput: 1877.21 samples/s lr: 1.39e-05 [09/21 11:05:29] lb.utils.events INFO: eta: 2:16:24 iteration: 360499/375342 consumed_samples: 369152000 total_loss: 0.3389 time: 0.5455 s/iter data_time: 0.0519 s/iter total_throughput: 1877.20 samples/s lr: 1.38e-05 [09/21 11:06:24] lb.utils.events INFO: eta: 2:15:30 iteration: 360599/375342 consumed_samples: 369254400 total_loss: 0.338 time: 0.5455 s/iter data_time: 0.0549 s/iter total_throughput: 1877.20 samples/s lr: 1.38e-05 [09/21 11:07:19] lb.utils.events INFO: eta: 2:14:37 iteration: 360699/375342 consumed_samples: 369356800 total_loss: 0.3453 time: 0.5455 s/iter data_time: 0.0527 s/iter total_throughput: 1877.19 samples/s lr: 1.37e-05 [09/21 11:08:14] lb.utils.events INFO: eta: 2:13:39 iteration: 360799/375342 consumed_samples: 369459200 total_loss: 0.3483 time: 0.5455 s/iter data_time: 0.0541 s/iter total_throughput: 1877.19 samples/s lr: 1.37e-05 [09/21 11:09:09] lb.utils.events INFO: eta: 2:12:42 iteration: 360899/375342 consumed_samples: 369561600 total_loss: 0.3441 time: 0.5455 s/iter data_time: 0.0524 s/iter total_throughput: 1877.18 samples/s lr: 1.36e-05 [09/21 11:10:04] lb.utils.events INFO: eta: 2:11:42 iteration: 360999/375342 consumed_samples: 369664000 total_loss: 0.341 time: 0.5455 s/iter data_time: 0.0524 s/iter total_throughput: 1877.18 samples/s lr: 1.36e-05 [09/21 11:10:59] lb.utils.events INFO: eta: 2:10:40 iteration: 361099/375342 consumed_samples: 369766400 total_loss: 0.3402 time: 0.5455 s/iter data_time: 0.0526 s/iter total_throughput: 1877.18 samples/s lr: 1.35e-05 [09/21 11:11:53] lb.utils.events INFO: eta: 2:09:39 iteration: 361199/375342 consumed_samples: 369868800 total_loss: 0.3405 time: 0.5455 s/iter data_time: 0.0524 s/iter total_throughput: 1877.18 samples/s lr: 1.35e-05 [09/21 11:12:48] lb.utils.events INFO: eta: 2:08:35 iteration: 361299/375342 consumed_samples: 369971200 total_loss: 0.3436 time: 0.5455 s/iter data_time: 0.0511 s/iter total_throughput: 1877.18 samples/s lr: 1.34e-05 [09/21 11:13:43] lb.utils.events INFO: eta: 2:07:31 iteration: 361399/375342 consumed_samples: 370073600 total_loss: 0.3491 time: 0.5455 s/iter data_time: 0.0524 s/iter total_throughput: 1877.18 samples/s lr: 1.34e-05 [09/21 11:14:37] lb.utils.events INFO: eta: 2:06:29 iteration: 361499/375342 consumed_samples: 370176000 total_loss: 0.3408 time: 0.5455 s/iter data_time: 0.0524 s/iter total_throughput: 1877.18 samples/s lr: 1.33e-05 [09/21 11:15:32] lb.utils.events INFO: eta: 2:05:27 iteration: 361599/375342 consumed_samples: 370278400 total_loss: 0.3405 time: 0.5455 s/iter data_time: 0.0515 s/iter total_throughput: 1877.18 samples/s lr: 1.33e-05 [09/21 11:16:26] lb.utils.events INFO: eta: 2:04:25 iteration: 361699/375342 consumed_samples: 370380800 total_loss: 0.3426 time: 0.5455 s/iter data_time: 0.0504 s/iter total_throughput: 1877.18 samples/s lr: 1.32e-05 [09/21 11:17:21] lb.utils.events INFO: eta: 2:03:26 iteration: 361799/375342 consumed_samples: 370483200 total_loss: 0.3454 time: 0.5455 s/iter data_time: 0.0510 s/iter total_throughput: 1877.18 samples/s lr: 1.32e-05 [09/21 11:18:16] lb.utils.events INFO: eta: 2:02:26 iteration: 361899/375342 consumed_samples: 370585600 total_loss: 0.3398 time: 0.5455 s/iter data_time: 0.0489 s/iter total_throughput: 1877.17 samples/s lr: 1.31e-05 [09/21 11:19:10] lb.utils.events INFO: eta: 2:01:28 iteration: 361999/375342 consumed_samples: 370688000 total_loss: 0.3359 time: 0.5455 s/iter data_time: 0.0484 s/iter total_throughput: 1877.17 samples/s lr: 1.31e-05 [09/21 11:20:05] lb.utils.events INFO: eta: 2:00:31 iteration: 362099/375342 consumed_samples: 370790400 total_loss: 0.3406 time: 0.5455 s/iter data_time: 0.0477 s/iter total_throughput: 1877.17 samples/s lr: 1.30e-05 [09/21 11:21:00] lb.utils.events INFO: eta: 1:59:34 iteration: 362199/375342 consumed_samples: 370892800 total_loss: 0.3433 time: 0.5455 s/iter data_time: 0.0540 s/iter total_throughput: 1877.17 samples/s lr: 1.30e-05 [09/21 11:21:55] lb.utils.events INFO: eta: 1:58:42 iteration: 362299/375342 consumed_samples: 370995200 total_loss: 0.3442 time: 0.5455 s/iter data_time: 0.0510 s/iter total_throughput: 1877.17 samples/s lr: 1.29e-05 [09/21 11:22:50] lb.utils.events INFO: eta: 1:57:51 iteration: 362399/375342 consumed_samples: 371097600 total_loss: 0.345 time: 0.5455 s/iter data_time: 0.0522 s/iter total_throughput: 1877.16 samples/s lr: 1.29e-05 [09/21 11:23:45] lb.utils.events INFO: eta: 1:56:59 iteration: 362499/375342 consumed_samples: 371200000 total_loss: 0.3457 time: 0.5455 s/iter data_time: 0.0543 s/iter total_throughput: 1877.16 samples/s lr: 1.29e-05 [09/21 11:24:39] lb.utils.events INFO: eta: 1:56:08 iteration: 362599/375342 consumed_samples: 371302400 total_loss: 0.3441 time: 0.5455 s/iter data_time: 0.0556 s/iter total_throughput: 1877.16 samples/s lr: 1.28e-05 [09/21 11:25:34] lb.utils.events INFO: eta: 1:55:17 iteration: 362699/375342 consumed_samples: 371404800 total_loss: 0.3451 time: 0.5455 s/iter data_time: 0.0534 s/iter total_throughput: 1877.16 samples/s lr: 1.28e-05 [09/21 11:26:29] lb.utils.events INFO: eta: 1:54:28 iteration: 362799/375342 consumed_samples: 371507200 total_loss: 0.346 time: 0.5455 s/iter data_time: 0.0526 s/iter total_throughput: 1877.15 samples/s lr: 1.27e-05 [09/21 11:27:24] lb.utils.events INFO: eta: 1:53:35 iteration: 362899/375342 consumed_samples: 371609600 total_loss: 0.3395 time: 0.5455 s/iter data_time: 0.0525 s/iter total_throughput: 1877.15 samples/s lr: 1.27e-05 [09/21 11:28:19] lb.utils.events INFO: eta: 1:52:42 iteration: 362999/375342 consumed_samples: 371712000 total_loss: 0.3366 time: 0.5455 s/iter data_time: 0.0544 s/iter total_throughput: 1877.15 samples/s lr: 1.26e-05 [09/21 11:29:14] lb.utils.events INFO: eta: 1:51:53 iteration: 363099/375342 consumed_samples: 371814400 total_loss: 0.3465 time: 0.5455 s/iter data_time: 0.0553 s/iter total_throughput: 1877.14 samples/s lr: 1.26e-05 [09/21 11:30:09] lb.utils.events INFO: eta: 1:51:00 iteration: 363199/375342 consumed_samples: 371916800 total_loss: 0.3489 time: 0.5455 s/iter data_time: 0.0528 s/iter total_throughput: 1877.14 samples/s lr: 1.26e-05 [09/21 11:31:04] lb.utils.events INFO: eta: 1:50:07 iteration: 363299/375342 consumed_samples: 372019200 total_loss: 0.3426 time: 0.5455 s/iter data_time: 0.0449 s/iter total_throughput: 1877.14 samples/s lr: 1.25e-05 [09/21 11:31:59] lb.utils.events INFO: eta: 1:49:17 iteration: 363399/375342 consumed_samples: 372121600 total_loss: 0.3459 time: 0.5455 s/iter data_time: 0.0492 s/iter total_throughput: 1877.13 samples/s lr: 1.25e-05 [09/21 11:32:54] lb.utils.events INFO: eta: 1:48:27 iteration: 363499/375342 consumed_samples: 372224000 total_loss: 0.3457 time: 0.5455 s/iter data_time: 0.0533 s/iter total_throughput: 1877.13 samples/s lr: 1.24e-05 [09/21 11:33:49] lb.utils.events INFO: eta: 1:47:33 iteration: 363599/375342 consumed_samples: 372326400 total_loss: 0.3438 time: 0.5455 s/iter data_time: 0.0520 s/iter total_throughput: 1877.12 samples/s lr: 1.24e-05 [09/21 11:34:44] lb.utils.events INFO: eta: 1:46:41 iteration: 363699/375342 consumed_samples: 372428800 total_loss: 0.3485 time: 0.5455 s/iter data_time: 0.0479 s/iter total_throughput: 1877.12 samples/s lr: 1.23e-05 [09/21 11:35:39] lb.utils.events INFO: eta: 1:45:48 iteration: 363799/375342 consumed_samples: 372531200 total_loss: 0.3468 time: 0.5455 s/iter data_time: 0.0519 s/iter total_throughput: 1877.11 samples/s lr: 1.23e-05 [09/21 11:36:34] lb.utils.events INFO: eta: 1:44:55 iteration: 363899/375342 consumed_samples: 372633600 total_loss: 0.3478 time: 0.5455 s/iter data_time: 0.0507 s/iter total_throughput: 1877.11 samples/s lr: 1.23e-05 [09/21 11:37:29] lb.utils.events INFO: eta: 1:44:00 iteration: 363999/375342 consumed_samples: 372736000 total_loss: 0.3492 time: 0.5455 s/iter data_time: 0.0514 s/iter total_throughput: 1877.10 samples/s lr: 1.22e-05 [09/21 11:38:24] lb.utils.events INFO: eta: 1:43:06 iteration: 364099/375342 consumed_samples: 372838400 total_loss: 0.346 time: 0.5455 s/iter data_time: 0.0534 s/iter total_throughput: 1877.10 samples/s lr: 1.22e-05 [09/21 11:39:19] lb.utils.events INFO: eta: 1:42:11 iteration: 364199/375342 consumed_samples: 372940800 total_loss: 0.343 time: 0.5455 s/iter data_time: 0.0544 s/iter total_throughput: 1877.10 samples/s lr: 1.22e-05 [09/21 11:40:14] lb.utils.events INFO: eta: 1:41:13 iteration: 364299/375342 consumed_samples: 373043200 total_loss: 0.3434 time: 0.5455 s/iter data_time: 0.0500 s/iter total_throughput: 1877.09 samples/s lr: 1.21e-05 [09/21 11:41:09] lb.utils.events INFO: eta: 1:40:12 iteration: 364399/375342 consumed_samples: 373145600 total_loss: 0.3424 time: 0.5455 s/iter data_time: 0.0524 s/iter total_throughput: 1877.09 samples/s lr: 1.21e-05 [09/21 11:42:03] lb.utils.events INFO: eta: 1:39:11 iteration: 364499/375342 consumed_samples: 373248000 total_loss: 0.3402 time: 0.5455 s/iter data_time: 0.0531 s/iter total_throughput: 1877.09 samples/s lr: 1.20e-05 [09/21 11:42:58] lb.utils.events INFO: eta: 1:38:12 iteration: 364599/375342 consumed_samples: 373350400 total_loss: 0.3418 time: 0.5455 s/iter data_time: 0.0519 s/iter total_throughput: 1877.09 samples/s lr: 1.20e-05 [09/21 11:43:53] lb.utils.events INFO: eta: 1:37:12 iteration: 364699/375342 consumed_samples: 373452800 total_loss: 0.3389 time: 0.5455 s/iter data_time: 0.0511 s/iter total_throughput: 1877.09 samples/s lr: 1.20e-05 [09/21 11:44:47] lb.utils.events INFO: eta: 1:36:11 iteration: 364799/375342 consumed_samples: 373555200 total_loss: 0.3395 time: 0.5455 s/iter data_time: 0.0500 s/iter total_throughput: 1877.09 samples/s lr: 1.19e-05 [09/21 11:45:42] lb.utils.events INFO: eta: 1:35:11 iteration: 364899/375342 consumed_samples: 373657600 total_loss: 0.3364 time: 0.5455 s/iter data_time: 0.0520 s/iter total_throughput: 1877.09 samples/s lr: 1.19e-05 [09/21 11:46:36] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0364999 [09/21 11:46:37] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 11:46:37] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 11:46:41] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0468 s/iter. Inference: 0.2476 s/iter. Eval: 0.0027 s/iter. Total: 0.2972 s/iter. ETA=0:00:10 [09/21 11:46:47] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0725 s/iter. Inference: 0.2603 s/iter. Eval: 0.0026 s/iter. Total: 0.3357 s/iter. ETA=0:00:07 [09/21 11:46:52] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0742 s/iter. Inference: 0.2549 s/iter. Eval: 0.0026 s/iter. Total: 0.3320 s/iter. ETA=0:00:01 [09/21 11:46:54] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 11:46:54] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.321885 (0.000286 s / iter per device, on 8 devices) [09/21 11:46:54] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000222 s / iter per device, on 8 devices) [09/21 11:46:54] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 11:46:54] lb.evaluation.utils INFO: copypaste: Acc@1=80.01599999999999 [09/21 11:46:54] lb.evaluation.utils INFO: copypaste: Acc@5=94.742 [09/21 11:46:54] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 80.01600, not better than best score 80.03600 @ iteration 359999. [09/21 11:46:54] lb.utils.events INFO: eta: 1:34:13 iteration: 364999/375342 consumed_samples: 373760000 total_loss: 0.3355 time: 0.5455 s/iter data_time: 0.0528 s/iter total_throughput: 1877.09 samples/s lr: 1.19e-05 [09/21 11:47:48] lb.utils.events INFO: eta: 1:33:12 iteration: 365099/375342 consumed_samples: 373862400 total_loss: 0.3386 time: 0.5455 s/iter data_time: 0.0507 s/iter total_throughput: 1877.09 samples/s lr: 1.18e-05 [09/21 11:48:43] lb.utils.events INFO: eta: 1:32:14 iteration: 365199/375342 consumed_samples: 373964800 total_loss: 0.3425 time: 0.5455 s/iter data_time: 0.0522 s/iter total_throughput: 1877.10 samples/s lr: 1.18e-05 [09/21 11:49:37] lb.utils.events INFO: eta: 1:31:16 iteration: 365299/375342 consumed_samples: 374067200 total_loss: 0.3505 time: 0.5455 s/iter data_time: 0.0502 s/iter total_throughput: 1877.09 samples/s lr: 1.17e-05 [09/21 11:50:32] lb.utils.events INFO: eta: 1:30:21 iteration: 365399/375342 consumed_samples: 374169600 total_loss: 0.3396 time: 0.5455 s/iter data_time: 0.0478 s/iter total_throughput: 1877.09 samples/s lr: 1.17e-05 [09/21 11:51:26] lb.utils.events INFO: eta: 1:29:26 iteration: 365499/375342 consumed_samples: 374272000 total_loss: 0.3377 time: 0.5455 s/iter data_time: 0.0485 s/iter total_throughput: 1877.10 samples/s lr: 1.17e-05 [09/21 11:52:21] lb.utils.events INFO: eta: 1:28:29 iteration: 365599/375342 consumed_samples: 374374400 total_loss: 0.34 time: 0.5455 s/iter data_time: 0.0495 s/iter total_throughput: 1877.10 samples/s lr: 1.16e-05 [09/21 11:53:16] lb.utils.events INFO: eta: 1:27:35 iteration: 365699/375342 consumed_samples: 374476800 total_loss: 0.3396 time: 0.5455 s/iter data_time: 0.0547 s/iter total_throughput: 1877.09 samples/s lr: 1.16e-05 [09/21 11:54:11] lb.utils.events INFO: eta: 1:26:42 iteration: 365799/375342 consumed_samples: 374579200 total_loss: 0.3394 time: 0.5455 s/iter data_time: 0.0534 s/iter total_throughput: 1877.09 samples/s lr: 1.16e-05 [09/21 11:55:06] lb.utils.events INFO: eta: 1:25:51 iteration: 365899/375342 consumed_samples: 374681600 total_loss: 0.3385 time: 0.5455 s/iter data_time: 0.0550 s/iter total_throughput: 1877.09 samples/s lr: 1.15e-05 [09/21 11:56:01] lb.utils.events INFO: eta: 1:25:00 iteration: 365999/375342 consumed_samples: 374784000 total_loss: 0.3459 time: 0.5455 s/iter data_time: 0.0552 s/iter total_throughput: 1877.08 samples/s lr: 1.15e-05 [09/21 11:56:56] lb.utils.events INFO: eta: 1:24:10 iteration: 366099/375342 consumed_samples: 374886400 total_loss: 0.3506 time: 0.5455 s/iter data_time: 0.0546 s/iter total_throughput: 1877.08 samples/s lr: 1.15e-05 [09/21 11:57:50] lb.utils.events INFO: eta: 1:23:20 iteration: 366199/375342 consumed_samples: 374988800 total_loss: 0.3444 time: 0.5455 s/iter data_time: 0.0509 s/iter total_throughput: 1877.08 samples/s lr: 1.14e-05 [09/21 11:58:45] lb.utils.events INFO: eta: 1:22:30 iteration: 366299/375342 consumed_samples: 375091200 total_loss: 0.3432 time: 0.5455 s/iter data_time: 0.0520 s/iter total_throughput: 1877.07 samples/s lr: 1.14e-05 [09/21 11:59:40] lb.utils.events INFO: eta: 1:21:39 iteration: 366399/375342 consumed_samples: 375193600 total_loss: 0.3387 time: 0.5455 s/iter data_time: 0.0480 s/iter total_throughput: 1877.07 samples/s lr: 1.14e-05 [09/21 12:00:35] lb.utils.events INFO: eta: 1:20:49 iteration: 366499/375342 consumed_samples: 375296000 total_loss: 0.3362 time: 0.5455 s/iter data_time: 0.0520 s/iter total_throughput: 1877.07 samples/s lr: 1.14e-05 [09/21 12:01:30] lb.utils.events INFO: eta: 1:19:57 iteration: 366599/375342 consumed_samples: 375398400 total_loss: 0.3424 time: 0.5455 s/iter data_time: 0.0515 s/iter total_throughput: 1877.07 samples/s lr: 1.13e-05 [09/21 12:02:25] lb.utils.events INFO: eta: 1:19:02 iteration: 366699/375342 consumed_samples: 375500800 total_loss: 0.3443 time: 0.5455 s/iter data_time: 0.0515 s/iter total_throughput: 1877.06 samples/s lr: 1.13e-05 [09/21 12:03:20] lb.utils.events INFO: eta: 1:18:06 iteration: 366799/375342 consumed_samples: 375603200 total_loss: 0.3496 time: 0.5455 s/iter data_time: 0.0483 s/iter total_throughput: 1877.06 samples/s lr: 1.13e-05 [09/21 12:04:15] lb.utils.events INFO: eta: 1:17:14 iteration: 366899/375342 consumed_samples: 375705600 total_loss: 0.3465 time: 0.5455 s/iter data_time: 0.0443 s/iter total_throughput: 1877.05 samples/s lr: 1.12e-05 [09/21 12:05:10] lb.utils.events INFO: eta: 1:16:21 iteration: 366999/375342 consumed_samples: 375808000 total_loss: 0.3453 time: 0.5455 s/iter data_time: 0.0488 s/iter total_throughput: 1877.05 samples/s lr: 1.12e-05 [09/21 12:06:05] lb.utils.events INFO: eta: 1:15:29 iteration: 367099/375342 consumed_samples: 375910400 total_loss: 0.3461 time: 0.5455 s/iter data_time: 0.0524 s/iter total_throughput: 1877.05 samples/s lr: 1.12e-05 [09/21 12:07:00] lb.utils.events INFO: eta: 1:14:35 iteration: 367199/375342 consumed_samples: 376012800 total_loss: 0.3483 time: 0.5455 s/iter data_time: 0.0515 s/iter total_throughput: 1877.04 samples/s lr: 1.11e-05 [09/21 12:07:55] lb.utils.events INFO: eta: 1:13:41 iteration: 367299/375342 consumed_samples: 376115200 total_loss: 0.3454 time: 0.5455 s/iter data_time: 0.0532 s/iter total_throughput: 1877.04 samples/s lr: 1.11e-05 [09/21 12:08:50] lb.utils.events INFO: eta: 1:12:44 iteration: 367399/375342 consumed_samples: 376217600 total_loss: 0.3437 time: 0.5455 s/iter data_time: 0.0528 s/iter total_throughput: 1877.04 samples/s lr: 1.11e-05 [09/21 12:09:45] lb.utils.events INFO: eta: 1:11:48 iteration: 367499/375342 consumed_samples: 376320000 total_loss: 0.3433 time: 0.5455 s/iter data_time: 0.0512 s/iter total_throughput: 1877.03 samples/s lr: 1.11e-05 [09/21 12:10:40] lb.utils.events INFO: eta: 1:10:53 iteration: 367599/375342 consumed_samples: 376422400 total_loss: 0.3449 time: 0.5455 s/iter data_time: 0.0481 s/iter total_throughput: 1877.03 samples/s lr: 1.10e-05 [09/21 12:11:34] lb.utils.events INFO: eta: 1:09:56 iteration: 367699/375342 consumed_samples: 376524800 total_loss: 0.3463 time: 0.5455 s/iter data_time: 0.0506 s/iter total_throughput: 1877.03 samples/s lr: 1.10e-05 [09/21 12:12:29] lb.utils.events INFO: eta: 1:08:59 iteration: 367799/375342 consumed_samples: 376627200 total_loss: 0.3457 time: 0.5455 s/iter data_time: 0.0509 s/iter total_throughput: 1877.03 samples/s lr: 1.10e-05 [09/21 12:13:24] lb.utils.events INFO: eta: 1:08:02 iteration: 367899/375342 consumed_samples: 376729600 total_loss: 0.3418 time: 0.5455 s/iter data_time: 0.0512 s/iter total_throughput: 1877.03 samples/s lr: 1.10e-05 [09/21 12:14:18] lb.utils.events INFO: eta: 1:07:04 iteration: 367999/375342 consumed_samples: 376832000 total_loss: 0.338 time: 0.5455 s/iter data_time: 0.0526 s/iter total_throughput: 1877.03 samples/s lr: 1.09e-05 [09/21 12:15:13] lb.utils.events INFO: eta: 1:06:06 iteration: 368099/375342 consumed_samples: 376934400 total_loss: 0.3403 time: 0.5455 s/iter data_time: 0.0533 s/iter total_throughput: 1877.03 samples/s lr: 1.09e-05 [09/21 12:16:08] lb.utils.events INFO: eta: 1:05:08 iteration: 368199/375342 consumed_samples: 377036800 total_loss: 0.3433 time: 0.5455 s/iter data_time: 0.0525 s/iter total_throughput: 1877.03 samples/s lr: 1.09e-05 [09/21 12:17:02] lb.utils.events INFO: eta: 1:04:10 iteration: 368299/375342 consumed_samples: 377139200 total_loss: 0.3473 time: 0.5455 s/iter data_time: 0.0514 s/iter total_throughput: 1877.03 samples/s lr: 1.09e-05 [09/21 12:17:57] lb.utils.events INFO: eta: 1:03:13 iteration: 368399/375342 consumed_samples: 377241600 total_loss: 0.3451 time: 0.5455 s/iter data_time: 0.0527 s/iter total_throughput: 1877.03 samples/s lr: 1.08e-05 [09/21 12:18:51] lb.utils.events INFO: eta: 1:02:16 iteration: 368499/375342 consumed_samples: 377344000 total_loss: 0.3421 time: 0.5455 s/iter data_time: 0.0514 s/iter total_throughput: 1877.03 samples/s lr: 1.08e-05 [09/21 12:19:46] lb.utils.events INFO: eta: 1:01:20 iteration: 368599/375342 consumed_samples: 377446400 total_loss: 0.3419 time: 0.5455 s/iter data_time: 0.0521 s/iter total_throughput: 1877.03 samples/s lr: 1.08e-05 [09/21 12:20:40] lb.utils.events INFO: eta: 1:00:24 iteration: 368699/375342 consumed_samples: 377548800 total_loss: 0.3405 time: 0.5455 s/iter data_time: 0.0518 s/iter total_throughput: 1877.03 samples/s lr: 1.08e-05 [09/21 12:21:35] lb.utils.events INFO: eta: 0:59:28 iteration: 368799/375342 consumed_samples: 377651200 total_loss: 0.3383 time: 0.5455 s/iter data_time: 0.0495 s/iter total_throughput: 1877.03 samples/s lr: 1.07e-05 [09/21 12:22:29] lb.utils.events INFO: eta: 0:58:33 iteration: 368899/375342 consumed_samples: 377753600 total_loss: 0.3374 time: 0.5455 s/iter data_time: 0.0498 s/iter total_throughput: 1877.03 samples/s lr: 1.07e-05 [09/21 12:23:24] lb.utils.events INFO: eta: 0:57:38 iteration: 368999/375342 consumed_samples: 377856000 total_loss: 0.3441 time: 0.5455 s/iter data_time: 0.0482 s/iter total_throughput: 1877.03 samples/s lr: 1.07e-05 [09/21 12:24:19] lb.utils.events INFO: eta: 0:56:43 iteration: 369099/375342 consumed_samples: 377958400 total_loss: 0.3506 time: 0.5455 s/iter data_time: 0.0531 s/iter total_throughput: 1877.02 samples/s lr: 1.07e-05 [09/21 12:25:14] lb.utils.events INFO: eta: 0:55:50 iteration: 369199/375342 consumed_samples: 378060800 total_loss: 0.3419 time: 0.5455 s/iter data_time: 0.0540 s/iter total_throughput: 1877.02 samples/s lr: 1.07e-05 [09/21 12:26:09] lb.utils.events INFO: eta: 0:54:57 iteration: 369299/375342 consumed_samples: 378163200 total_loss: 0.3402 time: 0.5455 s/iter data_time: 0.0550 s/iter total_throughput: 1877.02 samples/s lr: 1.06e-05 [09/21 12:27:03] lb.utils.events INFO: eta: 0:54:04 iteration: 369399/375342 consumed_samples: 378265600 total_loss: 0.3433 time: 0.5455 s/iter data_time: 0.0545 s/iter total_throughput: 1877.02 samples/s lr: 1.06e-05 [09/21 12:27:58] lb.utils.events INFO: eta: 0:53:12 iteration: 369499/375342 consumed_samples: 378368000 total_loss: 0.342 time: 0.5455 s/iter data_time: 0.0553 s/iter total_throughput: 1877.02 samples/s lr: 1.06e-05 [09/21 12:28:53] lb.utils.events INFO: eta: 0:52:19 iteration: 369599/375342 consumed_samples: 378470400 total_loss: 0.3398 time: 0.5455 s/iter data_time: 0.0526 s/iter total_throughput: 1877.01 samples/s lr: 1.06e-05 [09/21 12:29:48] lb.utils.events INFO: eta: 0:51:26 iteration: 369699/375342 consumed_samples: 378572800 total_loss: 0.3456 time: 0.5455 s/iter data_time: 0.0535 s/iter total_throughput: 1877.01 samples/s lr: 1.06e-05 [09/21 12:30:43] lb.utils.events INFO: eta: 0:50:34 iteration: 369799/375342 consumed_samples: 378675200 total_loss: 0.3488 time: 0.5455 s/iter data_time: 0.0551 s/iter total_throughput: 1877.01 samples/s lr: 1.05e-05 [09/21 12:31:38] lb.utils.events INFO: eta: 0:49:41 iteration: 369899/375342 consumed_samples: 378777600 total_loss: 0.3407 time: 0.5456 s/iter data_time: 0.0511 s/iter total_throughput: 1877.00 samples/s lr: 1.05e-05 [09/21 12:32:33] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0369999 [09/21 12:32:33] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 12:32:33] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 12:32:38] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0620 s/iter. Inference: 0.2512 s/iter. Eval: 0.0022 s/iter. Total: 0.3153 s/iter. ETA=0:00:11 [09/21 12:32:43] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0754 s/iter. Inference: 0.2616 s/iter. Eval: 0.0022 s/iter. Total: 0.3396 s/iter. ETA=0:00:07 [09/21 12:32:49] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0730 s/iter. Inference: 0.2583 s/iter. Eval: 0.0023 s/iter. Total: 0.3340 s/iter. ETA=0:00:02 [09/21 12:32:51] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 12:32:51] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.407109 (0.000288 s / iter per device, on 8 devices) [09/21 12:32:51] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000226 s / iter per device, on 8 devices) [09/21 12:32:51] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 12:32:51] lb.evaluation.utils INFO: copypaste: Acc@1=80.08800000000001 [09/21 12:32:51] lb.evaluation.utils INFO: copypaste: Acc@5=94.682 [09/21 12:32:51] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 80.08800, better than last best score 80.03600 @ iteration 359999. [09/21 12:32:51] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 12:32:51] lb.utils.events INFO: eta: 0:48:47 iteration: 369999/375342 consumed_samples: 378880000 total_loss: 0.3346 time: 0.5456 s/iter data_time: 0.0544 s/iter total_throughput: 1877.00 samples/s lr: 1.05e-05 [09/21 12:33:46] lb.utils.events INFO: eta: 0:47:54 iteration: 370099/375342 consumed_samples: 378982400 total_loss: 0.3426 time: 0.5456 s/iter data_time: 0.0575 s/iter total_throughput: 1877.00 samples/s lr: 1.05e-05 [09/21 12:34:41] lb.utils.events INFO: eta: 0:47:00 iteration: 370199/375342 consumed_samples: 379084800 total_loss: 0.3449 time: 0.5456 s/iter data_time: 0.0543 s/iter total_throughput: 1877.00 samples/s lr: 1.05e-05 [09/21 12:35:36] lb.utils.events INFO: eta: 0:46:06 iteration: 370299/375342 consumed_samples: 379187200 total_loss: 0.3449 time: 0.5456 s/iter data_time: 0.0495 s/iter total_throughput: 1876.99 samples/s lr: 1.04e-05 [09/21 12:36:31] lb.utils.events INFO: eta: 0:45:12 iteration: 370399/375342 consumed_samples: 379289600 total_loss: 0.3432 time: 0.5456 s/iter data_time: 0.0467 s/iter total_throughput: 1876.99 samples/s lr: 1.04e-05 [09/21 12:37:26] lb.utils.events INFO: eta: 0:44:18 iteration: 370499/375342 consumed_samples: 379392000 total_loss: 0.3442 time: 0.5456 s/iter data_time: 0.0488 s/iter total_throughput: 1876.98 samples/s lr: 1.04e-05 [09/21 12:38:21] lb.utils.events INFO: eta: 0:43:25 iteration: 370599/375342 consumed_samples: 379494400 total_loss: 0.3494 time: 0.5456 s/iter data_time: 0.0497 s/iter total_throughput: 1876.98 samples/s lr: 1.04e-05 [09/21 12:39:16] lb.utils.events INFO: eta: 0:42:30 iteration: 370699/375342 consumed_samples: 379596800 total_loss: 0.3392 time: 0.5456 s/iter data_time: 0.0503 s/iter total_throughput: 1876.98 samples/s lr: 1.04e-05 [09/21 12:40:11] lb.utils.events INFO: eta: 0:41:34 iteration: 370799/375342 consumed_samples: 379699200 total_loss: 0.3378 time: 0.5456 s/iter data_time: 0.0517 s/iter total_throughput: 1876.98 samples/s lr: 1.04e-05 [09/21 12:41:06] lb.utils.events INFO: eta: 0:40:38 iteration: 370899/375342 consumed_samples: 379801600 total_loss: 0.3428 time: 0.5456 s/iter data_time: 0.0523 s/iter total_throughput: 1876.97 samples/s lr: 1.03e-05 [09/21 12:42:00] lb.utils.events INFO: eta: 0:39:42 iteration: 370999/375342 consumed_samples: 379904000 total_loss: 0.3447 time: 0.5456 s/iter data_time: 0.0506 s/iter total_throughput: 1876.97 samples/s lr: 1.03e-05 [09/21 12:42:55] lb.utils.events INFO: eta: 0:38:46 iteration: 371099/375342 consumed_samples: 380006400 total_loss: 0.3456 time: 0.5456 s/iter data_time: 0.0500 s/iter total_throughput: 1876.97 samples/s lr: 1.03e-05 [09/21 12:43:50] lb.utils.events INFO: eta: 0:37:50 iteration: 371199/375342 consumed_samples: 380108800 total_loss: 0.3415 time: 0.5456 s/iter data_time: 0.0517 s/iter total_throughput: 1876.97 samples/s lr: 1.03e-05 [09/21 12:44:44] lb.utils.events INFO: eta: 0:36:55 iteration: 371299/375342 consumed_samples: 380211200 total_loss: 0.3385 time: 0.5456 s/iter data_time: 0.0528 s/iter total_throughput: 1876.97 samples/s lr: 1.03e-05 [09/21 12:45:39] lb.utils.events INFO: eta: 0:35:58 iteration: 371399/375342 consumed_samples: 380313600 total_loss: 0.338 time: 0.5456 s/iter data_time: 0.0522 s/iter total_throughput: 1876.97 samples/s lr: 1.03e-05 [09/21 12:46:34] lb.utils.events INFO: eta: 0:35:03 iteration: 371499/375342 consumed_samples: 380416000 total_loss: 0.3384 time: 0.5456 s/iter data_time: 0.0519 s/iter total_throughput: 1876.97 samples/s lr: 1.03e-05 [09/21 12:47:28] lb.utils.events INFO: eta: 0:34:07 iteration: 371599/375342 consumed_samples: 380518400 total_loss: 0.3457 time: 0.5456 s/iter data_time: 0.0525 s/iter total_throughput: 1876.97 samples/s lr: 1.02e-05 [09/21 12:48:23] lb.utils.events INFO: eta: 0:33:11 iteration: 371699/375342 consumed_samples: 380620800 total_loss: 0.3431 time: 0.5456 s/iter data_time: 0.0508 s/iter total_throughput: 1876.97 samples/s lr: 1.02e-05 [09/21 12:49:17] lb.utils.events INFO: eta: 0:32:15 iteration: 371799/375342 consumed_samples: 380723200 total_loss: 0.342 time: 0.5456 s/iter data_time: 0.0510 s/iter total_throughput: 1876.97 samples/s lr: 1.02e-05 [09/21 12:50:12] lb.utils.events INFO: eta: 0:31:20 iteration: 371899/375342 consumed_samples: 380825600 total_loss: 0.3383 time: 0.5456 s/iter data_time: 0.0513 s/iter total_throughput: 1876.97 samples/s lr: 1.02e-05 [09/21 12:51:06] lb.utils.events INFO: eta: 0:30:24 iteration: 371999/375342 consumed_samples: 380928000 total_loss: 0.3359 time: 0.5456 s/iter data_time: 0.0519 s/iter total_throughput: 1876.97 samples/s lr: 1.02e-05 [09/21 12:52:01] lb.utils.events INFO: eta: 0:29:28 iteration: 372099/375342 consumed_samples: 381030400 total_loss: 0.3425 time: 0.5456 s/iter data_time: 0.0523 s/iter total_throughput: 1876.97 samples/s lr: 1.02e-05 [09/21 12:52:56] lb.utils.events INFO: eta: 0:28:33 iteration: 372199/375342 consumed_samples: 381132800 total_loss: 0.3441 time: 0.5456 s/iter data_time: 0.0489 s/iter total_throughput: 1876.97 samples/s lr: 1.02e-05 [09/21 12:53:50] lb.utils.events INFO: eta: 0:27:38 iteration: 372299/375342 consumed_samples: 381235200 total_loss: 0.3423 time: 0.5456 s/iter data_time: 0.0465 s/iter total_throughput: 1876.97 samples/s lr: 1.02e-05 [09/21 12:54:45] lb.utils.events INFO: eta: 0:26:43 iteration: 372399/375342 consumed_samples: 381337600 total_loss: 0.3435 time: 0.5456 s/iter data_time: 0.0498 s/iter total_throughput: 1876.97 samples/s lr: 1.02e-05 [09/21 12:55:39] lb.utils.events INFO: eta: 0:25:48 iteration: 372499/375342 consumed_samples: 381440000 total_loss: 0.3403 time: 0.5456 s/iter data_time: 0.0495 s/iter total_throughput: 1876.97 samples/s lr: 1.01e-05 [09/21 12:56:34] lb.utils.events INFO: eta: 0:24:53 iteration: 372599/375342 consumed_samples: 381542400 total_loss: 0.3363 time: 0.5456 s/iter data_time: 0.0536 s/iter total_throughput: 1876.97 samples/s lr: 1.01e-05 [09/21 12:57:29] lb.utils.events INFO: eta: 0:24:00 iteration: 372699/375342 consumed_samples: 381644800 total_loss: 0.3372 time: 0.5456 s/iter data_time: 0.0538 s/iter total_throughput: 1876.97 samples/s lr: 1.01e-05 [09/21 12:58:24] lb.utils.events INFO: eta: 0:23:06 iteration: 372799/375342 consumed_samples: 381747200 total_loss: 0.3405 time: 0.5456 s/iter data_time: 0.0548 s/iter total_throughput: 1876.96 samples/s lr: 1.01e-05 [09/21 12:59:18] lb.utils.events INFO: eta: 0:22:11 iteration: 372899/375342 consumed_samples: 381849600 total_loss: 0.3455 time: 0.5456 s/iter data_time: 0.0529 s/iter total_throughput: 1876.96 samples/s lr: 1.01e-05 [09/21 13:00:13] lb.utils.events INFO: eta: 0:21:18 iteration: 372999/375342 consumed_samples: 381952000 total_loss: 0.3419 time: 0.5456 s/iter data_time: 0.0525 s/iter total_throughput: 1876.96 samples/s lr: 1.01e-05 [09/21 13:01:08] lb.utils.events INFO: eta: 0:20:24 iteration: 373099/375342 consumed_samples: 382054400 total_loss: 0.3388 time: 0.5456 s/iter data_time: 0.0518 s/iter total_throughput: 1876.96 samples/s lr: 1.01e-05 [09/21 13:02:03] lb.utils.events INFO: eta: 0:19:30 iteration: 373199/375342 consumed_samples: 382156800 total_loss: 0.3378 time: 0.5456 s/iter data_time: 0.0520 s/iter total_throughput: 1876.95 samples/s lr: 1.01e-05 [09/21 13:02:58] lb.utils.events INFO: eta: 0:18:36 iteration: 373299/375342 consumed_samples: 382259200 total_loss: 0.3383 time: 0.5456 s/iter data_time: 0.0531 s/iter total_throughput: 1876.95 samples/s lr: 1.01e-05 [09/21 13:03:53] lb.utils.events INFO: eta: 0:17:43 iteration: 373399/375342 consumed_samples: 382361600 total_loss: 0.3428 time: 0.5456 s/iter data_time: 0.0532 s/iter total_throughput: 1876.95 samples/s lr: 1.01e-05 [09/21 13:04:48] lb.utils.events INFO: eta: 0:16:49 iteration: 373499/375342 consumed_samples: 382464000 total_loss: 0.3459 time: 0.5456 s/iter data_time: 0.0536 s/iter total_throughput: 1876.95 samples/s lr: 1.01e-05 [09/21 13:05:43] lb.utils.events INFO: eta: 0:15:55 iteration: 373599/375342 consumed_samples: 382566400 total_loss: 0.3476 time: 0.5456 s/iter data_time: 0.0541 s/iter total_throughput: 1876.94 samples/s lr: 1.01e-05 [09/21 13:06:38] lb.utils.events INFO: eta: 0:15:01 iteration: 373699/375342 consumed_samples: 382668800 total_loss: 0.3466 time: 0.5456 s/iter data_time: 0.0544 s/iter total_throughput: 1876.94 samples/s lr: 1.00e-05 [09/21 13:07:33] lb.utils.events INFO: eta: 0:14:06 iteration: 373799/375342 consumed_samples: 382771200 total_loss: 0.3357 time: 0.5456 s/iter data_time: 0.0538 s/iter total_throughput: 1876.93 samples/s lr: 1.00e-05 [09/21 13:08:28] lb.utils.events INFO: eta: 0:13:12 iteration: 373899/375342 consumed_samples: 382873600 total_loss: 0.3357 time: 0.5456 s/iter data_time: 0.0508 s/iter total_throughput: 1876.93 samples/s lr: 1.00e-05 [09/21 13:09:23] lb.utils.events INFO: eta: 0:12:18 iteration: 373999/375342 consumed_samples: 382976000 total_loss: 0.3341 time: 0.5456 s/iter data_time: 0.0507 s/iter total_throughput: 1876.92 samples/s lr: 1.00e-05 [09/21 13:10:18] lb.utils.events INFO: eta: 0:11:23 iteration: 374099/375342 consumed_samples: 383078400 total_loss: 0.337 time: 0.5456 s/iter data_time: 0.0504 s/iter total_throughput: 1876.92 samples/s lr: 1.00e-05 [09/21 13:11:13] lb.utils.events INFO: eta: 0:10:28 iteration: 374199/375342 consumed_samples: 383180800 total_loss: 0.3374 time: 0.5456 s/iter data_time: 0.0523 s/iter total_throughput: 1876.92 samples/s lr: 1.00e-05 [09/21 13:12:08] lb.utils.events INFO: eta: 0:09:32 iteration: 374299/375342 consumed_samples: 383283200 total_loss: 0.3397 time: 0.5456 s/iter data_time: 0.0511 s/iter total_throughput: 1876.92 samples/s lr: 1.00e-05 [09/21 13:13:02] lb.utils.events INFO: eta: 0:08:37 iteration: 374399/375342 consumed_samples: 383385600 total_loss: 0.3406 time: 0.5456 s/iter data_time: 0.0506 s/iter total_throughput: 1876.92 samples/s lr: 1.00e-05 [09/21 13:13:57] lb.utils.events INFO: eta: 0:07:42 iteration: 374499/375342 consumed_samples: 383488000 total_loss: 0.3419 time: 0.5456 s/iter data_time: 0.0494 s/iter total_throughput: 1876.91 samples/s lr: 1.00e-05 [09/21 13:14:52] lb.utils.events INFO: eta: 0:06:47 iteration: 374599/375342 consumed_samples: 383590400 total_loss: 0.3475 time: 0.5456 s/iter data_time: 0.0521 s/iter total_throughput: 1876.91 samples/s lr: 1.00e-05 [09/21 13:15:46] lb.utils.events INFO: eta: 0:05:52 iteration: 374699/375342 consumed_samples: 383692800 total_loss: 0.3471 time: 0.5456 s/iter data_time: 0.0516 s/iter total_throughput: 1876.91 samples/s lr: 1.00e-05 [09/21 13:16:41] lb.utils.events INFO: eta: 0:04:56 iteration: 374799/375342 consumed_samples: 383795200 total_loss: 0.3454 time: 0.5456 s/iter data_time: 0.0522 s/iter total_throughput: 1876.91 samples/s lr: 1.00e-05 [09/21 13:17:36] lb.utils.events INFO: eta: 0:04:02 iteration: 374899/375342 consumed_samples: 383897600 total_loss: 0.3452 time: 0.5456 s/iter data_time: 0.0519 s/iter total_throughput: 1876.91 samples/s lr: 1.00e-05 [09/21 13:18:30] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_0374999 [09/21 13:18:31] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 13:18:31] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 13:18:36] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0593 s/iter. Inference: 0.2582 s/iter. Eval: 0.0024 s/iter. Total: 0.3200 s/iter. ETA=0:00:11 [09/21 13:18:41] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0607 s/iter. Inference: 0.2755 s/iter. Eval: 0.0024 s/iter. Total: 0.3387 s/iter. ETA=0:00:07 [09/21 13:18:46] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0660 s/iter. Inference: 0.2650 s/iter. Eval: 0.0027 s/iter. Total: 0.3340 s/iter. ETA=0:00:02 [09/21 13:18:48] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 13:18:48] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.331608 (0.000287 s / iter per device, on 8 devices) [09/21 13:18:48] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000231 s / iter per device, on 8 devices) [09/21 13:18:48] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 13:18:48] lb.evaluation.utils INFO: copypaste: Acc@1=80.096 [09/21 13:18:48] lb.evaluation.utils INFO: copypaste: Acc@5=94.702 [09/21 13:18:48] lb.engine.hooks INFO: Saved best model as latest eval score for Acc@1 is 80.09600, better than last best score 80.08800 @ iteration 369999. [09/21 13:18:48] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_best [09/21 13:18:49] lb.utils.events INFO: eta: 0:03:07 iteration: 374999/375342 consumed_samples: 384000000 total_loss: 0.3445 time: 0.5456 s/iter data_time: 0.0514 s/iter total_throughput: 1876.91 samples/s lr: 1.00e-05 [09/21 13:19:43] lb.utils.events INFO: eta: 0:02:12 iteration: 375099/375342 consumed_samples: 384102400 total_loss: 0.3428 time: 0.5456 s/iter data_time: 0.0516 s/iter total_throughput: 1876.91 samples/s lr: 1.00e-05 [09/21 13:20:38] lb.utils.events INFO: eta: 0:01:17 iteration: 375199/375342 consumed_samples: 384204800 total_loss: 0.3428 time: 0.5456 s/iter data_time: 0.0502 s/iter total_throughput: 1876.91 samples/s lr: 1.00e-05 [09/21 13:21:32] lb.utils.events INFO: eta: 0:00:22 iteration: 375299/375342 consumed_samples: 384307200 total_loss: 0.3432 time: 0.5456 s/iter data_time: 0.0512 s/iter total_throughput: 1876.91 samples/s lr: 1.00e-05 [09/21 13:21:55] lb.utils.checkpoint INFO: Saving checkpoint to ./commit_7a3a_swinv2/model_final [09/21 13:21:56] lb.utils.events INFO: eta: 0:00:00 iteration: 375341/375342 consumed_samples: 384350208 total_loss: 0.3397 time: 0.5456 s/iter data_time: 0.0538 s/iter total_throughput: 1876.91 samples/s lr: 1.00e-05 [09/21 13:21:56] lb.engine.hooks INFO: Overall training speed: 375340 iterations in 2 days, 8:53:05 (0.5456 s / it) [09/21 13:21:56] lb.engine.hooks INFO: Total training time: 2 days, 9:17:20 (0:24:15 on hooks) [09/21 13:21:56] lb.evaluation.evaluator INFO: with eval_iter 100000.0, reset total samples 50000 to 50000 [09/21 13:21:56] lb.evaluation.evaluator INFO: Start inference on 50000 samples [09/21 13:22:00] lb.evaluation.evaluator INFO: Inference done 11264/50000. Dataloading: 0.0536 s/iter. Inference: 0.2423 s/iter. Eval: 0.0026 s/iter. Total: 0.2984 s/iter. ETA=0:00:11 [09/21 13:22:06] lb.evaluation.evaluator INFO: Inference done 26624/50000. Dataloading: 0.0750 s/iter. Inference: 0.2553 s/iter. Eval: 0.0027 s/iter. Total: 0.3332 s/iter. ETA=0:00:07 [09/21 13:22:11] lb.evaluation.evaluator INFO: Inference done 43008/50000. Dataloading: 0.0734 s/iter. Inference: 0.2540 s/iter. Eval: 0.0027 s/iter. Total: 0.3304 s/iter. ETA=0:00:01 [09/21 13:22:13] lb.evaluation.evaluator INFO: Total valid samples: 50000 [09/21 13:22:13] lb.evaluation.evaluator INFO: Total inference time: 0:00:14.258968 (0.000285 s / iter per device, on 8 devices) [09/21 13:22:13] lb.evaluation.evaluator INFO: Total inference pure compute time: 0:00:11 (0.000222 s / iter per device, on 8 devices) [09/21 13:22:13] lb.engine.default INFO: Evaluation results for ImageNetDataset in csv format: [09/21 13:22:13] lb.evaluation.utils INFO: copypaste: Acc@1=80.074 [09/21 13:22:13] lb.evaluation.utils INFO: copypaste: Acc@5=94.71000000000001 [09/21 13:22:13] lb.engine.hooks INFO: Not saving as latest eval score for Acc@1 is 80.07400, not better than best score 80.09600 @ iteration 374999.