Evaluation/Finetuning of Resnet 50 in TF 2.X

[peri044]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not been filed already.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/official/vision/image_classification/resnet
For pretrained checkpoints, I used the ones linked in the README (https://github.com/tensorflow/models/tree/master/official/vision/image_classification/resnet#pretrained-models)

2. Describe the bug

1. I'm trying to evaluate and finetune the Resnet 50 model available at the URL(mentioned above). However I get near zero accuracy when I evaluate. I would like to know how to evaluate and finetune using the existing RN50 checkpoint.

The command I use for evaluating the existing model
python3 resnet_ctl_imagenet_main.py --model_dir=checkpoints/ --num_gpus=1 --batch_size=32 --train_epochs=1 --train_steps=1 --use_synthetic_data=false --data_dir imagenet_tfr_data/

The model_dir is set to checkpoints directory which has the downloaded checkpoint (from the README link). The checkpoint manager picks up this checkpoint, however does not seem to load as I get many unresolved object issues where the layer names mismatch.

W0518 15:35:57.377599 139869265008448 util.py:144] Unresolved object in checkpoint: (root).layer_with_weights-4.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-5.axis

2. Currently I seem to get this resnet model running for Tensorflow 2.2. There were multiple errors for 2.0 and 2.1. One such error is from tensorflow.python.keras.layers.preprocessing import image_preprocessing as image_ops. ImportError: cannot import name 'image_preprocessing'
   This may not be relevant to the actual issue but for me TF 2,0 and TF 2.1 seem to give import and not found attribute errors which drove me to try TF 2.2.

3. Evaluation works when I do the following

In the resnet_runnable.py, I use keras way of loading the checkpoint
self.model.load_weights(flags_obj.pretrained_filepath)

This probably loads the checkpoint according to network topology rather than names (used by tf.train.CheckpointManager. (Is this correct way of loading ? )
I disable training manually and run the self._evaluate_once(current_step) to get 76.476. Just wanted to confirm if this is same accuracy that you obtained?

The questions are

• Is there a plan to add standalone eval script to this repo ?
• If the way of evaluation described in 3) is recommended, can it be added to the repo as well as update the documentation as well on eval/finetuning steps?
  I would be happy to make a PR if required :)

Thank you !!

[peri044]

@saberkun Can you please provide some details here ? Thank you

[saberkun]

Hi @peri044, the evaluation accuracy is reasonable. Sorry, I do not have record of the exact evaluation metric.
In terms of the checkpoint loading, the problem in 1 is because the released checkpoint is saved by model.save_weights(). model.save_weights() is not compatible with the training checkpoint used by the checkpoint manager which is tf.train.Checkpoint(model=xxx). The root object is different. You should use model.load_weights() for this checkpoint.
The resnet runnable CTL implementation is probably one-off. Would you please use https://github.com/tensorflow/models/blob/master/official/vision/image_classification/classifier_trainer.py#L193 for most usages? Thanks

[peri044]

Hello @saberkun, Thanks for getting back and providing some info on the checkpoint loading.
I did try with classifier trainer implementation but I'm running into issues with it. Here's the following error when I try to finetune the network. I'm using TF 2.2 version. Can you please provide some help/suggestions on this issue ?
Command used:
python3 classifier_trainer.py --mode=train_and_eval --model_type=resnet --dataset=imagenet --model_dir=resnet/checkpoints/ --data_dir=/mnt/cdr/ImageNet/train-val-tfrecord/ --config_file=configs/examples/resnet/imagenet/gpu.yaml
resnet/checkpoints is the directory which has pre-trained checkpoints.
Error stacktrace: (Shortened a bit due for easy viewing)

File "classifier_trainer.py", line 220, in resume_from_checkpoint
model.load_weights(latest_checkpoint)
File "/home/dperi/Downloads/py3/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
File "/home/dperi/Downloads/py3/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1237, in load_weights
status = self._trackable_saver.restore(filepath)
File "/home/dperi/Downloads/py3/lib/python3.6/site-packages/tensorflow/python/training/tracking/util.py", line 1304, in restore
checkpoint=checkpoint, proto_id=0).restore(self._graph_view.root)
File "/home/dperi/Downloads/py3/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py", line 209, in restore
restore_ops = trackable._restore_from_checkpoint_position(self) # pylint: disable=protected-access
File "/home/dperi/Downloads/py3/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py", line 668, in getattribute
return super(OptimizerV2, self).getattribute(name)
AttributeError: 'LossScaleOptimizer' object has no attribute '_slots'

[peri044]

Hello @saberkun I was able to get past the above error by setting the datatype to tf.float32 instead of the default tf.float16. tf.float16 was using mixed_precision loss scale optimizer.
However I'm facing a new error

ValueError: Failed to find data adapter that can handle input: <class 'tensorflow.python.distribute.input_lib.DistributedDatasetsFromFunction'>, <class 'NoneType'>

I'm using a single GPU and tfrecords dataset for my finetuning experiment.
Command used:
python3 classifier_trainer.py --mode=train_and_eval --model_type=resnet --dataset=imagenet --model_dir=resnet/checkpoints/ --data_dir=/mnt/cdr/ImageNet/train-val-tfrecord/ --config_file=configs/examples/resnet/imagenet/gpu.yaml --params_override='runtime.num_gpus=1'

The error occurs at model.fit() call. Upon digging further, the issue happens here https://github.com/tensorflow/tensorflow/blob/r2.2/tensorflow/python/keras/engine/data_adapter.py#L683
Any suggestions to resolve this ? Thank you

[saberkun]

The handle for tensorflow.python.distribute.input_lib.DistributedDatasetsFromFunction is added inside TF-nightly I know for sure.
@omalleyt12 Is it added for TF 2.2?

Hi peri044@, classifier_trainer.py is not included in the mode garden 2.2 release. It is added after that which is targeting to the coming 2.3 release.

[omalleyt12]

@peri044 Thanks for the issue!

The handle for tensorflow.python.distribute.input_lib.DistributedDatasetsFromFunction is added inside TF-nightly I know for sure.
@omalleyt12 Is it added for TF 2.2?

Unfortunately it looks like that support didn't make it into 2.2. At head we handle distributed datasets: code

In TF2.2, we expect model.fit to be passed a non-distributed dataset, and then we call tf.distribute.Strategy.distribute_dataset on it

[saberkun]

Hi @peri044 , please use tf-nightly with the master head. Uses tf 2.2 with the 2.2 release tag. Thanks