Adding checks for broken bottleneck files

[rizasif]

An exception is raised when a broken cached bottleneck files is read. Such a file can be created as a result of sudden system failure.

This commit refers the issue #2296 (#2296). Although closed but the problem still persists. Please view comments by @rizasif92 at the end. The changes contain nothing but error handling.

[tensorflow-jenkins]

Can one of the admins verify this patch?

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please let us know the company's name.

[rizasif]

I signed it!

[googlebot]

CLAs look good, thanks!

[petewarden]

Thanks @rizasif , I really appreciate you putting this fix together! Could I request a slightly different approach though? I would prefer to keep the get_or_create_bottleneck() function always returning a bottleneck.

To ensure that, we could move the section nested inside the if not os.path.exists(bottleneck_path): condition into a new function, create_bottleneck_file. Then we can call it from that condition, but also have a flag that catches float exceptions when reading and calls it in that case too. Here's some untested code to try and show what I mean:

  if not os.path.exists(bottleneck_path):
    create_bottleneck_file(...)
  did_hit_error = False
  try:
    bottleneck_values = [float(x) for x in bottleneck_string.split(',')]
  except:
    print("Invalid float found, recreating bottleneck")
    did_hit_error = True
  if did_hit_error:
    create_bottleneck_file(...)
    # Allow exceptions to propagate here, since they shouldn't happen after a fresh creation
    bottleneck_values = [float(x) for x in bottleneck_string.split(',')]

I'm hoping with this approach none of the other changes outside this function should be needed since the contract to always return a list of bottleneck values is preserved.

Does that make sense?

[rizasif]

@petewarden I agree with your solution and was thinking to implement it. I have made the changes as suggested, and tested them as well; I am attaching a screenshot of the output. Kindly review and let me know if anything else is required.
Thank you :)

[petewarden]

This looking great, thanks for the updates! I have a couple of minor comments, but once those are addressed this looks good to go in.

[petewarden]

I think we should put this line back?

[petewarden]

Can we revert this change, since now the bottlenecks will never be None?

[rizasif]

@petewarden done

[rmlarsen]

@tensorflow-jenkins test this please

[rmlarsen]

@rizasif it looks like you added some 4 character indents, which breaks the sanity check. TF uses 2 character indents. Please fix.

See: https://ci.tensorflow.org/job/tensorflow-pull-requests-sanity/2923/console

[rizasif]

Indentation changes made, kindly re-run the tests. Thanks :)

[rmlarsen]

@tensorflow-jenkins test this please

[rmlarsen]

Thanks @rizasif. @petewarden can you take another look? If the logics right to you, approve and I'll get it merged.

[petewarden]

This looks great, thanks!

[rizasif]

Did the tests pass? :/

[rmlarsen]

@rizasif it looks like you still have some bad indents, e.g.

FAIL: Found 1 non-whitelited pylint errors:
tensorflow/examples/image_retraining/retrain.py:353: [E0001(syntax-error), ] unexpected indent

[rizasif]

@rmlarsen the code seems to be working at my end; so we might have to do this a few more times to resolve these indentation issues. I hope this commit works fine. @tensorflow-jenkins please test :)

[rmlarsen]

@rizasif Thanks :)

@tensorflow-jenkins test this please

[rmlarsen]

@rizasif thanks for the contribution!

[rizasif]

Pleasure's all mine @rmlarsen
Thank you so much for all the support. Hope to see you guys again. Take Care :)