Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add custom_objects in the function layer_test #36273

Merged
merged 5 commits into from
Mar 13, 2020

Conversation

MokkeMeguru
Copy link
Contributor

@MokkeMeguru MokkeMeguru commented Jan 28, 2020

Add a custom_objects argument to layer_test, to enable running layer_test for custom layers.

reference: #36272

layer test for custom layer

Usage:
```
testing_utils.layer_test(
    SyncBatchNormalization,
    # keras.layers.BatchNormalization,
    kwargs={
      'momentum': 0.9,
      'epsilon': 0.1,
      'gamma_regularizer': keras.regularizers.l2(0.01),
      'beta_regularizer': keras.regularizers.l2(0.01)
    },
    input_shape=(3, 4, 2),
    custom_objects={'SyncBatchNormalization': SyncBatchNormalization})
```
@tensorflow-bot tensorflow-bot bot added the size:S CL Change Size: Small label Jan 28, 2020
@googlebot
Copy link

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@MokkeMeguru
Copy link
Contributor Author

@googlebot I signed it!

@googlebot
Copy link

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

@ravikyram ravikyram self-assigned this Jan 28, 2020
@ravikyram ravikyram added the comp:keras Keras related issues label Jan 28, 2020
@gbaned gbaned added the awaiting review Pull request awaiting review label Feb 5, 2020
@gbaned gbaned requested a review from tomerk February 13, 2020 08:25
@tomerk
Copy link

tomerk commented Feb 13, 2020

Hi Mokke,

Just realized I accidentally commented on the issue instead of this PR:
Thanks for the PR! I have two thoughts about this:

layer_test is a totally internal method, we don't want external users relying on it. This change seems explicitly designed for external users of Keras implementing their own layers and trying to use layer_test.

The layer_test utility isn't an effective test of layer behavior, it just makes sure certain things don't crash. You would be better served writing your own set of unit tests that actually verify layer behavior (and that make sure training, evaluation, etc. work correctly).

That said, I'll add @yhliang2018 to see if she thinks that there are some sort of testing utilities we should be exposing to external users in Keras' API.

Copy link

@tomerk tomerk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(See my above comment)

@robieta robieta removed their request for review February 14, 2020 17:54
@tensorflowbutler tensorflowbutler removed the awaiting review Pull request awaiting review label Feb 21, 2020
@gbaned
Copy link
Contributor

gbaned commented Feb 26, 2020

@MokkeMeguru Can you please check tomerk's comments and keep us posted? Thanks!

@gbaned gbaned added the stat:awaiting response Status - Awaiting response from author label Feb 26, 2020
@MokkeMeguru
Copy link
Contributor Author

MokkeMeguru commented Feb 26, 2020

I waited @yhliang2018 's comment.
But I want to say "If We Want To Update Some Layers are Something Wrong, We Need To Pass Your UNITTEST At Least." (refs. #18222)
So this pull request gives you (and us) comfortable development of Tensorflow by removing ridiculous requests.

@@ -93,7 +93,9 @@ def layer_test(layer_cls, kwargs=None, input_shape=None, input_dtype=None,
string or integer values.
adapt_data: Optional data for an 'adapt' call. If None, adapt() will not
be tested for this layer. This is only relevant for PreprocessingLayers.

custom_objects: Optional Objects for your custom layer. If you write a
your layer, you can use this variable.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: custom_objects: Optional dictionary mapping name strings to custom objects in the layer class. This is helpful to test a custom layer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks your help!

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @yhliang2018 meant that you should update this description to instead say:

custom_objects: Optional dictionary mapping name strings to custom objects in the layer class. This is helpful for testing custom layers.

Then we can go ahead and approve this.

@yhliang2018
Copy link
Contributor

Thanks for the PR! I think it's okay to add an extra arg to layer_test to make it work better with custom layers. But the two points mentioned by @tomerk still apply: we may not expose it as an public API, and for your custom layers, you may need to test more to make it behave as expected.

@tensorflowbutler tensorflowbutler removed the stat:awaiting response Status - Awaiting response from author label Feb 29, 2020
@gbaned gbaned added the stat:awaiting response Status - Awaiting response from author label Mar 3, 2020
@MokkeMeguru
Copy link
Contributor Author

MokkeMeguru commented Mar 3, 2020

response? Should I tell you anything more?
I want to cooperate with this PR.

@gbaned
Copy link
Contributor

gbaned commented Mar 4, 2020

@MokkeMeguru Can you please address the reviewers comments and resolve conflicts? Thanks!

@tensorflow-bot tensorflow-bot bot removed the ready to pull PR ready for merge process label Mar 6, 2020
@MokkeMeguru
Copy link
Contributor Author

I fixed import error for organizing imports.

yhliang2018
yhliang2018 previously approved these changes Mar 6, 2020
Copy link
Contributor

@yhliang2018 yhliang2018 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. LGTM.

@tensorflow-bot tensorflow-bot bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Mar 6, 2020
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Mar 6, 2020
@MokkeMeguru
Copy link
Contributor Author

I don't know why this pr is not accepted by Unity Sanity with pylint error.
Can you help me?

logs.

=== Sanity check step 2 of 15: do_pylint (Python 3 pylint) ===

ERROR_WHITELIST="^tensorflow/python/framework/function_test\.py.*\[E1123.*noinline ^tensorflow/python/platform/default/_gfile\.py.*\[E0301.*non-iterator ^tensorflow/python/platform/default/_googletest\.py.*\[E0102.*function\salready\sdefined ^tensorflow/python/feature_column/feature_column_test\.py.*\[E0110.*abstract-class-instantiated ^tensorflow/contrib/layers/python/layers/feature_column\.py.*\[E0110.*abstract-class-instantiated ^tensorflow/contrib/eager/python/evaluator\.py.*\[E0202.*method-hidden ^tensorflow/contrib/eager/python/metrics_impl\.py.*\[E0202.*method-hidden ^tensorflow/contrib/rate/rate\.py.*\[E0202.*method-hidden ^tensorflow/python/training/tracking/tracking\.py.*\[E0202.*method-hidden ^tensorflow/python/platform/gfile\.py.*\[E0301.*non-iterator ^tensorflow/python/keras/callbacks\.py.*\[E1133.*not-an-iterable ^tensorflow/python/keras/engine/base_layer.py.*\[E0203.*access-member-before-definition ^tensorflow/python/keras/layers/recurrent\.py.*\[E0203.*access-member-before-definition ^tensorflow/python/kernel_tests/constant_op_eager_test.py.*\[E0303.*invalid-length-returned ^tensorflow/python/keras/utils/data_utils.py.*\[E1102.*not-callable ^tensorflow/python/autograph/.*_py3_test\.py.*\[E0001.*syntax-error ^tensorflow/python/keras/preprocessing/image\.py.*\[E0240.*Inconsistent method resolution "
Running pylint on 2595 files with 32 parallel jobs...


pylint took 160 s

Found a whitelisted error:
  tensorflow/python/autograph/pyct/static_analysis/activity_py3_test.py:52: [E0001(syntax-error), ] invalid syntax
Found a whitelisted error:
  tensorflow/python/autograph/impl/api_test.py:450: [E0202(method-hidden), ApiTest.test_converted_call_callable_metaclass.TestMetaclass.__call__] An attribute defined in api_test line 460 hides this method
Found a whitelisted error:
  tensorflow/python/distribute/multi_worker_test_base.py:196: [E1120(no-value-for-parameter), create_in_process_cluster] No value for argument 'reason' in unbound method call
Found a whitelisted error:
  tensorflow/python/kernel_tests/constant_op_eager_test.py:250: [E0303(invalid-length-returned), ConstantTest.testInvalidLength.BadList.__len__] __len__ does not return non-negative integer
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3293: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_invalid_type] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3293: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_invalid_type] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3300: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_one_feature_column] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3300: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_one_feature_column] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3310: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_two_feature_columns] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3310: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_two_feature_columns] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3311: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_two_feature_columns] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3311: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_two_feature_columns] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3323: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_equal_keys_different_parse_spec] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3323: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_equal_keys_different_parse_spec] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3324: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_equal_keys_different_parse_spec] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3324: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_equal_keys_different_parse_spec] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3331: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_equal_keys_equal_parse_spec] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3331: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_equal_keys_equal_parse_spec] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3332: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_equal_keys_equal_parse_spec] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3332: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_equal_keys_equal_parse_spec] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3345: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_multiple_features_dict] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3345: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_multiple_features_dict] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3346: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_multiple_features_dict] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/feature_column/feature_column_test.py:3346: [E0110(abstract-class-instantiated), MakeParseExampleSpecTest.test_multiple_features_dict] Abstract class '_TestFeatureColumn' with abstract methods instantiated
Found a whitelisted error:
  tensorflow/python/keras/saving/saved_model/json_utils.py:37: [E0202(method-hidden), Encoder.default] An attribute defined in json.encoder line 158 hides this method
Found a whitelisted error:
  tensorflow/python/keras/utils/data_utils.py:604: [E1102(not-callable), terminate_keras_multiprocessing_pools] multiprocessing.active_children is not callable
Found a whitelisted error:
  tensorflow/python/keras/utils/data_utils.py:896: [E1102(not-callable), init_pool_generator] multiprocessing.current_process is not callable
Found a whitelisted error:
  tensorflow/python/keras/utils/version_utils_test.py:120: [E0110(abstract-class-instantiated), SplitUtilsTest.test_user_provided_metaclass] Abstract class 'AbstractModel' with abstract methods instantiated

FAIL: Found 2 non-whitelisted pylint errors:
tensorflow/python/keras/testing_utils.py:201: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).

tensorflow/python/keras/testing_utils.py:257: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).

@MokkeMeguru
Copy link
Contributor Author

this means we should fix the code,

  recovered_model = keras.models.Model.from_config(
    model_config, custom_objects)

to

  recovered_model = keras.models.Model.from_config(
      model_config, custom_objects)

? (two space to four space)

TensorFlow's linting requires four spaces.
@tensorflow-bot tensorflow-bot bot removed the ready to pull PR ready for merge process label Mar 9, 2020
@MokkeMeguru
Copy link
Contributor Author

MokkeMeguru commented Mar 9, 2020

I updated PR for passing your lint test.
Other tests were passed.

@tensorflow-bot tensorflow-bot bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Mar 9, 2020
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Mar 9, 2020
@MokkeMeguru
Copy link
Contributor Author

I don't know why windows build will be crash.
I watched its log, but this error is no related to my PR.

@gbaned gbaned added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process and removed ready to pull PR ready for merge process labels Mar 11, 2020
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Mar 11, 2020
@tomerk
Copy link

tomerk commented Mar 13, 2020

I updated the description because I think the curly braces in the code snippets are causing issues with the merging.

@gbaned how do we restart the merging?
Here is the code snippet I removed:

Usage:

testing_utils.layer_test(
    SyncBatchNormalization,
    # keras.layers.BatchNormalization,
    kwargs={
      'momentum': 0.9,
      'epsilon': 0.1,
      'gamma_regularizer': keras.regularizers.l2(0.01),
      'beta_regularizer': keras.regularizers.l2(0.01)
    },
    input_shape=(3, 4, 2),
    custom_objects={'SyncBatchNormalization': SyncBatchNormalization})

@tensorflow-copybara tensorflow-copybara merged commit db48d9e into tensorflow:master Mar 13, 2020
@MokkeMeguru
Copy link
Contributor Author

Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes comp:keras Keras related issues ready to pull PR ready for merge process size:S CL Change Size: Small
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants