Create Jupyter notebook for clustering #330

Ruomei · 2020-03-31T12:33:06Z

This PR adds the Jupyter notebook for clustering.

googlebot · 2020-03-31T12:33:10Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

googlebot · 2020-03-31T12:33:10Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

review-notebook-app · 2020-03-31T12:33:13Z

Check out this pull request on

You'll be able to see Jupyter notebook diff and discuss changes. Powered by ReviewNB.

Ruomei · 2020-03-31T15:19:57Z

@googlebot I signed it!

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).

The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.

The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

googlebot · 2020-03-31T15:20:03Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

googlebot · 2020-03-31T15:20:04Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

tensorflow_model_optimization/g3doc/guide/clustering/clustering_with_keras.ipynb

tensorflow_model_optimization/g3doc/guide/clustering/clustering_comprehensive_guide.ipynb

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb

tensorflow_model_optimization/g3doc/guide/clustering/clustering_comprehensive_guide.ipynb

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb

tensorflow_model_optimization/g3doc/guide/clustering/clustering_comprehensive_guide.ipynb

TamasArm · 2020-06-18T18:18:14Z

tensorflow_model_optimization/g3doc/guide/clustering/clustering_comprehensive_guide.ipynb

@@ -0,0 +1,536 @@
+{


why is apply_clustering_to_first named as such?

Reply via ReviewNB

How should I improve it? The version I updated also has the name apply_clustering_to_dense.

The new apply_clustering_to_dense name seems good to me!

Sorry, I am confused. What was your thought on apply_clustering_to_first?

I just didn't understand what "first" stood for in the name. Because it looked like it just applies clustering to the layer if it's in the list of layers to be clustered and leaves it otherwise. So I just didn't quite understand what the 'first' referred to.

tensorflow_model_optimization/g3doc/guide/clustering/clustering_comprehensive_guide.ipynb

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb

Ruomei · 2020-07-06T15:04:45Z

@alanchiao @TamasArm @akarmi Nearly all of the review comments are addressed. There are at least two TODOs 1) find the version which I can "pip install" which includes the newly merged PRs; 2) change the way cluster_weights is imported after clustering APIs are exposed to the public.
Thanks a lot for reviewing and please let me know if there is anything else.

alanchiao · 2020-07-13T16:39:42Z

tensorflow_model_optimization/g3doc/guide/clustering/clustering_comprehensive_guide.ipynb

@@ -0,0 +1,470 @@
+{


Nit: "tips" -> "use cases"

Not all of these are tips - some is just a "how to" and the emphasis is still on use cases (do they care about deployment (might not - they could just be doing some training experiments), do they want to checkpoint (might not for quick experiments), ... etc.

Reply via ReviewNB

alanchiao · 2020-07-13T16:39:43Z

tensorflow_model_optimization/g3doc/guide/clustering/clustering_comprehensive_guide.ipynb

@@ -0,0 +1,470 @@
+{


"The optimal number of clusters per layer can be found via hyperparameter tuning."

Don't think this tip is useful since it seems like a given since all parameters are things to tune in ML.

The rest are bit more specific to compression specifically.

Reply via ReviewNB

alanchiao · 2020-07-13T16:39:43Z

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb

@@ -0,0 +1,686 @@
+{


determine if you should use it ->

determine if you should use it (including what's supported)

since some people may consider assume that already want to use weight clustering for its compression benefits but have forgotten
to first consider what's supported (e.g. custom Keras layers).

Reply via ReviewNB

alanchiao · 2020-07-13T16:39:43Z

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb

@@ -0,0 +1,686 @@
+{


Please see the pruning example with how it doesn't use tf.nn.softmax on the last layer directly and instead SparseCategoricalCrossEntropy with from_logits.

The reasoning is that it's not numerically stable to compute log (softmax(..)) directly, as described in places such as https://blog.feedly.com/tricks-of-the-trade-logsumexp/ . The log part is in the `sparse_categorical_crossentropy' loss, which operates on the softmax from tf.nn.softmax.

SparseCategoricalCrossEntropy(from_logits=True) can use the numerically stable code. In the current code with softmax and log in separate functions, it's hard for the framework to always use the numerically stable part (by fusing them and replacing them with a stable version).

Reply via ReviewNB

Thanks! This is interesting.
By overcoming numerical instability, do you mean we need to stabilize the softmax function against underflow and overflow? If yes, I agree that we need to use the tensorflow implementation.
Side questions:
1) If we use SparseCategoricalCrossEntropy with from_logits=false combined with the last layer of the model as softmax, it will end up with this clipping in Keras, which generates a bound [1e-7, 1-1e-7] on the output probability. However, will this clipping work if the input values are NaNs already? It should be no? But if yes, is the bound keeping the output of log(softmax(...)) stable?
2) In general, what use cases will be suited for the Keras clipping?

re 1): The probability calculation never generates nans. But it can saturate to 0.0 or 1.0, which will generate nans in the loss calculation, so clipping prevents the nans from being generated.

Also note that in graph mode keras can bypasses the softmax and executes the from_logits=True path, and so the results can be different depending on how you're using the model. That's bad.

re 2): None. It keeps beginners out of trouble, but for any serious application you shouldn't use it. Any example that hits the saturation gets a gradient of zero => is being ignored by the training procedure. For a small number of classes this will be all your worst classified examples (which would be an important signal). For a large number of classes this could be a significant fraction of your data.

If you want to export a model that outputs probabillities, add a softmax layer immediately before exporting.

The probability calculation never generates nans
What is the probability here?
Nans can be generated after softmax function (when the denominator of the softmax becomes 0), which is the input for keras clipping afaik.

Also note that in graph mode keras can bypasses the softmax and executes the from_logits=True path,
Did not know that. Will look for the code at some point. Thanks!

re 2):
Agreed. Thanks!

What is the probability here?

The softmax output.

denominator of the softmax becomes 0

How do you make the denominator 0? it't the sum of exponentials. Sure some of those exponentials could underflow, but any good implementations subtracts the largest logit from all the logits (logits = logits - max(logits)) So it's impossible for everything to underflow.... unless all your logits are -inf? If you have infs or nans in your logits then you have a different problem.

Right?

Thanks!

How do you make the denominator 0?

I was thinking in exp(x) when x is very very negative (not sure whether it has to be -inf), the exp(x) will underflow.

Sure some of those exponentials could underflow, but any good implementations subtracts the largest logit from all the logits

Yes, indeed. And I did not know all implementations already have this. Will check the details myself.

alanchiao · 2020-07-13T16:50:32Z

Left a few remaining comments. Looks good to me otherwise and we can merge this as an initial version.
A future PR can update the imports once we have some kind of pip release.

Added @MarkDaoust as a final reviewer. I'm adding the ready to pull label so I can more easily test run this Colab, but will wait until everything is addressed first. Once everything is addressed, please squash the commits.

FYI @lamberta.

alanchiao · 2020-07-13T18:47:31Z

I ran the e2e tutorial on TF 2.2 and saw the following error message. @arovir01. Please address.

Ruomei · 2020-07-14T14:54:44Z

Hi Both,

I ran the e2e tutorial on TF 2.2 and saw the following error message. @arovir01. Please address.

Did you use the dev package or the ! pip install -q tensorflow-model-optimizationat the beginning of this notebook?

I tried it locally with the top commit in tfmot (939bed8) with this end-to-end example and it works.

When I tried it in colab it fails because of the package installed at the beginning of the file. The reason is that the latest tfmot master will include the fix for the function strip_clustering(), while the tfmot released package does not have it afaik.

If we decide to use the latest tfmot dev package in this notebook, there is also the change for 'cluster_centroids_init': cluster_config.CentroidInitialization.LINEAR which requires the import:from tensorflow_model_optimization.python.core.clustering.keras import cluster_config I will need to update that.

alanchiao · 2020-07-14T16:19:08Z

| Did you use the dev package or the ! pip install -q tensorflow-model-optimizationat the beginning of this notebook?

Ah I mistake - I ran it with the latest TFMOT release instead of more recent commit. Looks good to me then.

tensorflow_model_optimization/g3doc/guide/clustering/clustering_comprehensive_guide.ipynb

MarkDaoust · 2020-07-20T14:47:47Z

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb

@@ -0,0 +1,686 @@
+{


You've got a TODO here, I guess because of the internal looking import?

Reply via ReviewNB

MarkDaoust · 2020-07-20T14:47:47Z

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb

@@ -0,0 +1,686 @@
+{


Reading the available docs it's not clear what's happening inside, so I'm really left guessing.

I assume:

The clustering tools replace weights with a smaller weight-cluster array, and indices into that array (can't that be gzipped so show the size difference directly?)
strip_clustering then resets to the original style of direct-weight array, but using the clustered weights? Either I don't understand:
How this all works,
Or why strip_clustering is necessary to show the size difference here.

So maybe it could use little more explanation.

Reply via ReviewNB

Good point - I'll make the equivalent change on the pruning side.

I think something along the lines of "strip_clustering removes any tf.Variable that clustering only needs during training, which would otherwise add to model size during serving" would work. This avoids any references to implementation details (e.g. the wrapper)

MarkDaoust · 2020-07-20T14:47:47Z

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb

@@ -0,0 +1,686 @@
+{


Here I'm getting an error in colab:

--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-20-01035994e382> in <module>() 1 clustered_tflite_file = '/tmp/clustered_mnist.tflite' 2 converter = tf.lite.TFLiteConverter.from_keras_model(final_model) ----> 3 tflite_clustered_model = converter.convert() 4 with open(clustered_tflite_file, 'wb') as f: 5 f.write(tflite_clustered_model)
3 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/convert_to_constants.py in _get_tensor_data(func)
215 data = map_index_to_variable[idx].numpy()
216 else:
--> 217 data = val_tensor.numpy()
218 tensor_data[tensor_name] = {
219 "data": data,

AttributeError: 'Tensor' object has no attribute 'numpy'

Reply via ReviewNB

@Ruomei: you can install the new package at

"pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple tensorflow-model-optimization==0.4.0.dev2"

For testing.

MarkDaoust · 2020-07-20T14:52:07Z

Also, don't forget to add this to the _book.yaml, or it won't be correctly visible on the site.

You could also add the overview at the same time. https://www.tensorflow.org/model_optimization/guide/clustering at the same time

Ruomei · 2020-07-21T20:34:36Z

All done, thanks @alanchiao @MarkDaoust
I will squash the commits once everything looks okay.

Ruomei · 2020-07-21T20:36:09Z

Also, don't forget to add this to the _book.yaml, or it won't be correctly visible on the site.

You could also add the overview at the same time. https://www.tensorflow.org/model_optimization/guide/clustering at the same time

Thanks. @arovir01 was addressing this today afaik.

alanchiao · 2020-07-21T20:56:30Z

All done, thanks @alanchiao Mark Daoust
I will squash the commits once everything looks okay.

Looks good to me Ruomei. Thanks! Once Mark finishes the review with you and you've squashed it, I'll merge it.

Ruomei · 2020-07-22T13:02:56Z

Once Mark finishes the review

@alanchiao @MarkDaoust Somehow I did not see this line. The commits are now squashed but it is fine. I can do it again if necessary.

MarkDaoust

Thanks, I think all my concerns are addressed.

MarkDaoust · 2020-07-22T13:24:56Z

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb

@@ -0,0 +1,686 @@
+{


re 1): The probability calculation never generates nans. But it can saturate to 0.0 or 1.0, which will generate nans in the loss calculation, so clipping prevents the nans from being generated.

Also note that in graph mode keras can bypasses the softmax and executes the from_logits=True path, and so the results can be different depending on how you're using the model. That's bad.

re 2): None. It keeps beginners out of trouble, but for any serious application you shouldn't use it. Any example that hits the saturation gets a gradient of zero => is being ignored by the training procedure. For a small number of classes this will be all your worst classified examples (which would be an important signal). For a large number of classes this could be a significant fraction of your data.

If you want to export a model that outputs probabillities, add a softmax layer immediately before exporting.

googlebot added the cla: no PR contributor has not signed CLA label Mar 31, 2020

googlebot added cla: yes PR contributor has signed CLA and removed cla: no PR contributor has not signed CLA labels Mar 31, 2020

alanchiao reviewed Mar 31, 2020

View reviewed changes

Ruomei commented Mar 31, 2020

View reviewed changes

tensorflow_model_optimization/g3doc/guide/clustering/clustering_with_keras.ipynb Outdated Show resolved Hide resolved

Ruomei commented Mar 31, 2020

View reviewed changes

tensorflow_model_optimization/g3doc/guide/clustering/clustering_with_keras.ipynb Outdated Show resolved Hide resolved

alanchiao reviewed Apr 7, 2020

View reviewed changes

tensorflow_model_optimization/g3doc/guide/clustering/clustering_comprehensive_guide.ipynb Show resolved Hide resolved

alanchiao reviewed May 14, 2020

View reviewed changes

alanchiao reviewed Jun 16, 2020

View reviewed changes

TamasArm reviewed Jun 18, 2020

View reviewed changes

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb Show resolved Hide resolved

tensorflow_model_optimization/g3doc/guide/clustering/clustering_example.ipynb Show resolved Hide resolved

alanchiao reviewed Jul 13, 2020

View reviewed changes

alanchiao requested a review from MarkDaoust July 13, 2020 16:51

alanchiao added the ready to pull Working to get PR submitted to internal repository, after which merging to Github happens. label Jul 13, 2020

MarkDaoust reviewed Jul 20, 2020

View reviewed changes

alanchiao approved these changes Jul 21, 2020

View reviewed changes

arovir01 mentioned this pull request Jul 22, 2020

Add clustering overview document and Jupyter notebooks to _book.yaml #473

Merged

Create Jupyter notebooks for clustering

e616f48

Ruomei force-pushed the toupstream/clustering_jupyter_notebook branch from 19e9550 to e616f48 Compare July 22, 2020 12:56

MarkDaoust approved these changes Jul 22, 2020

View reviewed changes

alanchiao added ready to pull Working to get PR submitted to internal repository, after which merging to Github happens. and removed ready to pull Working to get PR submitted to internal repository, after which merging to Github happens. labels Jul 22, 2020

copybara-service bot merged commit 9b12529 into tensorflow:master Jul 23, 2020

Create Jupyter notebook for clustering #330

Create Jupyter notebook for clustering #330

Uh oh!

Conversation

Ruomei commented Mar 31, 2020

Uh oh!

googlebot commented Mar 31, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

googlebot commented Mar 31, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

review-notebook-app bot commented Mar 31, 2020

Uh oh!

Ruomei commented Mar 31, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

googlebot commented Mar 31, 2020

Uh oh!

googlebot commented Mar 31, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TamasArm Jun 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

TamasArm Jun 18, 2020 •

edited

Loading

alanchiao Jul 13, 2020 •

edited

Loading

alanchiao Jul 13, 2020 •

edited

Loading

Ruomei Jul 22, 2020 •

edited

Loading

MarkDaoust Jul 20, 2020 •

edited

Loading