Add LookupTableSize and LookupTableSizeV2 ops #4755

fneubaum-sulvo · 2021-02-26T15:21:33Z

Added LookupTableSize and LookupTableSizeV2 operations. What I do is get the size of the hash table and return it as a Rank 0 Tensor. The PR includes unit tests and also I tried it with the model I'm trying to convert, as described in this issue, LookupTableSizeV2 isn't shown as unsupported anymore.

This change is

google-cla · 2021-02-26T15:21:37Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-02-26T15:22:13Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

pyu10055

Thank you for the contribution!

Reviewed 7 of 7 files at r1.
Reviewable status: complete! 1 of 1 approvals obtained (waiting on @lina128)

lina128

Thank you @fneubaum , overall looks good, I left some minor comments.

Reviewable status: complete! 1 of 1 approvals obtained (waiting on @fneubaum-sulvo)

tfjs-converter/metadata/kernel2op.json, line 262 at r2 (raw file):

  "LookupTableImportV2": [],
  "LookupTableSize": [
    "scalar"

Remove 'scalar', just leave it blank. Because there's no tfjs op that maps to this kernel.

tfjs-converter/metadata/kernel2op.json, line 265 at r2 (raw file):

  ],
  "LookupTableSizeV2": [
    "scalar"

Same.

tfjs-converter/python/tensorflowjs/op_list/hash_table.json, line 206 at r2 (raw file):

      }
    ],
    "attrs": [

Remove the attrs field, basically match what is described in the TF op pbtxt file: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/ops/ops.pbtxt#L22817-L22828

tfjs-converter/python/tensorflowjs/op_list/hash_table.json, line 231 at r2 (raw file):

      }
    ],
    "attrs": [

Remove attrs.

tfjs-converter/src/operations/executors/hash_table_executor.ts, line 77 at r2 (raw file):

      const hashTable = resourceManager.getHashTableById(handle.id);
      return [await tfOps.scalar(hashTable.size())];

scalar(hashTable.size(), 'int32')

tfjs-converter/src/operations/executors/hash_table_executor_test.ts, line 490 at r2 (raw file):

        const size = await result[0].data();

        expect(size[0]).toEqual(0);

test_util.expectArraysClose(size, [0]);

tfjs-converter/src/operations/executors/hash_table_executor_test.ts, line 523 at r2 (raw file):

        const size = await result[0].data();

        expect(size[0]).toEqual(2);

test_util.expectArraysClose(size, [2]);

tfjs-converter/src/operations/executors/hash_table_executor_test.ts, line 558 at r2 (raw file):

        const size = await result[0].data();

        expect(size[0]).toEqual(0);

Same.

tfjs-converter/src/operations/executors/hash_table_executor_test.ts, line 591 at r2 (raw file):

        const size = await result[0].data();

        expect(size[0]).toEqual(2);

Same.

tfjs-converter/src/operations/op_list/hash_table.ts, line 124 at r2 (raw file):

      {'start': 0, 'name': 'tableHandle', 'type': 'tensor'}
    ],
    'attrs': [

Remove attrs.

tfjs-converter/src/operations/op_list/hash_table.ts, line 139 at r2 (raw file):

      {'start': 0, 'name': 'tableHandle', 'type': 'tensor'}
    ],
    'attrs': [

Remove attrs.

google-cla · 2021-03-01T14:03:03Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-03-01T14:05:03Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

fneubaum-sulvo

Thank you very much @lina128 ! You're very helpful.
I made the changes and now the code should be in better shape, I left a couple of questions but mostly to be sure the code is correct.

I need the following operations too as I have a model that depends on them: SparseSegmentMean, StringToHashBucketFast, SparseReshape, SparseFillEmptyRows. Sorry to ask but is there some help or pointers you could provide? Mainly for the Sparse ops as only SparseToDense is implemented. Thanks in advance!

Reviewable status: complete! 1 of 1 approvals obtained (waiting on @lina128)

tfjs-converter/metadata/kernel2op.json, line 262 at r2 (raw file):