-
Notifications
You must be signed in to change notification settings - Fork 2k
[tfjs-converter] Allow custom shard size for more conversion pairs #2774
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
pyu10055
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the great contribution. The PR looks great, one high level comment, we have the wizard.ts file https://github.com/tensorflow/tfjs/blob/master/tfjs-converter/python/tensorflowjs/converters/wizard.py
can you add support for the shard size param too? thanks
Reviewed 1 of 6 files at r1.
Reviewable status: 0 of 1 approvals obtained (waiting on @pyu10055)
f474f78 to
3b24cef
Compare
|
@pyu10055 Thanks for the review. I added support for the shard size param in the wizard. |
pyu10055
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 1 approvals obtained (waiting on @pvaneck)
tfjs-converter/python/tensorflowjs/converters/wizard.py, line 486 at r2 (raw file):
'when': lambda answers: (value_in_list(answers, common.OUTPUT_FORMAT, (common.TFJS_LAYERS_MODEL, common.TFJS_GRAPH_MODEL)) or
should this be 'and' instead of 'or'?
pvaneck
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 1 approvals obtained (waiting on @pyu10055)
tfjs-converter/python/tensorflowjs/converters/wizard.py, line 486 at r2 (raw file):
Previously, pyu10055 (Ping Yu) wrote…
should this be 'and' instead of 'or'?
Ideally, we would only need the first condition before the or, however, if the input format is TF_SAVED_MODEL or TF_HUB_MODEL, the implication is that the model will be converted to TFJS_GRAPH_MODEL. Since this is implied, the user isn't asked in the wizard to select an output format, so the OUTPUT_FORMAT key is never set in answers. Thus, the first condition where we just check OUTPUT_FORMAT would fail.
So, that second or condition is just another way of checking if the OUTPUT_FORMAT is TFJS_GRAPH_MODEL based on the input format.
pyu10055
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status:
complete! 1 of 1 approvals obtained (waiting on @pvaneck)
tfjs-converter/python/tensorflowjs/converters/wizard.py, line 486 at r2 (raw file):
Previously, pvaneck (Paul Van Eck) wrote…
Ideally, we would only need the first condition before the
or, however, if the input format is TF_SAVED_MODEL or TF_HUB_MODEL, the implication is that the model will be converted to TFJS_GRAPH_MODEL. Since this is implied, the user isn't asked in the wizard to select an output format, so theOUTPUT_FORMATkey is never set inanswers. Thus, the first condition where we just checkOUTPUT_FORMATwould fail.So, that second
orcondition is just another way of checking if the OUTPUT_FORMAT is TFJS_GRAPH_MODEL based on the input format.
make sense, thank you.
Previously, only the tfjs_layers_model to tfjs_layers_model conversion pair worked with the
weight_shard_size_bytesflag. This PR allows the flag to be used for any conversion where the output_format istfjs_layers_modelortfjs_graph_model.This change is