Skip to content

Support export >2G model when using optimizer.py only#6014

Merged
gh-yewang merged 2 commits into
masterfrom
wangye/raw
Dec 8, 2020
Merged

Support export >2G model when using optimizer.py only#6014
gh-yewang merged 2 commits into
masterfrom
wangye/raw

Conversation

@gh-yewang
Copy link
Copy Markdown
Contributor

Description: Describe your changes.

Motivation and Context

  • Why is this change required? What problem does it solve?
  • If it fixes an open issue, please link to the issue here.

@gh-yewang gh-yewang requested a review from a team as a code owner December 3, 2020 02:27
@gh-yewang gh-yewang requested a review from hanbitmyths December 3, 2020 02:27
Comment thread onnxruntime/python/tools/transformers/optimizer.py

_setup_logger(args.verbose)

if os.path.realpath(args.input) == os.path.realpath(args.output):
Copy link
Copy Markdown
Collaborator

@hanbitmyths hanbitmyths Dec 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also need to add checking dirname is the same when args.use_external_data_format is true? If input and output uses the same path, data will be overwritten anyway. And what do you think if exiting instead of warning? Do you think we have any case of using the same input and output name?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm..I think this line is enough for check paths for whatever the args.use_external_data_format is if we were to use this file separately?
I am not quite following the second question is that saying we should like return the program if input and output share the same path? As warning user may loss the original model...
From our side, no, we will output it to a different path.

@gh-yewang
Copy link
Copy Markdown
Contributor Author

/azp run orttraining-linux-gpu-ci-pipeline

@gh-yewang gh-yewang merged commit fa06be2 into master Dec 8, 2020
@gh-yewang gh-yewang deleted the wangye/raw branch December 8, 2020 01:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants