Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ONNX] Add huggingface models into CI tests #107247

Closed
wants to merge 21 commits into from
Closed

[ONNX] Add huggingface models into CI tests #107247

wants to merge 21 commits into from

Conversation

titaiwangms
Copy link
Collaborator

@titaiwangms titaiwangms commented Aug 15, 2023 •
edited

Stack from ghstack (oldest at bottom):

  1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit uint8/bool conflict when a newer version of transformers is used.
    • Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    • Falcon and MPT has unsupported user coding to Dynamo.
  2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that non-persistent buffer is not supported

@titaiwangms
[ONNX] Add huggingface models into CI tests
aae282d 
[ONNX] Add transformers models into no runtime test of fx exporter

[ghstack-poisoned]
titaiwangms added a commit that referenced this pull request Aug 15, 2023
@titaiwangms
[ONNX] Add huggingface models into CI tests
f1e11a0 
[ONNX] Add transformers models into no runtime test of fx exporter

ghstack-source-id: 01663e84842d7dd46fd7a72154ea1b2d5bb655d7
Pull Request resolved: #107247
@titaiwangms titaiwangms marked this pull request as draft August 15, 2023 18:42
@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Aug 15, 2023
@pytorch-bot
Copy link

pytorch-bot bot commented Aug 15, 2023 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/107247

Note: Links to docs will display an error until the docs builds have been completed.

✅ 1 Unrelated Failure

As of commit 487fa54 with merge base a4eae43 (image):

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@titaiwangms titaiwangms mentioned this pull request Aug 15, 2023
Closed
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
dc16fb9 
[ONNX] Add transformers models into no runtime test of fx exporter

[ghstack-poisoned]
titaiwangms added a commit that referenced this pull request Aug 15, 2023
@titaiwangms
[ONNX] Add huggingface models into CI tests
d3be05e 
[ONNX] Add transformers models into no runtime test of fx exporter

ghstack-source-id: 84f2b39baf9f4a44e2ce4c6c50c4a0364ea7b8eb
Pull Request resolved: #107247
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
1e3a8f7 
[ONNX] Add transformers models into no runtime test of fx exporter

[ghstack-poisoned]
titaiwangms added a commit that referenced this pull request Aug 15, 2023
@titaiwangms
[ONNX] Add huggingface models into CI tests
b4d6e23 
[ONNX] Add transformers models into no runtime test of fx exporter

ghstack-source-id: 3408fc7f4f1db69f5d6368520749022f9a0ab945
Pull Request resolved: #107247
@titaiwangms titaiwangms marked this pull request as ready for review August 15, 2023 22:51
@titaiwangms titaiwangms added module: onnx Related to torch.onnx release notes: onnx torch.onnx related changes that should show up in the release notes labels Aug 15, 2023
@titaiwangms titaiwangms removed the request for review from jeffdaily August 15, 2023 22:58
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
9fccefd 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
titaiwangms added a commit that referenced this pull request Aug 15, 2023
@titaiwangms
[ONNX] Add huggingface models into CI tests
0af4cb9 
[ONNX] Add transformers models into no runtime test of fx exporter

ghstack-source-id: 56fdbc3a94c35f13010e1349581c3f3a5d505b91
Pull Request resolved: #107247
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
aec630d 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
titaiwangms added a commit that referenced this pull request Aug 16, 2023
@titaiwangms
[ONNX] Add huggingface models into CI tests
565d558 
[ONNX] Add transformers models into no runtime test of fx exporter

ghstack-source-id: a404d54be14f9d24966bc59f226431aae19dc0a0
Pull Request resolved: #107247
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
8c25e11 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
titaiwangms added a commit that referenced this pull request Aug 16, 2023
@titaiwangms
[ONNX] Add huggingface models into CI tests
ca682f2 
[ONNX] Add transformers models into no runtime test of fx exporter

ghstack-source-id: fe1a4c2f9d0c554a49047ce4c543444eeb39565c
Pull Request resolved: #107247
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
6cdc7a0 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
titaiwangms added a commit that referenced this pull request Aug 16, 2023
@titaiwangms
[ONNX] Add huggingface models into CI tests
6212c59 
[ONNX] Add transformers models into no runtime test of fx exporter

ghstack-source-id: 2640837ce94afb45879c22c2f65189dad34d1dd3
Pull Request resolved: #107247
@titaiwangms titaiwangms removed a link to an issue Aug 18, 2023
[ONNX] ONNX doesn't support exporting non-persistent buffer included models in FakeMode #107211
Closed
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
866bf27 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
ec896d0 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
b9b122d 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
f365371 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
dcea1d4 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
titaiwangms
titaiwangms commented Aug 22, 2023
View reviewed changes
test/onnx/test_fx_to_onnx.py
# TODO: From Config/Model
@pytorch_test_common.skip_in_ci(
"Skip this test in CI because of memory issue."
"SymFloat in OnnxFUnction attribute is not supported yet."
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#107715

wschin
wschin reviewed Aug 22, 2023
View reviewed changes
test/onnx/test_fx_to_onnx_with_onnxruntime.py
def create_kwargs(model_name=model_name):
tokenizer = AutoTokenizer.from_pretrained(model_name)
return tokenizer("Hello world!", return_tensors="pt")
def create_kwargs():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In _test_fake_tensor_mode_exporter, the exported ONNX model and PyTorch model are run and their outputs are compared. However, I am wondering if that really checks dynamic_shapes. If not, probably with the next (or 2nd next), let's modify create_args, create_kwargs, and test_fake_tensor_mode_exporter to guard dynamic_shapes supports different batch and sequence sizes.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is assert_dynamic_shapes inside the _test_fake_tensor_mode_exporter should address your concern.

wschin
wschin approved these changes Aug 22, 2023
View reviewed changes
Copy link
Collaborator

@wschin wschin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good first steps to testing important HF models. There are more works required, especially for testing dynamic_shapes=True + FakeTensorMode, but can be done in another PR.

@wschin
Copy link
Collaborator

wschin commented Aug 22, 2023

Remember to create issues against proper repos (pytorch, hf, onnx, or ort) for the errors/problems encountered during the test.

@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
8927ebc 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
@titaiwangms titaiwangms added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 22, 2023
BowenBao
BowenBao reviewed Aug 22, 2023
View reviewed changes
test/onnx/test_fx_to_onnx_with_onnxruntime.py

def create_model() -> nn.Module:
return transformers.AutoModel.from_pretrained(model_name)
return transformers.AutoModel.from_pretrained(model_name).to(device).eval()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this still needs to be cached though right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. Added back the tiny-gpt2 cache.

BowenBao
BowenBao approved these changes Aug 23, 2023
View reviewed changes
Copy link
Collaborator

@BowenBao BowenBao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG w/ comment on tests with pretrained model and CI caching.

@titaiwangms
Update on "[ONNX] Add huggingface models into CI tests"
487fa54 
1. Add a list of HF models to CI tests. The PR intends to build them from Config, but some of them are not supported with Config. NOTE: Loaded from pre-trained model could potentially hit [uint8/bool conflict](huggingface/transformers#21013) when a newer version of transformers is used.
    - Dolly has torch.fx.Node in OnnxFunction attribute, which is currently not supported.
    - Falcon and MPT has unsupported user coding to Dynamo.
2. Only update GPT2 exporting with real tensor to Config, as FakeMode rises unequal input errors between PyTorch and ORT. The reason is that [non-persistent buffer is not supported](#107211)

[ghstack-poisoned]
@titaiwangms
Copy link
Collaborator Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@facebook-github-bot facebook-github-bot deleted the gh/titaiwangms/43/head branch August 26, 2023 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wschin wschin wschin approved these changes

@BowenBao BowenBao BowenBao approved these changes

@abock abock Awaiting requested review from abock

@thiagocrepaldi thiagocrepaldi Awaiting requested review from thiagocrepaldi

Assignees
No one assigned
Labels
ciflow/trunk Trigger trunk jobs on your pull request Merged module: onnx Related to torch.onnx open source release notes: onnx torch.onnx related changes that should show up in the release notes topic: not user facing topic category
Projects
ONNX
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[ONNX] Model Test List for FX-to-ONNX Exporters for FakeTensorMode and dynamic=True
5 participants
@titaiwangms @wschin @pytorchmergebot @BowenBao @pytorchbot