Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable FunctionHookMode tracing by default #3338

Draft
wants to merge 13 commits into
base: develop
Choose a base branch
from
Draft

Enable FunctionHookMode tracing by default #3338

wants to merge 13 commits into from

Conversation

AlexanderDokuchaev
Copy link
Collaborator

@AlexanderDokuchaev AlexanderDokuchaev commented Mar 11, 2025 •
edited
Loading

Changes

  • Enable FunctionHookMode tracing by default
  • Disable wrap operetators by register_operator decorator if new tracing enabled
  • Update get_nodes_with_missed_input_edges to check missed input edges based on number of TensorMeta in arguments
  • Update examples:
    • torch/ssd300_vgg16: remove disable tracing for normalization (is works correctly but changed final metric 0.5148 vs 0.5145)
    • qat/torch/resnet18: update save / load config
    • qat/torch/anomalib: in model added new FQ in result changes graph, on windows changed metric (win: 0.9692, linux: 0.9767)
  • test_save_load on cuda
  • check model devices to insert modules in load_from_config

Related tickets

152996

Tests

https://github.com/openvinotoolkit/nncf/actions/runs/13846083796
https://github.com/openvinotoolkit/nncf/actions/runs/13846090769
manual/job/post_training_quantization/630/

@github-actions github-actions bot added NNCF PT Pull requests that updates NNCF PyTorch experimental labels Mar 11, 2025
@github-actions github-actions bot added NNCF Common Pull request that updates NNCF Common NNCF PTQ Pull requests that updates NNCF PTQ API Public API-impacting changes labels Mar 13, 2025
@github-actions github-actions bot removed the API Public API-impacting changes label Mar 13, 2025
@AlexanderDokuchaev AlexanderDokuchaev changed the title enable experimental tracing by default Enable FunctionHookMode tracing by default Mar 13, 2025
@AlexanderDokuchaev
Copy link
Collaborator Author

Diff in ssd300:
image

@AlexanderDokuchaev
Copy link
Collaborator Author

Diff in anomalib
image
image

@AlexanderDokuchaev AlexanderDokuchaev marked this pull request as ready for review March 13, 2025 23:31
@AlexanderDokuchaev AlexanderDokuchaev requested a review from a team as a code owner March 13, 2025 23:31
@alexsu52
Copy link
Contributor

Diff in anomalib image image

Is the left operand of the multiply ops quantized?

alexsu52
alexsu52 reviewed Mar 14, 2025
View reviewed changes
Copy link
Contributor

@alexsu52 alexsu52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check your PR with #3322 and run the nightly training test for Torch backend.
cc' @ljaljushkin

ljaljushkin
ljaljushkin reviewed Mar 14, 2025
View reviewed changes
nncf/experimental/torch2/function_hook/serialization.py
Comment on lines +108 to +119
device = None
if not is_multidevice(wrapped_model):
device = get_model_device(wrapped_model)
else:
nncf_logger.warning("Model is on multiple devices. Cannot determine device for loaded modules.")

hook_storage = get_hook_storage(wrapped_model)
transformation_commands = cast(List[S_COMMAND], config[COMPRESSION_STATE_ATTR])
for command in transformation_commands:
module_cls = COMPRESSION_MODULES.get(command["module_cls_name"])
module = module_cls.from_config(command["module_config"])
module.to(device)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I encountered the same problem with the model on various devices:
https://github.com/openvinotoolkit/nncf/pull/3341/files#diff-f257219eb5631518ca7ec37bb9858c77491049dc999ca14d845a46517f263900R135-R136

Is there a reliable method to match the compression module from the insertion command to the target layer and its parameters? With this matching, it would be possible to determine the device.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently we have limited support for multidevice models.
For not multi device model it works.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multidevice case is very popular for LLMs. @AlexanderDokuchaev, do you have any idea how torch2 will support it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not look at it. Currently multidevice support for new tracing works similar like for current.

@github-actions github-actions bot added the documentation Improvements or additions to documentation label Mar 14, 2025
alexsu52
alexsu52 reviewed Mar 17, 2025
View reviewed changes
docs/usage/training_time_compression/quantization_aware_training/Usage.md
@@ -96,7 +96,7 @@ resuming_checkpoint = torch.load(path)
nncf_config = resuming_checkpoint['nncf_config']
state_dict = resuming_checkpoint['state_dict']

quantized_model = nncf.torch.load_from_config(model, nncf_config, dummy_input)
quantized_model = nncf.torch.load_from_config(model, nncf_config)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the main Readme:

  • using nncf.torch.get_config() / nncf.torch.load_from_config() in Training-Time Quantization
  • add note that create_compressed API is deprecated in Training-Time Compression.

OpenVINO documentation should be also update regarding the changes.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using nncf.torch.get_config() / nncf.torch.load_from_config() in Training-Time Quantization

done

add note that create_compressed API is deprecated in Training-Time Compression.

we have already have deprecation message in create_compressed sine previous release, what a reason to duplicate it in readme now?

nncf/experimental/torch2/function_hook/serialization.py
Comment on lines +108 to +119
device = None
if not is_multidevice(wrapped_model):
device = get_model_device(wrapped_model)
else:
nncf_logger.warning("Model is on multiple devices. Cannot determine device for loaded modules.")

hook_storage = get_hook_storage(wrapped_model)
transformation_commands = cast(List[S_COMMAND], config[COMPRESSION_STATE_ATTR])
for command in transformation_commands:
module_cls = COMPRESSION_MODULES.get(command["module_cls_name"])
module = module_cls.from_config(command["module_config"])
module.to(device)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multidevice case is very popular for LLMs. @AlexanderDokuchaev, do you have any idea how torch2 will support it?

@AlexanderDokuchaev AlexanderDokuchaev marked this pull request as draft March 19, 2025 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ljaljushkin ljaljushkin

@alexsu52 alexsu52

Requested changes must be addressed to merge this pull request.

Assignees

@alexsu52 alexsu52

Labels
documentation Improvements or additions to documentation experimental NNCF Common Pull request that updates NNCF Common NNCF PT Pull requests that updates NNCF PyTorch NNCF PTQ Pull requests that updates NNCF PTQ
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@AlexanderDokuchaev @alexsu52 @ljaljushkin