Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Benchmark] Extend Benchmark to all model type extensions #5241

Merged

Conversation

patrickvonplaten
Copy link
Contributor

This PR does the following changes:

    • The default model class to benchmark is the one that can be found under config.architectures
    • Improve plotting file

@codecov
Copy link

codecov bot commented Jun 24, 2020

Codecov Report

Merging #5241 into master will increase coverage by 0.44%.
The diff coverage is 40.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #5241      +/-   ##
==========================================
+ Coverage   77.49%   77.93%   +0.44%     
==========================================
  Files         138      138              
  Lines       23787    23806      +19     
==========================================
+ Hits        18433    18554     +121     
+ Misses       5354     5252     -102     
Impacted Files Coverage Δ
src/transformers/benchmark/benchmark.py 74.01% <33.33%> (-5.12%) ⬇️
src/transformers/benchmark/benchmark_tf.py 79.81% <37.50%> (-2.88%) ⬇️
src/transformers/benchmark/benchmark_args_utils.py 89.36% <100.00%> (+0.23%) ⬆️
src/transformers/benchmark/benchmark_utils.py 69.84% <100.00%> (+0.07%) ⬆️
src/transformers/data/processors/squad.py 28.66% <0.00%> (-28.03%) ⬇️
src/transformers/modeling_tf_pytorch_utils.py 88.05% <0.00%> (-1.26%) ⬇️
src/transformers/tokenization_bert.py 90.45% <0.00%> (-0.83%) ⬇️
src/transformers/file_utils.py 76.42% <0.00%> (-0.39%) ⬇️
src/transformers/tokenization_utils.py 94.81% <0.00%> (-0.38%) ⬇️
src/transformers/modeling_tf_utils.py 86.00% <0.00%> (+0.29%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ae132a...a007369. Read the comment docs.

has_model_class_in_config = hasattr(config, "architecture") and len(config.architectures) > 1
if not self.args.only_pretrain_model and has_model_class_in_config:
try:
model_class = config.architectures[0]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tagging @LysandreJik @julien-c for notification. Benchmarks are extended to use the model class as written in their config.architecture list.

Merging for now, I think this should be good.

@patrickvonplaten patrickvonplaten merged commit 9fe09ce into huggingface:master Jun 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant