Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[microNPU] Add MobileNetv2 test to network tests #10242

Closed
wants to merge 2 commits into from

Conversation

jacobbohlin
Copy link
Contributor

This adds MobileNetv2 tflite testcase to the microNPU's network tests.

Co-authored-by: Jacob Bohlin jacob.bohlin@arm.com

@jacobbohlin
Copy link
Contributor Author

Copy link
Contributor

@lhutton1 lhutton1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jacobbohlin! I left a few comments, let me know what you think

Comment on lines 94 to 95
if __name__ == "__main__":
test_forward_mobilenet_v1()
test_forward_mobilenet_v1(ACCEL_TYPES[0])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could just remove this section completely?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that's probably better.


mod = partition_for_ethosu(relay_mod, params)
compiled_models = infra.build_source(
mod, input_data, output_data, accel_type, output_tolerance=10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be interesting to see if the output tolerance could be lowered now that we support more operators

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can give that a try.

@@ -65,5 +65,31 @@ def test_forward_mobilenet_v1(accel_type):
infra.verify_source(compiled_models, accel_type)


def test_forward_mobilenet_v2(accel_type="ethos-u55-256"):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth parameterising this network like the above?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot say I have a definitive answer for this. Probably not the most efficient way to improve test coverage relative to test time.

@manupak
Copy link
Contributor

manupak commented Feb 15, 2022

Thanks @jacobbohlin

I think we could refactor the tests here to be a single one with stimulus being the model url and accelerator variant.
Moreover, lets write out a tuples of model url and accelerator variants for each tests -- therefore we have more control/visibility of what is being tested as opposed default parametric explosion of the test space.

Some references that we have done this at other places, also feel free to migrate any utility that you feel we can re-use to aot_test_utils

def _get_relay_module_and_inputs_from_tflite_file(tflite_model_file):
with open(tflite_model_file, "rb") as f:
tflite_model_buf = f.read()
mod, params = convert_to_relay(tflite_model_buf)
inputs = dict()
for param in mod["main"].params:
name = str(param.name_hint)
data_shape = [int(i) for i in param.type_annotation.shape]
dtype = str(param.type_annotation.dtype)
in_min, in_max = (np.iinfo(dtype).min, np.iinfo(dtype).max)
data = np.random.randint(in_min, high=in_max, size=data_shape, dtype=dtype)
inputs[name] = data
return mod, inputs, params

@pytest.mark.parametrize(
"model_url, usmp_algo, workspace_size,",
[
(MOBILENET_V1_URL, "greedy_by_size", 4845696),
(MOBILENET_V1_URL, "greedy_by_conflicts", 4444288),
(MOBILENET_V1_URL, "hill_climb", 3240064),
],
)
def test_tflite_model_u1_usecase(model_url, usmp_algo, workspace_size):

@jacobbohlin
Copy link
Contributor Author

Thanks @manupa-arm. That is a much better approach, I will update according to your suggestions.

Copy link
Contributor

@lhutton1 lhutton1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jacobbohlin LGTM!

@jacobbohlin
Copy link
Contributor Author

Closing this since it has been superseded by #10362.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants