Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add comprehensive testing strategy for real-world deployment on iCub and ErgoCub #76

Closed
flferretti opened this issue Feb 2, 2024 · 5 comments · Fixed by #106
Closed
Assignees

Comments

@flferretti
Copy link
Collaborator

The current testing framework for jaxsim lacks validation against the actual robots intended for deployment. This issue aims to address this gap by establishing a comprehensive testing strategy that ensures the reliability and performance of the simulator in real-world scenarios.

Perhaps we can test against also the iCub/ergoCub models, as most probably those are the one we are going to use the feature against?

Kind of unrelated to the PR, but just because I realized it: testing real world models is ok, but in my experience you may have some blinds spots. In iDynTree I historically found many problems with programmatically generated random models, generated by getRandomModel function.

It could make sense to do something similar here, or also directly use it here as we can export the randomly generated model to URDF, something like

import idyntree.bindings as idt
nrOfJoints = 20
randomModel = idt.getRandomModel(nrOfJoints)
mdlExport = idt.ModelExporter()
mdlExport.init(randomModel)
urdfString = "dummy"
urdfFilePath = "/dummy/dummy"
mdlExport.exportModelToString(urdfString)
# or
mdlExport.exportModelToFile(urdfFilePath)

Originally posted by @traversaro in #75 (comment)

@diegoferigo
Copy link
Member

diegoferigo commented Feb 2, 2024

I agree about testing on iCub/ErgoCub models, but note that the JIT performance we currently have would make the tests take really a lot of time, and they are already very slow. This was the original reason to test smaller fixed-base and floating-base models.

@traversaro
Copy link
Contributor

I agree about testing on iCub/ErgoCub models, but note that the JIT performance we currently have would make the tests take really a lot of time, and they are already very slow. This was the original reason to test smaller fixed-base and floating-base models.

Ah, good point. I was not considering that.

@diegoferigo
Copy link
Member

In #106, I'm testing -among other models- a reduced version of the ErgoCub robot. It could be a good tradeoff between CI time and comprehensiveness of the testing strategy.

@diegoferigo diegoferigo linked a pull request Mar 11, 2024 that will close this issue
@diegoferigo
Copy link
Member

Furthermore, random models could be created with rod similarly to the existing box and sphere models.

@diegoferigo
Copy link
Member

The testing suite targeting the new functional APIs test against a reduced version of ErgoCub (only the DoFs of the legs are kept). It seems a acceptable trade-off between the comprehensiveness of the tests and the time necessary to run them (~20 mins at the time of writing).

I guess we can close this issue now.

@diegoferigo diegoferigo self-assigned this Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants