Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add performance report for latest version with minor fixes to scripts #193

Merged
merged 1 commit into from
May 21, 2023

Conversation

glpuga
Copy link
Collaborator

@glpuga glpuga commented May 20, 2023

Proposed changes

  • Fixes a permissions issue that prevented scripts in beluga_benchmark from running if not built with --symlink-install
  • Adds the --params-file optional argument to the parameterized_run script, which was cited in reports, but was missing from script.
  • Adds an up-to-date report comparing beluga_amcl to nav2_amcl, here: link to report
  • Includes a summary of the report in the beluga_amcl README.md
  • Improves the output of the compare_results.py script by adding a grid to the plots and making the axes logarithmic.

Type of change

  • 🐛 Bugfix (change which fixes an issue)
  • 🚀 Feature (change which adds functionality)
  • 📚 Documentation (change which fixes or extends documentation)

Checklist

Put an x in the boxes that apply. This is simply a reminder of what we will require before merging your code.

  • Lint and unit tests (if any) pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)
  • All commmits have been signed for DCO

@glpuga glpuga changed the title Add report for latest version with minor fixes Add report for latest version with minor fixes to scripts May 20, 2023
@glpuga glpuga force-pushed the glpuga/report_may branch 2 times, most recently from d618404 to 35cf6d7 Compare May 20, 2023 23:04
@glpuga
Copy link
Collaborator Author

glpuga commented May 20, 2023

@nahueespinosa If we confirm these results, the multi threaded configuration should not be the default for normal amounts of particles. Look at the CPU usage for beluga in both cases when particle count is below 2k.

Beluga (non-multithreaded) vs. AMCL

beluga_seq_vs_amcl

Beluga (multithreaded) vs. AMCL

beluga_par_vs_amcl

@glpuga glpuga changed the title Add report for latest version with minor fixes to scripts Add performance report for latest version with minor fixes to scripts May 20, 2023
@glpuga glpuga force-pushed the glpuga/report_may branch 3 times, most recently from 7e57e3b to cd79676 Compare May 21, 2023 13:52
@glpuga
Copy link
Collaborator Author

glpuga commented May 21, 2023

I extended the report with results for the beam model too. The report is here: https://github.com/Ekumen-OS/beluga/blob/glpuga/report_may/beluga_benchmark/docs/reports/2023-05-20/REPORT.md

Copy link
Member

@nahueespinosa nahueespinosa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@glpuga Awesome!

the multi threaded configuration should not be the default for normal amounts of particles

I agree!

beluga_amcl/README.md Outdated Show resolved Hide resolved
beluga_amcl/README.md Outdated Show resolved Hide resolved
beluga_benchmark/docs/reports/2023-05-20/REPORT.md Outdated Show resolved Hide resolved
beluga_benchmark/docs/reports/2023-05-20/REPORT.md Outdated Show resolved Hide resolved
beluga_benchmark/docs/reports/2023-05-20/REPORT.md Outdated Show resolved Hide resolved
@nahueespinosa nahueespinosa added documentation Improvements or additions to documentation enhancement New feature or request labels May 21, 2023
Signed-off-by: Gerardo Puga <glpuga@ekumenlabs.com>
Copy link
Member

@nahueespinosa nahueespinosa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@glpuga glpuga merged commit 705dedd into main May 21, 2023
5 checks passed
@glpuga glpuga deleted the glpuga/report_may branch May 21, 2023 14:32
glpuga added a commit that referenced this pull request May 30, 2023
This PR proposes chaging the default threading mode to "sequential" on
the basis that while parallel has lower latency, the required increase
in CPU usage makes it expensive for users not concerned about latency in
the lower end of particle counts. See #193 .

Signed-off-by: Gerardo Puga <glpuga@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants