Autoware Sensing Perception Working Group Meeting 2024-05-29

[drwnz]

Administrative

• Previous meeting
• Meeting time: Every second Wednesday, 10am UTC (alternating with Mapping WG)
• WG Google Drive
• Wiki page for this Working Group
• Shared issue list

Attendees

Chaired by David

Ryohsuke
Taekjin
Koji
Shunsuke
Kaan
Fatih
Dai
Armagan
Leonardo

Agenda

• Introduction of new attendees
  • Welcome Leonardo Rossetti from Red Hat
• Review of Issue List and assignment
• Review of Project Board and assignment
• Nebula update: https://github.com/tier4/nebula/tree/develop

Discussion topics

• Armagan: Issue related to team orientation and priority setting

• In Autoware labs there are some items to be achieved

  • https://github.com/orgs/autowarefoundation/discussions/4550
  • Some issues related to datasets and improving data, and also performance of sensing-perception
  • Want to to see performance of current systems
  • Less focus on new features, more on current performace
  • Therefore evaluation is necessary - how can we evaluate Autoware?
  • Needs more than just checking videos
  • Minoda: Autoware perception evaluator: https://github.com/tier4/autoware_perception_evaluation
    • Driving log replayer: https://github.com/tier4/driving_log_replayer
    • Also have evaluator within TIER IV's product, Web.Auto
  • Armagan: is Perception Evaluator mandatory for development? Any value in testing in AWSIM for evaluation?
    • Minoda: not mandatory. Only works with collected data, not simulation.
    • Takejin: Probably more in tracking, AWSIM would be useful (currently pass/fail tests bit not metric performance)
  • Armagan: what about testing in against a leaderboard?
    • Minoda: We internally evaluate on ML dataset metrics when integrating new dataset, and Autoware Evaluation pipeline.
    • Armagan: results should be repeatable and universal
  • David: what about multiple reference designs?
    • Aramgan: you can choose, as long as it is common
  • David: Universal dataset conversation has come up multiple times: perhaps dataset from Co-MLops?
  • Fatih: Autoware Perception Evaluator or other Autoware Evaluation methods: even if we don't see the datasets, we still want to know the test date, sensor configuration, size of dataset. Nuscenes also gave Autoware permission to evaluate against their dataset.
    • Minoda: do you want evaluation results in PR for example?
    • Fatih: not every PR, but results could be shared with existing algos (in documetnation for example)
  • David: Let's introduce the Autoware Evaluator as used by TIER IV at next WG. Maybe also have a report on evaluation progress every WG. Does LeoDrive have any evaluator work?
    • Currently no, but Kaan did some work on Waymo dataset: https://github.com/autowarefoundation/perception_benchmark_tool
    • Pending task assigned to LeoDrive member: Evaluate prediction accuracy with nuScenes autoware.universe#6806
  • Fatih: do you have speed benchmarks, or just accuracy?
    • Minoda: we were working on it but not sure of latest status
    • Taekjin: messages to monitor processing times, but not direct measurement
    • Fatih: could set up an AWS instance to run specific parts of Autoware and monitor CPU/GPU usage automatically

• David: Nebula update refactor(hesai)!: combine Hesai ROS wrappers into a single node tier4/nebula#127

  • Mehmet's update should be reviewed and merged
  • Robosense PR also needs a review from Mehmet

• Armagan: currently testing and having issues with trees

  • The tree filter based on height does not work
  • Two issues: trees and hanging branches, z-filter is not enough, and bushes on the side of the road
  • Minoda: have initiative for unknown object detection, trying to add algorightms with an architecture modification. Attempting to use semantic segmentation in camera to remove false positives.
  • Dai: Currently still working on semantic segmentation bvased pointcloud filtering, currently dealing with bugs in semantic segmentation causing performance to not be sufficient
    • feat(tensorrt yolox): inference and publish mask image from yolox model with semantic segmentation header autoware.universe#5553
    • feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter autoware.universe#5562
  • Also merged intensity filter: Add intensity for poincloud processing in perception and an intensity_based_unknown_validator autoware.universe#6785

Action items

• Prepare presentation for TIER IV's evaluation for next WG meeting