First Person Perception Stack

Overview

The fp_perception package provides a standardized, plugin-based architecture for managing drivers (and related processing) that power perception systems in human-robot interaction (HRI) scenarios.

It enables modular development and dynamic loading of components at runtime using ROS 2's pluginlib, simplifying integration, testing, and extension of perception pipelines.

Documentation

Start here:

• Server plugin loading and device and data acquisition
• ROS topics and services
• Testing the System and Interfaces
• Base Classes
• Plugin Configuration

Drivers:

• Audio Drivers and AudioData Acquisition
• Vision Drivers and ImageData Acquisition
• Transcription Drivers and Transcription Service
• Speech Drivers and Speech Synthesis
• Sentiment Drivers and Sentiment Analysis
• Image Analysis Drivers

Key Features

• Plugin-based driver framework Integrate new sensors and drivers without changing the core system.

• Reusable base classes for Physical device drivers (DriverBase) and REST-backed drivers (RestBase)

• Designed with HRI in mind – enabling perception for robots that does not have built-in perception capabilities.

Plugins

Driver Plugins

• DriverBase is used to implement drivers that interact with perception hardware (e.g., cameras, audio devices).

• RestBase is used to implement drivers that interact with REST APIs (e.g., OpenAI, HuggingFace).

• Driver READMEs:

  • fp_perception_driver_vision/readme.md
  • fp_perception_driver_audio/readme.md

• Plugin Configuration:

  • docs/plugin_configuration.md

Build the system

Create the workspace folder

mkdir -p perception_ws/src
cd perception_ws/src

Clone the repository

git clone https://github.com/CollaborativeRoboticsLab/fp_perception.git

Install dependencies

cd ..
sudo apt update
sudo apt install libportaudio2 portaudio19-dev python3-pyaudio ros-${ROS_DISTRO}-vision-opencv

for any missing dependencies

rosdep install --from-paths src --ignore-src -r -y

build the workspace

colcon build

Start the system

If using microphone or speaker, configure the audio drivers by device_name when possible. The test guide shows how to confirm the selected PortAudio device and map it to an ALSA plughw:X,Y route for direct WAV playback checks.

python3 src/fp_perception/fp_perception_driver_audio/find_devices.py

See Test the System for startup validation, generated WAV checks, and service-call examples.

The launch files load the installed config from install/fp_perception/share/fp_perception/config/config.yaml. After editing src/fp_perception/fp_perception/config/config.yaml, rebuild fp_perception and re-source the workspace before relaunching.

export OPENAI_API_KEY=
export HUGGINGFACE_API_KEY=
source install/setup.bash
ros2 launch fp_perception server.launch.py

Writing Your Own Plugins

1. Inherit from fp_perception::DriverBase (or fp_perception::RestBase for REST-backed drivers).
2. Implement required methods (initialize, deinitialize) and whichever data methods you need (getData*, setData*).
3. Register your plugin using PLUGINLIB_EXPORT_CLASS.
4. Define a plugin XML manifest.

Future Extensions

• Dynamic runtime reconfiguration
• Richer diagnostics dashboards and tooling
• Visualization tools for debugging perception pipelines
• Performance benchmarking and logging