Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix exclusive hardware control mode switching on controller failed activation #1522

Open
wants to merge 19 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
7440aea
added an actuator interface with exclusive interface handling
Apr 19, 2024
d181630
ran precommit
Apr 19, 2024
a3814c0
renamed into test_actuator_exclusive_interfaces
Apr 19, 2024
a8a1a68
formatting fix
Apr 19, 2024
7b07afc
added a test controller that fails at activation
Apr 19, 2024
daa9ce7
forgot to run pre-commit
Apr 19, 2024
6a29b34
Fix issues in the setup of the test controller that fails on activate
saikishor Apr 29, 2024
848c8ae
Added test to reproduce the failure of switching with exclusive inter…
saikishor Apr 29, 2024
17bb7a2
Throw an exception when requesting already claimed exclusive interface
saikishor Apr 29, 2024
77e932e
If the controller activation fails then release the claimed interfaces
saikishor Apr 29, 2024
4a4d780
add the perform switching logic to fix the issue with exclusive inter…
saikishor Apr 29, 2024
1236e5c
Merge branch 'master' into fix/exclusive_hw_interface_switching
saikishor Apr 30, 2024
de9dc03
remove the scoping
saikishor May 2, 2024
96b5a8c
Merge branch 'master' into fix/exclusive_hw_interface_switching
saikishor May 2, 2024
8b17d8e
Merge branch 'master' into fix/exclusive_hw_interface_switching
saikishor May 3, 2024
6df3c72
Merge branch 'master' into fix/exclusive_hw_interface_switching
bmagyar May 7, 2024
fba2abd
Collect all failed activated controllers and perform command switch a…
saikishor May 13, 2024
a670590
Merge branch 'master' into fix/exclusive_hw_interface_switching
saikishor May 13, 2024
7755df0
added some documentation about the determinism
saikishor May 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions controller_manager/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -171,13 +171,28 @@ if(BUILD_TESTING)
DESTINATION lib
)

add_library(test_controller_failed_activate SHARED
test/test_controller_failed_activate/test_controller_failed_activate.cpp
)
target_link_libraries(test_controller_failed_activate PUBLIC
controller_manager
)
target_compile_definitions(test_controller_failed_activate PRIVATE "CONTROLLER_MANAGER_BUILDING_DLL")
pluginlib_export_plugin_description_file(
controller_interface test/test_controller_failed_activate/test_controller_failed_activate.xml)
install(
TARGETS test_controller_failed_activate
DESTINATION lib
)

ament_add_gmock(test_release_interfaces
test/test_release_interfaces.cpp
APPEND_ENV AMENT_PREFIX_PATH=${ament_index_build_path}_$<CONFIG>
)
target_link_libraries(test_release_interfaces
controller_manager
test_controller_with_interfaces
test_controller_failed_activate
ros2_control_test_assets::ros2_control_test_assets
)

Expand Down
19 changes: 18 additions & 1 deletion controller_manager/src/controller_manager.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1508,10 +1508,27 @@ void ControllerManager::activate_controllers(
{
RCLCPP_ERROR(
get_logger(),
"After activation, controller '%s' is in state '%s' (%d), expected '%s' (%d).",
"After activation, controller '%s' is in state '%s' (%d), expected '%s' (%d). Releasing "
"interfaces!",
controller->get_node()->get_name(), new_state.label().c_str(), new_state.id(),
hardware_interface::lifecycle_state_names::ACTIVE,
lifecycle_msgs::msg::State::PRIMARY_STATE_ACTIVE);
controller->release_interfaces();
{
saikishor marked this conversation as resolved.
Show resolved Hide resolved
// Now prepare and perform the stop interface switching as this is needed for exclusive
// interfaces
if (
!command_interface_names.empty() &&
(!resource_manager_->prepare_command_mode_switch({}, command_interface_names) ||
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not call this method here, as we are in the RT (update) loop, and this method is declared ad non-realtime relevant in the hardware description, as it is usually called from the non-rt loop (service callback).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you! I completely forgot that this is a non-RT method. We can do it at the end as you mentioned in the other comment

!resource_manager_->perform_command_mode_switch({}, command_interface_names)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have here a bit of a chicken-egg problem. Currently we don't switch controller if this command fails (see line 2112), and we react on an error in general and not an error of specific hardware. We will have to change this in the future, but not for now as it seems not be a problem.

Back to the topic...

I thinks that a cleaner soluation would be to gather all the command interfaces of controller that failed to start and then do only once call on this method, to poke arround HW as little as possible. In current implementation we are doing N-time check in M HW regaridng the interface, if we add this to the line 2136 then we will do only 1xM checks.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would do as you said, to trigger for all failed controllers at the end!

{
RCLCPP_ERROR(
get_logger(),
"Error switching back the interfaces in the hardware as the controller activation "
"failed.");
}
}
return;
}

// if it is a chainable controller, make the reference interfaces available on activation
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
// Copyright 2021 Department of Engineering Cybernetics, NTNU.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

#include "test_controller_failed_activate.hpp"

#include <memory>
#include <string>

#include "lifecycle_msgs/msg/transition.hpp"

namespace test_controller_failed_activate
{
TestControllerFailedActivate::TestControllerFailedActivate()
: controller_interface::ControllerInterface()
{
}

rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn
TestControllerFailedActivate::on_init()
{
return rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn::SUCCESS;
}

controller_interface::return_type TestControllerFailedActivate::update(
const rclcpp::Time & /*time*/, const rclcpp::Duration & /*period*/)
{
return controller_interface::return_type::OK;
}

rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn
TestControllerFailedActivate::on_configure(const rclcpp_lifecycle::State & /*previous_state&*/)
{
return rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn::SUCCESS;
}

rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn
TestControllerFailedActivate::on_activate(const rclcpp_lifecycle::State & /*previous_state&*/)
{
// Simply simulate a controller that can not be activated
return rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn::FAILURE;
}

rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn
TestControllerFailedActivate::on_cleanup(const rclcpp_lifecycle::State & /*previous_state*/)
{
return rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn::SUCCESS;
}

} // namespace test_controller_failed_activate

#include "pluginlib/class_list_macros.hpp"

PLUGINLIB_EXPORT_CLASS(
test_controller_failed_activate::TestControllerFailedActivate,
controller_interface::ControllerInterface)
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
// Copyright 2020 Department of Engineering Cybernetics, NTNU
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

#ifndef TEST_CONTROLLER_FAILED_ACTIVATE__TEST_CONTROLLER_FAILED_ACTIVATE_HPP_
#define TEST_CONTROLLER_FAILED_ACTIVATE__TEST_CONTROLLER_FAILED_ACTIVATE_HPP_

#include <memory>
#include <string>

#include "controller_interface/visibility_control.h"
#include "controller_manager/controller_manager.hpp"

namespace test_controller_failed_activate
{
// Corresponds to the name listed within the pluginglib xml
constexpr char TEST_CONTROLLER_WITH_INTERFACES_CLASS_NAME[] =
"controller_manager/test_controller_failed_activate";
// Corresponds to the command interface to claim
constexpr char TEST_CONTROLLER_COMMAND_INTERFACE[] = "joint2/velocity";
class TestControllerFailedActivate : public controller_interface::ControllerInterface
{
public:
CONTROLLER_MANAGER_PUBLIC
TestControllerFailedActivate();

CONTROLLER_MANAGER_PUBLIC
virtual ~TestControllerFailedActivate() = default;

controller_interface::InterfaceConfiguration command_interface_configuration() const override
{
return controller_interface::InterfaceConfiguration{
controller_interface::interface_configuration_type::INDIVIDUAL,
{TEST_CONTROLLER_COMMAND_INTERFACE}};
}

controller_interface::InterfaceConfiguration state_interface_configuration() const override
{
return controller_interface::InterfaceConfiguration{
controller_interface::interface_configuration_type::NONE};
}

CONTROLLER_MANAGER_PUBLIC
controller_interface::return_type update(
const rclcpp::Time & time, const rclcpp::Duration & period) override;

CONTROLLER_MANAGER_PUBLIC
rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn on_init() override;

CONTROLLER_MANAGER_PUBLIC
rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn on_configure(
const rclcpp_lifecycle::State & previous_state) override;

CONTROLLER_MANAGER_PUBLIC
rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn on_activate(
const rclcpp_lifecycle::State & previous_state) override;

CONTROLLER_MANAGER_PUBLIC
rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn on_cleanup(
const rclcpp_lifecycle::State & previous_state) override;
};

} // namespace test_controller_failed_activate

#endif // TEST_CONTROLLER_FAILED_ACTIVATE__TEST_CONTROLLER_FAILED_ACTIVATE_HPP_
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<library path="test_controller_failed_activate">

<class name="controller_manager/test_controller_failed_activate" type="test_controller_failed_activate::TestControllerFailedActivate" base_class_type="controller_interface::ControllerInterface">
<description>
Controller used for testing
</description>
</class>

</library>
80 changes: 80 additions & 0 deletions controller_manager/test/test_release_interfaces.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
#include "controller_manager/controller_manager.hpp"
#include "controller_manager_test_common.hpp"
#include "lifecycle_msgs/msg/state.hpp"
#include "test_controller_failed_activate/test_controller_failed_activate.hpp"
#include "test_controller_with_interfaces/test_controller_with_interfaces.hpp"

using ::testing::_;
Expand All @@ -31,6 +32,19 @@ class TestReleaseInterfaces : public ControllerManagerFixture<controller_manager
{
};

class TestReleaseExclusiveInterfaces
: public ControllerManagerFixture<controller_manager::ControllerManager>
{
public:
TestReleaseExclusiveInterfaces()
: ControllerManagerFixture<controller_manager::ControllerManager>(
std::string(ros2_control_test_assets::urdf_head) +
std::string(ros2_control_test_assets::hardware_resources_with_exclusive_interface) +
std::string(ros2_control_test_assets::urdf_tail))
{
}
};

TEST_F(TestReleaseInterfaces, switch_controllers_same_interface)
{
std::string controller_type =
Expand Down Expand Up @@ -199,3 +213,69 @@ TEST_F(TestReleaseInterfaces, switch_controllers_same_interface)
abstract_test_controller2.c->get_state().id());
}
}

TEST_F(TestReleaseExclusiveInterfaces, test_exclusive_interface_switching_failure)
{
std::string controller_type =
test_controller_failed_activate::TEST_CONTROLLER_WITH_INTERFACES_CLASS_NAME;

// Load two controllers of different names
std::string controller_name1 = "test_controller1";
std::string controller_name2 = "test_controller2";
ASSERT_NO_THROW(cm_->load_controller(controller_name1, controller_type));
ASSERT_NO_THROW(cm_->load_controller(
controller_name2, test_controller_with_interfaces::TEST_CONTROLLER_WITH_INTERFACES_CLASS_NAME));
ASSERT_EQ(2u, cm_->get_loaded_controllers().size());
controller_manager::ControllerSpec abstract_test_controller1 = cm_->get_loaded_controllers()[0];
controller_manager::ControllerSpec abstract_test_controller2 = cm_->get_loaded_controllers()[1];

// Configure controllers
ASSERT_EQ(controller_interface::return_type::OK, cm_->configure_controller(controller_name1));
ASSERT_EQ(controller_interface::return_type::OK, cm_->configure_controller(controller_name2));

ASSERT_EQ(
lifecycle_msgs::msg::State::PRIMARY_STATE_INACTIVE,
abstract_test_controller1.c->get_state().id());
ASSERT_EQ(
lifecycle_msgs::msg::State::PRIMARY_STATE_INACTIVE,
abstract_test_controller2.c->get_state().id());

{ // Test starting the first controller
RCLCPP_INFO(cm_->get_logger(), "Starting controller #1");
std::vector<std::string> start_controllers = {controller_name1};
std::vector<std::string> stop_controllers = {};
auto switch_future = std::async(
std::launch::async, &controller_manager::ControllerManager::switch_controller, cm_,
start_controllers, stop_controllers, STRICT, true, rclcpp::Duration(0, 0));
ASSERT_EQ(std::future_status::timeout, switch_future.wait_for(std::chrono::milliseconds(100)))
<< "switch_controller should be blocking until next update cycle";
ControllerManagerRunner cm_runner(this);
EXPECT_EQ(controller_interface::return_type::OK, switch_future.get());
ASSERT_EQ(
lifecycle_msgs::msg::State::PRIMARY_STATE_INACTIVE,
abstract_test_controller1.c->get_state().id());
ASSERT_EQ(
lifecycle_msgs::msg::State::PRIMARY_STATE_INACTIVE,
abstract_test_controller2.c->get_state().id());
}

{ // Test starting the second controller when the first is running
// Fails as they have the same command interface
RCLCPP_INFO(cm_->get_logger(), "Starting controller #2");
std::vector<std::string> start_controllers = {controller_name2};
std::vector<std::string> stop_controllers = {};
auto switch_future = std::async(
std::launch::async, &controller_manager::ControllerManager::switch_controller, cm_,
start_controllers, stop_controllers, STRICT, true, rclcpp::Duration(0, 0));
ASSERT_EQ(std::future_status::timeout, switch_future.wait_for(std::chrono::milliseconds(100)))
<< "switch_controller should be blocking until next update cycle";
ControllerManagerRunner cm_runner(this);
EXPECT_EQ(controller_interface::return_type::OK, switch_future.get());
ASSERT_EQ(
lifecycle_msgs::msg::State::PRIMARY_STATE_INACTIVE,
abstract_test_controller1.c->get_state().id());
ASSERT_EQ(
lifecycle_msgs::msg::State::PRIMARY_STATE_ACTIVE,
abstract_test_controller2.c->get_state().id());
}
}
10 changes: 6 additions & 4 deletions hardware_interface_testing/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,14 @@ foreach(Dependency IN ITEMS ${THIS_PACKAGE_INCLUDE_DEPENDS})
endforeach()

add_library(test_components SHARED
test/test_components/test_actuator.cpp
test/test_components/test_sensor.cpp
test/test_components/test_system.cpp)
test/test_components/test_actuator.cpp
test/test_components/test_sensor.cpp
test/test_components/test_system.cpp
test/test_components/test_actuator_exclusive_interfaces.cpp
)
ament_target_dependencies(test_components hardware_interface pluginlib ros2_control_test_assets)
install(TARGETS test_components
DESTINATION lib
DESTINATION lib
)
pluginlib_export_plugin_description_file(
hardware_interface test/test_components/test_components.xml)
Expand Down
Loading
Loading