retry rtiddsgen(_server) invocation if it fails with a return code

[dirk-thomas]

When the buildfarm rebuilds the majority of the Debian packages of a ROS distro it frequently happens that packages containing messages fail to build.

The invocation of rtiddsgen_server returns with a non-zero code and the following error message (from http://build.ros2.org/view/Ebin_uB64/job/Ebin_uB64__action_msgs__ubuntu_bionic_amd64__binary/35/consoleFull):

ERROR com.rti.ndds.nddsgen.Main Fail: com.rti.ndds.nddsgen.Main$ArgumentException: File /tmp/binarydeb/ros-eloquent-action-msgs-0.8.0/obj-x86_64-linux-gnu/rosidl_generator_dds_idl/action_msgs/msg/dds_connext/GoalInfo_.idl doesn't exist. Please make sure that you have specified the correct path.

The logic already retries the invocation it finished with a rc 0 but didn't generate the desired files (

rosidl_typesupport_connext/rosidl_typesupport_connext_cpp/rosidl_typesupport_connext_cpp/__init__.py

Lines 67 to 88 in 2a56f06

	count = 1
	max_count = 5
	while True:
	subprocess.check_call(cmd)
	# fail safe if the generator does not work as expected
	any_missing = False
	for suffix in ['.h', '.cxx', 'Plugin.h', 'Plugin.cxx', 'Support.h', 'Support.cxx']:
	filename = os.path.join(output_path, msg_name + suffix)
	if not os.path.exists(filename):
	any_missing = True
	break
	if not any_missing:
	break
	print("'%s' failed to generate the expected files for '%s/%s'" %
	(idl_pp, pkg_name, msg_name), file=sys.stderr)
	if count < max_count:
	count += 1
	print('Running code generator again (retry %d of %d)...' %
	(count, max_count), file=sys.stderr)
	continue
	raise RuntimeError('failed to generate the expected files')

). But that doesn't seem to be the problem here. The command fails with a non-zero return code.

The logic already explicitly checks that the IDL file passed to the generator exists (

rosidl_typesupport_connext/rosidl_typesupport_connext_cpp/rosidl_typesupport_connext_cpp/__init__.py

Line 38 in 2a56f06

assert os.path.exists(idl_file), 'Could not find IDL file: ' + idl_file

) but the invocation still results in the above error message.

I can't reproduce the problem locally and newly triggered builds of the same job usually pass. So this patch is mostly an attempt to recover from this error condition and hopefully will make the binarydeb jobs pass reliably on the first attempt.

It could be a race deleting the IDL after the initial check (hence the new additional check after an invocation failed) but it is unclear why the file would be deleted. The other possibility is that the code generator fails for whatever other reason but reports a bogus error message...

[jacobperron]

I think it would good to open a issue tracking the problem and reference it in the code.

[dirk-thomas]

I think it would good to open a issue tracking the problem and reference it in the code.

I added HACK(...) to the new retry logic in f1ff333. I would only open a new ticket after seeing the results of the rebuild depending if they confirm or debunk the hunch.

[jacobperron]

Ah, I think this is the relevant ticket: #7

[jacobperron]

Let's see what happens I suppose.

[dirk-thomas]

Let's see what happens I suppose.

(Assuming there is no logic bug in this patch) worst case should be a no-op when the generator passes and take extra time when it repeatedly fails.

Sanity check that it still builds on all platforms:

• Linux
• macOS
• Windows

[dirk-thomas]

@mjcarroll Can you approve this being backported and released into Eloquent?

[mjcarroll]

Seems reasonable, and CI looks good. (also yes to eloquent)

[dirk-thomas]

Fast forwarded the commit from master to eloquent, released 0.8.3, and fast forwarded the release from the eloquent branch back to master. So both branches are still in sync.

[dirk-thomas]

A build where the anticipated failure happened: http://build.ros2.org/view/Ebin_uB64/job/Ebin_uB64__rcl_interfaces__ubuntu_bionic_amd64__binary/42/consoleFull

But all retries failed with the same error message even though they passed the assertion that the passed IDL file does exist between each attempt 😟