http_relay: 1.0.1-1 in 'noetic/distribution.yaml' [bloom]

[peci1]

Increasing version of package(s) in repository http_relay to 1.0.1-1:

• upstream repository: https://github.com/ctu-vras/http_relay.git
• release repository: https://gitlab.fel.cvut.cz/cras/ros-release/http_relay.git
• distro file: noetic/distribution.yaml
• bloom version: 0.11.2
• previous version for package: null

http_relay

* Noetic compatibility.
* Added sigkill_on_stream_stop so that the relay is only killed if there is a stale request.
* Added sigkill_timeout for automatic restarting of the node even if it hangs on a connection.
* Support NTRIP in Python 3
* Fix Python3 NTRIP relay.
* Initial commit.
* Contributors: Martin Pecka

[clalancette]

See my comments in #37716 (review) about this package.

[clalancette]

Reproducing the conversation from #37716

clalancette

This is another name that I think is pretty generic, and I'm actually not sure at all why it needs to be a ROS package.
That is, it really just reverse proxies HTTP requests, and the only thing it seems to use ROS for is to get parameters from 
the global parameter server. It seems to me that this functionality could just be satisfied by pip installing a Python package.

That said, I could be missing something about how it operates, so feel free to correct me.

peci1:

I agree this functionality isn't directly ROS-related. But it is robot-related.
A few times we had devices which could not access outer networks directly.

I searched the whole apt and pip, but i haven't found any simple package for http proxying (except for big beefy frameworks).

And as it wasn't the first time we needed to run a http relay on the robots, I felt this would be a good package candidate.

Our latest use-case is directly embedding the proxy into sensor ROS driver.

I think this package follows the naming REP. The name clearly says what it does.
Name collisions don't matter because any other package trying to achieve similar functionality would basically do the same.
And org prefix doesn't make sense because there is nothing org-specific there.

[clalancette]

I searched the whole apt and pip, but i haven't found any simple package for http proxying (except for big beefy frameworks).

That is very surprising, but OK.

And as it wasn't the first time we needed to run a http relay on the robots, I felt this would be a good package candidate.

I think my big issue with adding this is that we aren't aiming to be a generic Linux distribution like Debian or Ubuntu. ROS distributions are more focused on robotics than that. While I agree that doing http proxying is sometimes useful and necessary in robotics, there is nothing robot-specific about it. There should be packages from the wider ecosystem that can do this for you. If something like this doesn't exist in PyPI.org, for instance, then I would suggest that this would be a contribution there.

[sloretz]

If something like this doesn't exist in PyPI.org, for instance, then I would suggest that this would be a contribution there.

If they later want to release a package that depends on this code (say a *_bringup package containing launch files to launch everything on a robot), then it will be required to be apt-installable.

[peci1]

If they later want to release a package that depends on this code (say a *_bringup package containing launch files to launch everything on a robot), then it will be required to be apt-installable.

Exactly. I agree it would make sense to try to push such a package into Ubuntu repos, but that is a much more far-fetched way then getting it via ROS buildfarm. So this ROS package is a kind of shortcut. But there's no way of getting the package to Ubuntu repos for Focal, so the ROS way is actually the only possible currently.

[emersonknapp]

I agree it would make sense to try to push such a package into Ubuntu repos, but that is a much more far-fetched way then getting it via ROS buildfarm

I might suggest reading through https://wiki.debian.org/DebianMentorsFaq#Packaging - especially "How do I add a new package to the archive?". I don't think the process is so very long, and this would expand your reach beyond ROS distributions.

this ROS package is a kind of shortcut

The ROS buildfarm is not "free" in the sense of requiring no resources to maintain - so I'd personally prefer if we avoided using it as a shortcut to properly releasing generically useful software. Anybody who wants the package may check it out as source, so they are not blocked from using it - and PyPI has no barrier to entry at all.

[nuclearsandwich]

I am usually the first person to encourage pushing dependencies further upstream and maintaining a tight focus in the ROS distributions on packages that either rely on ROS packages or are quite exclusively dependencies of ROS packages with little other utility.

I don't disagree with any of the arguments toward pushing this functionality upstream. There are a couple of points of nuance I want to recognize:

Putting packages in PyPI is of extremely limited utility if your ultimate goal is depending on it in an officially released ROS package

The most immediate value I see for releasing a Python package to PyPI is that PyPI is the closest thing to a global registry for Python module names, thus being present in that registry makes it less likely to collide with another python package when submitting your package for inclusion in Debian/Ubuntu and Fedora/EPEL.

For dependencies which are already on the path to being upstreamed, I think that these can be considered for vendoring in order to start building on them today. If there's a package that's in Debian Unstable or even an Intent-To-Package bug filed for it, and the package itself is not something that is already available on our target platforms but at a different version. I would consider allowing that to be included in a ROS distribution with the expectation that it will eventually be removed in favor of the upstream package.

[sloretz]

Holding for Noetic sync

[130s]

In this thread there's some words with wisdom that I want to see spread across the communities. I'm also afraid this sort of discussion keeps happening (may not be many times though).
If no one cares to, I'm tempted to challenge myself to document it. Where can be a good place for such an info? Under How To Guide? Under Tutorials/Advanced?

[peci1]

For dependencies which are already on the path to being upstreamed, I think that these can be considered for vendoring in order to start building on them today. If there's a package that's in Debian Unstable or even an Intent-To-Package bug filed for it, and the package itself is not something that is already available on our target platforms but at a different version. I would consider allowing that to be included in a ROS distribution with the expectation that it will eventually be removed in favor of the upstream package.

Thanks, I decided to go that way. I created pip package https://pypi.org/project/http-relay/ and now I'll try to make the ITP. I'll ping here when that's done to decide what to do with this proposed ROS package.

[peci1]

The ITP is finished! https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050035 . I uploaded the built ubuntu packages to https://launchpad.net/~peci1/+archive/ubuntu/http-relay . Do you know of a Debian sponsor who might be interested?

[mjcarroll]

@peci1, @j-rivero has offered to help connect you with a Debian sponsor.

[peci1]

Great, thanks! Should I contact him or will he contact me?

[sloretz]

Holding for Noetic sync

[j-rivero]

Great, thanks! Should I contact him or will he contact me?

Could you please mail me (jrivero@osrf..) with the repository where you have the debian/ metadata? Thanks.

[sloretz]

Holding for Noetic sync

[github-actions]

This PR hasn't been activity in 14 days. If you are still are interested in getting it merged please provide an update. Otherwise it will likely be closed by a rosdistro maintainer following our contributing policy. It's been labeled "stale" for visibility to the maintainers. If this label isn't appropriate, you can ask a maintainer to remove the label and add the 'persistent' label.

[sloretz]

Holding for Noetic sync

[github-actions]

This PR hasn't been activity in 14 days. If you are still are interested in getting it merged please provide an update. Otherwise it will likely be closed by a rosdistro maintainer following our contributing policy. It's been labeled "stale" for visibility to the maintainers. If this label isn't appropriate, you can ask a maintainer to remove the label and add the 'persistent' label.

[mjcarroll]

@j-rivero I think @peci1 is still waiting on you here

[sloretz]

Holding for Noetic sync

[j-rivero]

@j-rivero I think @peci1 is still waiting on you here

Ouch sorry got distracted with some other things, I'll mail @peci1. The process of sponsoring, reveiew, new queue and wait for the FTP-masters can take some weeks, not sure if this affect to this issue or you want to leave it open until we complete the process.

[sloretz]

Holding for Noetic sync

[mjcarroll]

@j-rivero and @peci1 any updates here?

[j-rivero]

@j-rivero and @peci1 any updates here?

Code is ready and I've asked for sponsoring in Debian. Let you know here when we are ready.

[peci1]

Thanks to @j-rivero the package has just been accepted to Debian unstable: https://packages.debian.org/sid/python3-http-relay .

It got already pulled to noble-proposed: https://launchpad.net/ubuntu/+source/http-relay/2.1.3-1 .

Now holding my breath whether it will make it into noble in time :) When that happens, I think we can move on with this PR?

[peci1]

So, the package has landed in Noble! https://packages.ubuntu.com/noble/python3-http-relay .

I've tried to update this PR according to the results of the previous discussion. However, I'm not sure I did it right and technically correct:

• I've created a rosdep key pointing to APT python3-http-relay for noble+ and new debians, using pip for other systems, and having null for noetic and buster.
• I've renamed the temporary ROS package to python3-http-relay and released it for Noetic (thus focal, maybe buster?).
  • The ROS package is now created by merging pip branch into master of https://github.com/ctu-vras/http_relay . master branch adds package.xml, CMakeLists.txt, removes pyproject.toml and changes setup.py.
• The plan (in another PR) is to release it for all non-EOL'd ROS 2 distros and substitute the pip keys with the released version.

I'm not sure whether rosdep is clever enough to look for the key in rosdistro when it finds null in rosdep system package list. Tests are green, but I haven't actually tried it on Noetic. I can't figure out how to test rosdistro index changes locally without setting up whole buildfarm (I guess I'd need to regenerate the cache .gz files manually?)

[peci1]

Or would it be better to do the release for older debian-based systems via reprepro?

[sloretz]

Holding for Noetic sync

[mjcarroll]

CC: @nuclearsandwich or @cottsay. Do you have any input on @peci1's current state?

[cottsay]

I did some local testing, and it appears that the rosdep database takes precedence over the ROS distribution index even when the rule is explicitly null, so this won't work as expected for Noetic.

$ rosdep resolve --rosdistro noetic --os ubuntu:focal python3-http-relay

ERROR: [python3-http-relay] defined as "not available" for OS version [focal]

[python3-http-relay] defined as "not available" for OS version [focal]
	rosdep key : python3-http-relay
	OS name    : ubuntu
	OS version : focal
	Data:
_is_ros: true
		arch:
		  pip:
		    packages:
		    - http-relay
		debian:
		  '*':
		  - python3-http-relay
		  bookworm:
		    pip:
		      packages:
		      - http-relay
		  bullseye:
		    pip:
		      packages:
		      - http-relay
		  buster: null
		fedora:
		  pip:
		    packages:
		    - http-relay
		gentoo:
		  pip:
		    packages:
		    - http-relay
		nixos:
		  pip:
		    packages:
		    - http-relay
		osx:
		  pip:
		    packages:
		    - http-relay
		rhel:
		  pip:
		    packages:
		    - http-relay
		ubuntu:
		  '*':
		  - python3-http-relay
		  focal: null
		  jammy:
		    pip:
		      packages:
		      - http-relay

EDIT:
Here's the spot in rosdep where the rules are merged. The comment states that the merge happens at the os_name level, so because there are ANY ubuntu rules in rosdep, the rule supplied by the distribution index are ignored.

[peci1]

Uh-oh... thanks for the testing, @cottsay . So, would it be acceptable to change the behavior of rosdep? Or does anyone see a better approach?

[cottsay]

I haven't looked into this deeply, but I don't see any downside to deepening the "merge" so that the ROS package can satisfy the nulled rule for focal specifically.

I don't think it will be trivial, though. We would need to handle os_code_name wildcards on either side of the merge, as well as explicit null on either side. Certainly new tests are appropriate.

[peci1]

I'll have a look at it next week. Although a new behavior, I think it is desirable to add it because what I try to do with this package is the recommended way of adding packages that will get upstreamed to distros but need to be released in ROS repos for older distros.