ERL-1040: Allow any term for httpc profile

[OTP-Maintainer]

Original reporter: vances
Affected version: Not Specified
Component: inets
Migrated from: https://bugs.erlang.org/browse/ERL-1040

---

Multiple instances of {{httpc_manager}} may be started, each of which has it's own set of options ({{profile}}) to be used in requests. These may be started and stopped dynamically during run time.

Currently the type for profile is {{atom()}}. This is a problem if an application will dynamically start and stop {{httpc}} with a unique {{profile}} for each as the atom table isn't garbage collected.

[OTP-Maintainer]

ingela said:

The intention was never that you should start a new profiles per request. 
A profile could be though of as incognito window in a browser and your application could have its own httpc_profile to make sure it did not interfere with other http_clients
run on the same node in a different application.  The atom is used to register a name for the client manager process.  Although it could be changed this is not something of a big interest to us. We are more interested in making a light "httpc-request"  more like a wget. The effect can already be accomplished by adding the HTTP header
connection:close to your request, but it could be made more efficient. 

[OTP-Maintainer]

vances said:

My use case is for subscribers to a notification service associated with other REST APIs (httpd). Clients POST a JSON object containing a subscription request to a "hub" collection resource which creates a subscription. Subscribed events are POST'd (using httpc) to the client as long as the subscription remains in the hub. These are "long lived" but transient.

I became concerned by a [comment|https://github.com/phoenixframework/phoenix/issues/1478#issuecomment-173195841] by Thomas Arts that suggested the {{socket_closed_remotely}} errors we were experiencing may be related to multiple processes sharing a profile. I have since tested that and found no impact. That issue is simply the race condition described in ERL\-473.

Because some options, notably keep_alive_timeout  related to the above, may only be set on the profile it seems that separate profiles for different clients is the correct solution. That said as long as we don't think they interfere with each other it's not a large issue.

[OTP-Maintainer]

ingela said:

keep_alive_timeout needs to be associated to a specific  TCP connection so that is why it is tied to a profile. httpc automates pipelining or connection reuse (you have to choose one of the two modes). However if you do want a new connections for each request  you can make sure that you get  that by setting max_keep_alive_length/max_pipeline_length to 1. Would that work for your use case?  

[OTP-Maintainer]

vances said:

bq. httpc automates pipelining or connection reuse (you have to choose one of the two modes).
I understand\(?) that httpc will distribute requests over several connections, with or without pipelining. It would be nice if the race condition described in ERL\-473 would get handled by that magic instead of returning {{\{error, socket_closed_remotely}}} however that problem is easily avoided by setting {{keep_alive_timeout}} below the threshold which causes the server to close the connection.

For my current use case we have dynamically added client processes, each having a different destination server. These clients are streaming (POST) requests to the servers and may live for minutes, days, weeks. If I need to configure httpc options for the clients' server (e.g. {{proxy}}, {{max_sessions}}, {{keep_alive_timeout}}, etc.) I would need separate profiles for each client. For this reason the correct design would seem to be to create a profile for each client and to do so the current issue (ERL\-1040) becomes a (soft) requirement. +However I think I can live with the limitation of having only global options at the moment.+

I would just like to be reassured that multiple clients, each with their own concurrent sessions to the same of different servers, would not interfere with one another. It has been suggested that this may cause a problem.

[OTP-Maintainer]

ingela said:

[~vances] Humm ... well yes ERL-473 ought to be prioritized to be fixed, I think it just never made it due to lot of other more prioritized issues, but sometimes small not so serious things should be prioritized, if only due to they being around for such a long time.

As for allowing other than atom names for profiles we think it could be a possible addition if someone wants to make a PR but I do not see it happening in the near future if should be driven by OTP.