allow outbound http to components in same app by default #1710

michelleN · 2023-08-26T04:15:06Z

add ability for one component to make outbound calls
to another component in the same Spin app using special value self in allowed_http_hosts
make requests with relative routes in app once self is set
update example for rust outbound http to demonstrate capability
adds e2e
resolves Option to allow HTTP request to host running the current application #957
resolves outbound-http: Support "relative" URIs to current app #1533

itowlson

This is great - thanks for making it happen!

My comments on the functional side of it are, I think, entirely suggestions or nits, not blockers. I did, though, get a bit lost in the sample - could we change / rationalise some of the naming there please? Thanks!

itowlson · 2023-08-27T20:35:43Z

examples/http-rust-outbound-http/spin.toml

+
+[[component]]
+id = "component-outbound-http"
+source = "component-outbound-http/target/wasm32-wasi/release/http_rust_outbound_http.wasm"


It would be great to make the difference between the outbound-http and component-outbound-http directories clearer. Maybe outbound-http-to-other-component or outbound-http-to-same-app?

I think this might be because of trying to fit the "local component call" sample into the existing sample - if we're doing that then we should be careful not to lose the simplicity of the existing sample.

itowlson · 2023-08-27T20:36:47Z

examples/http-rust-outbound-http/spin.toml

+
+[[component]]
+id = "hello"
+source = "http-rust/target/wasm32-wasi/release/http_rust.wasm"


The http-rust directory name isn't very helpful, and doesn't obviously relate to the hello component when reading the code that makes the outbound request.

itowlson · 2023-08-27T20:40:31Z

examples/http-rust-outbound-http/component-outbound-http/Cargo.toml

@@ -0,0 +1,19 @@
+[package]
+name = "http-rust-outbound-http"


Another Cargo package already has this name - this has caused us problems in the past because when a user refers to the Rust SDK, Cargo scans the entire repo and complains about duplicate names.

itowlson · 2023-08-27T21:30:29Z

examples/http-rust-outbound-http/component-outbound-http/src/lib.rs

+
+    let parsed = Url::parse(spin_full_url)?;
+
+    let beforepath = Url::parse(&parsed[..Position::BeforePath]).unwrap();


I did not know about Position! Thank you!

itowlson · 2023-08-27T21:36:24Z

examples/http-rust-outbound-http/spin.toml

+id = "component-outbound-http"
+source = "component-outbound-http/target/wasm32-wasi/release/http_rust_outbound_http.wasm"
+[component.trigger]
+route = "/outbound-hello"


Maybe align the route to the component ID? The hello and outbound-hello components feel a bit too similar to me.

itowlson · 2023-08-27T21:59:56Z

examples/http-rust-outbound-http/spin.toml

+
+[[component]]
+id = "component-outbound-http"
+source = "component-outbound-http/target/wasm32-wasi/release/http_rust_outbound_http.wasm"


Suggest adding a comment highlighting that this component doesn't need an allowed_http_hosts because it only sends requests to other components within the same app, and those are always allowed.

itowlson · 2023-08-27T22:03:34Z

crates/outbound-http/src/lib.rs

        Ok(self.allowed_hosts.allow(&url))
    }
+
+    /// Check if the outbound request host and port are the same as origin
+    fn is_same_origin(&self, url: &Url) -> bool {


Maybe

Suggested change

fn is_same_origin(&self, url: &Url) -> bool {

fn is_same_as_origin(&self, url: &Url) -> bool {

? "Same origin" implies to me whether two things have the same origin, whereas you are interested in whether the URL (the destination) is the same as the origin.

itowlson · 2023-08-27T22:08:33Z

crates/outbound-http/src/lib.rs

@@ -31,8 +33,19 @@ impl OutboundHttp {
    /// If `None` is passed, the guest module is not allowed to send the request.
    fn is_allowed(&self, url: &str) -> Result<bool, HttpError> {
        let url = Url::parse(url).map_err(|_| HttpError::InvalidUrl)?;
+
+        if self.is_same_origin(&url) {


There is a possible sneaky gap here, if two applications are hosted on the same host but at different base URLs. I don't think that's possible with any current Spin implementation, so not a blocker (mentioning it mostly because this was the justification for have base in the first place).

itowlson · 2023-08-27T22:15:28Z

crates/outbound-http/src/lib.rs

@@ -21,6 +21,8 @@ pub const ALLOWED_HTTP_HOSTS_KEY: MetadataKey<Vec<String>> = MetadataKey::new("a
 pub struct OutboundHttp {
    /// List of hosts guest modules are allowed to make requests to.
    pub allowed_hosts: AllowedHttpHosts,
+    /// origin is the host of the component that created this outbound HTTP request.


This struct isn't associated with a specific HTTP request, so we could make this more explicit:

Suggested change

/// origin is the host of the component that created this outbound HTTP request.

/// During an incoming HTTP request, origin is set to the host of that incoming HTTP request.

/// This is used to automatically allow outbound requests to the same host.

that's really great wording thank you

itowlson · 2023-08-28T02:58:52Z

@karthik2804 noted that Cloud might need a bit of work to support this behaviour. It would be good if that could happen before this goes into a Spin release, so as to avoid any "works locally but not in cloud" issues.

karthik2804 · 2023-08-28T03:26:57Z

This feature looks great! I have a question about how this works. From looking at the PR, component-to-component HTTP requests for the same app are enabled by default. This is a deviation from what Spin usually does as in that to allow something, some explicit configuration is needed. I am wondering if that is the desired behavior we would want?

From the linked issue I see there is a proposed option for a self configuration option that would allow for an explicit configuration (ref: #957 (comment)).

I wanted to bring this up and see what people thought.

mikkelhegn · 2023-08-30T09:04:35Z

examples/http-rust-outbound-http/outbound-http-to-same-app/src/lib.rs

+    Ok(res)
+}
+
+fn base_url(req: Request) -> Result<Url> {


Is it an option to provide a function like this in the SDKs, so users don't have to implement this in every component?

Alternatively we could reserve some fake host name to represent "self", e.g. https://self.local.

https://my.ego :-) Demo time "So here I call my ego" 👍🏻

lann · 2023-08-30T12:41:05Z

I do think we should require e.g. allowed_http_hosts = ["self"]. Part of the component model security story is that you don't have to completely trust components in your own app, and while making a request to your own app isn't "as bad" as e.g. database access it still breaks that story.

lann · 2023-08-30T19:05:44Z

crates/outbound-http/src/lib.rs

+            let mut url = Url::parse(&req.uri).map_err(|_| HttpError::InvalidUrl)?;
+
+            if url.host_str() == Some("self.local") {
+                url.set_host(self.origin.host())


Should we update the scheme here too? Thinking about testing on http://localhost and then switching to https://my.spin-self-hosting.com.

yea right now the url here would have the same scheme specified in request made in the app so you'd have to update http://self.local to https://self.local before deploying and that seems like a paper cut. Let me try updating the scheme like you suggested.

Updated the implementation so you don't need to use self.local in the uri when making a request and instead you can use relative paths and the correct origin info will be added to the uri as you originally suggested in #1533

self.local is now self because... one word is better than two!

We may still want to allow http://self.local/abc (in addition to just /abc) for code that expects an absolute URL, like (I think?) the Go SDK.

I'm not seeing where the Go SDK requires an absolute URL. Could you point me in right direction?

I was thinking that since we replace the default http client (here) it might break something in that path if you pass a relative URL, but I haven't tested.

lann · 2023-08-31T16:58:50Z

examples/http-tinygo-outbound-http/outbound-http-to-same-app/main.go

+
+func init() {
+	spinhttp.Handle(func(w http.ResponseWriter, r *http.Request) {
+		resp, err := spinhttp.Get("/hello")


If convenient, could you try this as just http.Get (like here)?

that worked as well

itowlson

Looks fine - I left some style suggestions but nothing blocking. Thanks!

itowlson · 2023-09-01T02:41:13Z

crates/outbound-http/src/lib.rs

-    fn is_allowed(&self, url: &str) -> Result<bool, HttpError> {
+    fn is_allowed(&mut self, url: &str) -> Result<bool, HttpError> {
+        if url.starts_with('/') {
+            if self.allowed_hosts.includes(AllowedHttpHost::host("self")) {


Rather than requiring callers to know how AllowedHttpHosts represents the self-link permission (and requiring them to construct an AllowedHttpHost just in order to ask), consider putting this into a AllowedHttpHosts::allow_relative_url (or some better name) method.

Suggested change

if self.allowed_hosts.includes(AllowedHttpHost::host("self")) {

return self.allowed_hosts.allow_relative_url();

(and remove the includes method)

itowlson · 2023-09-01T02:45:31Z

crates/outbound-http/src/lib.rs

@@ -49,7 +61,14 @@ impl outbound_http::Host for OutboundHttp {
            }

            let method = method_from(req.method);
-            let url = Url::parse(&req.uri).map_err(|_| HttpError::InvalidUrl)?;
+
+            let req_url: Url = if req.uri.starts_with('/') {


To avoid having the parse and error map on both branches, I wonder if this could be reorganised as something like

Suggested change

let req_url: Url = if req.uri.starts_with('/') {

let abs_url = if req.uri.starts_with('/') {

format!(...)

} else {

req.uri.to_string()

};

let req_url = Url::parse(&abs_uri).map_err(...)?;

itowlson · 2023-09-01T02:47:30Z

crates/trigger-http/src/spin.rs

+                if let Some(outbound_http_handle) = engine
+                .engine
+                .find_host_component_handle::<std::sync::Arc<outbound_http::OutboundHttpComponent>>(
+                )


Oh rustfmt no

(no action for you, I am just sad at the tool)

I don't think rustfmt produced this, I think it gave up on this code for some reason...

Looks like it gave up because the function call was too long. If you add uses for Arc and OutboundHttpComponent it will happily reformat this.

good tip! Moving the logic to its own function helped but I added the uses anyway for readability.

itowlson · 2023-09-01T02:48:46Z

crates/trigger-http/src/spin.rs

+
+                outbound_http_data.origin = format!("{}://{}", scheme, authority);
+            }
+            }


Two consecutive close braces at the same indent feels weird to me but I suppose this is another rustfmt special...

itowlson · 2023-09-01T02:53:18Z

crates/trigger-http/src/spin.rs

        let EitherInstance::Component(instance) = instance else {
            unreachable!()
        };

+        if let Some(authority) = req.uri().authority() {


This has grown to a big enough block that it rather breaks up the flow of the function - consider pulling it out into a set_http_origin_from_request(&mut store, &req) (or otherwise meaningfully named) function.

itowlson · 2023-09-01T02:54:25Z

examples/http-tinygo-outbound-http/tinygo-hello/go.sum

@@ -0,0 +1,2 @@
+github.com/julienschmidt/httprouter v1.3.0 h1:U0609e9tgbseu3rBINet9P48AI/D3oJs4dN7jwJOQ1U=


Is it correct that we are adding these in this PR?

good catch. removed!

turns out we did need that go.sum file (integration tests failed and other examples include it) so I added this back in.

+ use special value `self` in allowed_http_hosts to allow components in the same spin app to make outbound http requests to each other + make requests with relative routes in app once self is set + update example for rust outbound http to demonstrate capability + adds e2e + resolves fermyon#957 + resolves fermyon#1533 Signed-off-by: Michelle Dhanani <michelle@fermyon.com>

Signed-off-by: Michelle Dhanani <michelle@fermyon.com>

itowlson requested changes Aug 27, 2023

View reviewed changes

michelleN force-pushed the outboundhttp branch 3 times, most recently from 32f8662 to 46fd5bc Compare August 30, 2023 00:10

mikkelhegn reviewed Aug 30, 2023

View reviewed changes

itowlson approved these changes Aug 30, 2023

View reviewed changes

lann reviewed Aug 30, 2023

View reviewed changes

michelleN force-pushed the outboundhttp branch 10 times, most recently from 919c1cc to 6475e5c Compare August 31, 2023 16:56

lann reviewed Aug 31, 2023

View reviewed changes

michelleN force-pushed the outboundhttp branch from 6475e5c to 37e31d0 Compare August 31, 2023 18:00

michelleN mentioned this pull request Aug 31, 2023

add validation for self in allowed_http_hosts fermyon/cloud-plugin#74

Merged

michelleN force-pushed the outboundhttp branch from 37e31d0 to 6cb693a Compare August 31, 2023 19:29

michelleN mentioned this pull request Aug 31, 2023

update outbound http docs when Spin 1.5 ships to include new self feature fermyon/developer#846

Closed

michelleN force-pushed the outboundhttp branch 3 times, most recently from 49f405c to 8ba038b Compare September 1, 2023 00:29

itowlson approved these changes Sep 1, 2023

View reviewed changes

melissaklein24 added this to the 1.5.0 milestone Sep 1, 2023

michelleN force-pushed the outboundhttp branch from 8ba038b to 88f66a8 Compare September 1, 2023 15:10

michelleN force-pushed the outboundhttp branch from 88f66a8 to d31f03b Compare September 1, 2023 15:18

add outbound http to same app tinygo example

c233253

Signed-off-by: Michelle Dhanani <michelle@fermyon.com>

michelleN force-pushed the outboundhttp branch from d31f03b to c233253 Compare September 1, 2023 19:54

michelleN merged commit 089e135 into fermyon:main Sep 5, 2023
9 checks passed

tpmccallum mentioned this pull request Sep 13, 2023

In preparation for future Spin 1.5 CLI fermyon/developer#765

Merged

9 tasks


		let parsed = Url::parse(spin_full_url)?;

		let beforepath = Url::parse(&parsed[..Position::BeforePath]).unwrap();

	fn is_same_origin(&self, url: &Url) -> bool {
	fn is_same_as_origin(&self, url: &Url) -> bool {

	/// origin is the host of the component that created this outbound HTTP request.
	/// During an incoming HTTP request, origin is set to the host of that incoming HTTP request.
	/// This is used to automatically allow outbound requests to the same host.

	if self.allowed_hosts.includes(AllowedHttpHost::host("self")) {
	return self.allowed_hosts.allow_relative_url();

-            let req_url: Url = if req.uri.starts_with('/') {
+            let abs_url = if req.uri.starts_with('/') {
+                format!(...)
+            } else {
+                req.uri.to_string()
+            };
+            let req_url = Url::parse(&abs_uri).map_err(...)?;

		@@ -0,0 +1,2 @@
		github.com/julienschmidt/httprouter v1.3.0 h1:U0609e9tgbseu3rBINet9P48AI/D3oJs4dN7jwJOQ1U=

allow outbound http to components in same app by default #1710

allow outbound http to components in same app by default #1710

Conversation

michelleN commented Aug 26, 2023 • edited Loading

itowlson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itowlson commented Aug 28, 2023

karthik2804 commented Aug 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lann commented Aug 30, 2023

lann Aug 30, 2023 • edited Loading

Choose a reason for hiding this comment

michelleN Aug 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itowlson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michelleN Sep 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michelleN Sep 1, 2023 • edited Loading

Choose a reason for hiding this comment

michelleN commented Aug 26, 2023 •

edited

Loading

lann Aug 30, 2023 •

edited

Loading

michelleN Aug 30, 2023 •

edited

Loading

michelleN Sep 1, 2023 •

edited

Loading

michelleN Sep 1, 2023 •

edited

Loading