feature request: define domain-based policies for web socket interception

[GlenDC]

Hi, first of all I would like to ask if it is possible I contribute the code for this feature request myself, with guidance from you. There are other feature requests I'll have soon and contributions I would like to make. For now I know g3 more or less as a user and also having studies the code, but this will be my first code contributions to your code so guidance will be appreciated. If possible of course.

Currently there is already websocket_inspect_policy that you can define for an auditor. Which applies to all traffic. This allows you for example to intercept or block any ws(s) traffic. I would like to request however to make it also possible to do so for specific domains.

E.g. I noticed there is no ws_interception config yet. So my first thought was that it might be an idea to add such a config and start using that so one can add an allow list. However for all other http traffic we currently make use already of route_upstream to block traffic, rather than have it in the http interception configs. So I imagine you might rather want to have supported in that fashion? I'll already play around with some ideas myself locally in a quick and dirty way, just to get it into my fingers.

I would appreciate your feedback, guidance and thoughts on this matter. Once we are aligned and I have your seal of approval on the approach I can deliver you a first PR, if that's alright for you.

[GlenDC]

I have made a first very dirty shortsighted demo PoC here:

diff --git a/g3proxy/src/inspect/websocket/h1.rs b/g3proxy/src/inspect/websocket/h1.rs
index 37eb7f7f..8c40fc31 100644
--- a/g3proxy/src/inspect/websocket/h1.rs
+++ b/g3proxy/src/inspect/websocket/h1.rs
@@ -91,7 +91,19 @@ impl<SC: ServerConfig> H1WebsocketInterceptObject<SC> {
 
     pub(crate) async fn intercept(mut self) -> ServerTaskResult<()> {
         let r = match self.ctx.websocket_inspect_policy() {
-            ProtocolInspectPolicy::Intercept => self.do_intercept().await,
+            ProtocolInspectPolicy::Intercept => {
+                // quick PoC test in case zh-jq wants to have it as a ws_interception config,
+                // in such case we could block the traffic here in a more worked out manner
+                if self.upstream.host_eq(
+                    &UpstreamAddr::from_host_str_and_port("websocketstest.com", 443).unwrap(),
+                ) {
+                    log::info!("h1: block WS based on upstream address: {}", self.upstream);
+                    self.do_block().await
+                } else {
+                    log::info!("h1: allow WS based on upstream address: {}", self.upstream);
+                    self.do_intercept().await
+                }
+            }
             #[cfg(feature = "quic")]
             ProtocolInspectPolicy::Detour => self.do_detour().await,
             ProtocolInspectPolicy::Bypass => self.do_bypass().await,
diff --git a/g3proxy/src/inspect/websocket/h2.rs b/g3proxy/src/inspect/websocket/h2.rs
index 7694653d..f3eb817a 100644
--- a/g3proxy/src/inspect/websocket/h2.rs
+++ b/g3proxy/src/inspect/websocket/h2.rs
@@ -75,7 +75,19 @@ impl<SC: ServerConfig> H2WebsocketInterceptObject<SC> {
         ups_w: SendStream<Bytes>,
     ) {
         let r = match self.ctx.websocket_inspect_policy() {
-            ProtocolInspectPolicy::Intercept => self.do_intercept(clt_r, clt_w, ups_r, ups_w).await,
+            ProtocolInspectPolicy::Intercept => {
+                // quick PoC test in case zh-jq wants to have it as a ws_interception config,
+                // in such case we could block the traffic here in a more worked out manner
+                if self.upstream.host_eq(
+                    &UpstreamAddr::from_host_str_and_port("websocketstest.com", 443).unwrap(),
+                ) {
+                    log::info!("h2: block WS based on upstream address: {}", self.upstream);
+                    self.do_block(clt_w, ups_w).await
+                } else {
+                    log::info!("h2: allow WS based on upstream address: {}", self.upstream);
+                    self.do_intercept(clt_r, clt_w, ups_r, ups_w).await
+                }
+            }
             #[cfg(feature = "quic")]
             ProtocolInspectPolicy::Detour => self.do_detour(clt_r, clt_w, ups_r, ups_w).await,
             ProtocolInspectPolicy::Bypass => self.do_bypass(clt_r, clt_w, ups_r, ups_w).await,

That is the locations I am aware of so far in case we want to go towards the approach for ws_interception Config (newly introduced). But as I am new to this code base I might be forgetting some locations. Do let me know. I am open for anything.

If you want to go the route_upstream approach (dunno if applicable here) or something similar, I will need even more guidance / hints.

With that approach you should be able to go to https://websocketstest.com/ and see that it no longer works (the test) while other web sockets do still work.

[zh-jq]

Hi, first of all I would like to ask if it is possible I contribute the code for this feature request myself, with guidance from you. There are other feature requests I'll have soon and contributions I would like to make. For now I know g3 more or less as a user and also having studies the code, but this will be my first code contributions to your code so guidance will be appreciated. If possible of course.

Yes, external contributions will always be welcomed.

I have to cut the new v1.10 LTS branch this month, so all the new feaures will only be available in the next 1.11 development vesion.

Currently there is already websocket_inspect_policy that you can define for an auditor. Which applies to all traffic. This allows you for example to intercept or block any ws(s) traffic. I would like to request however to make it also possible to do so for specific domains.

That seems like a feature useful for all protocols.

You can extend the inspect policy enum to be an ACL like struct data type and add domain based rules.

E.g. I noticed there is no ws_interception config yet. So my first thought was that it might be an idea to add such a config and start using that so one can add an allow list.

The *_interception config options are used to control protocol interception params insted of policy. We will add ws_interception config once we want to inspect into websocket sub protocols directly inside g3roxy.

However for all other http traffic we currently make use already of route_upstream to block traffic, rather than have it in the http interception configs. So I imagine you might rather want to have supported in that fashion?

That's just a cheap way to do it but not the best. The config structure may be too large if there are many domain specific rules.

I'll already play around with some ideas myself locally in a quick and dirty way, just to get it into my fingers.

I would appreciate your feedback, guidance and thoughts on this matter. Once we are aligned and I have your seal of approval on the approach I can deliver you a first PR, if that's alright for you.

Thanks. That will work for me.

[GlenDC]

so all the new feaures will only be available in the next 1.11 development vesion

Works for me.

You can extend the inspect policy enum to be an ACL like struct data type and add domain based rules.

Even better. Before I start with the real work of enabling that and hooking it up to the config and whatnot,
do you mean to have the ProtocolInspectPolicy be this:

#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub enum ProtocolInspectPolicy {
    #[default]
    Intercept(Option<g3_types::acl_set::AclDstHostRuleSet>),
    #[cfg(feature = "quic")]
    Detour,
    Bypass,
    Block,
}

If not let me know how you see that policy struct. I can take it from there.

[zh-jq]

Before I start with the real work of enabling that and hooking it up to the config and whatnot, do you mean to have the ProtocolInspectPolicy be this:

#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub enum ProtocolInspectPolicy {
    #[default]
    Intercept(Option<g3_types::acl_set::AclDstHostRuleSet>),
    #[cfg(feature = "quic")]
    Detour,
    Bypass,
    Block,
}

If not let me know how you see that policy struct. I can take it from there.

It will be like AclDstHostRuleSet but with the enum AclAction value replaced as enum InspectAction, which should be the original enum ProtocolInspectPolicy.

[GlenDC]

You mean to replace the current enum ProtocolInspectPolicy with:

struct ProtocolInspectPolicy {
    exact: Option<InspectExactHostRule>,
    child: Option<InspectChildDomainRule>,
    regex: Option<InspectRegexSetRule>,
    subnet: Option<InspectNetworkRule>,
    missed_action: InspectAction,
}

struct InspectAction {
   kind: InspectActionKind,
   log: bool,
}

enum InspectActionKind {
    Intercept,
    #[cfg(feature = "quic")]
    Detour,
    Bypass,
    Block,
}

struct InspectExactHostRule { ... }

struct InspectChildDomainRule { ... }

struct InspectRegexSetRule { ... } 

struct InspectNetworkRule { ... }

I don't mind doing that work. But seems like a lot of duplicated code from your acl module.

Also where do you want this inspect (type) code to live?

[zh-jq]

You mean to replace the current enum ProtocolInspectPolicy with:

struct ProtocolInspectPolicy {
    exact: Option<InspectExactHostRule>,
    child: Option<InspectChildDomainRule>,
    regex: Option<InspectRegexSetRule>,
    subnet: Option<InspectNetworkRule>,
    missed_action: InspectAction,
}

struct InspectAction {
   kind: InspectActionKind,
   log: bool,
}

enum InspectActionKind {
    Intercept,
    #[cfg(feature = "quic")]
    Detour,
    Bypass,
    Block,
}

struct InspectExactHostRule { ... }

struct InspectChildDomainRule { ... }

struct InspectRegexSetRule { ... } 

struct InspectNetworkRule { ... }

I don't mind doing that work. But seems like a lot of duplicated code from your acl module.

Yes. Maybe it could be AclDstHostRuleSet<InspectionAction>? I'm not sure whether it's possible/suitable.

Also where do you want this inspect (type) code to live?

If we can't reuse the ACL data types, it can go to the same module as the current ProtocolInspectPolicy.

[GlenDC]

Replacing ProtocolInspectPolicy with AclDstHostRuleSet<InspectionAction> you mean? Would break existing configs though, unless you also would want me to still support a direct InspectAction string as well when parsing the config.

And thus that would mean to redefine AclDstHostRuleSet as

pub struct AclDstHostRuleSet<Action> {
    exact: Option<AclExactHostRule<Action>>,
    child: Option<AclChildDomainRule<Action>>,
    regex: Option<AclRegexSetRule<Action>>,
    subnet: Option<AclNetworkRule<Action>>,
    missed_action: Action,
}

I am ok with any approach, so also this. But does seem like a pretty invasive change.
Do tell me what you see best for me as an approach here. The details we can refine in the PR, but would be good to at least already decide if this is the direction you want (direction 1), or if you rather prefer to for now keep the entire inspect policy and its rules in to live independent from Acl in g3_dpi::config (direction 2).

[zh-jq]

Direction 1 the best, if possible. The breaking change in config maybe avoided by adding some workround code.

[GlenDC]

Ok I’ll get to work, I know enough for now, thx!

[GlenDC]

#344 is ready for review @zh-jq . It's my first real code contribution to your codebase, so I would not be surprised that you have a lot of requests and feedback.

I tested it on a local g3-based proxy and it seems to work as expected for the web sockets. Where I block some, but intercept all others (as that is the default I used in my test).

Given this PR is in line with the direction we agreed upon I hope we can find an alignment quickly.

[GlenDC]

@zh-jq I can confirm that web the inspect policy now can contain ACL Rules, which is pretty neat. I consider that part finished until you deemed otherwise in a review whenever you have time for it.

However... if we zoom in on web sockets, I do see an additional issue. By the time the web socket policy triggers, and thus closes the connection, it has already been upgraded and established. This can however give odd behavior in a proxy client as it would essentially look like an EOF after ws upgrade was established and active.

As such I think that besides the feature I implemented in my PR, which I anyway can also use for other stuff given the policy is applicable to more then just web sockets, I also need to be able to specifically filter on http headers and block the traffic like that.

Correct me if I'm wrong @zh-jq but I believe the proper solution for web sockets in specific would be to send a 400 BAD Request response when a web socket http request comes in. E.g. by filtering on Upgrade: websocket. Which should be basically as easy as saying "if domain = X and contains(Upgrade: websocket)" than send this response. Can be a pretty generic feature.

I could however not found anything of this kind of support in g3, or am I wrong?
Whatever the case may be, while the current block policy does block any actual web socket traffic from passing on, it does so however at the cost of confusing potential web socket clients that might take it as a temporary connection error and just retry web sockets over and over again.

Curious to hear your opinion on this matter.

[zh-jq]

@GlenDC Yes you are right, websocket should be blocked gracefully. The check of inspect policy can be added when received H1 upgrade or H2 extended-connect-upgrade request. This just like the inspect policy check when parsing ALPN in TLS interception code.

[GlenDC]

But I am correct in thinking this will need an additional PR / patch, right? This is not something you can do today? If so, can you guide me on how I would do that?

[zh-jq]

But I am correct in thinking this will need an additional PR / patch, right?

Yes this should be considered as a BUG FIX.

This is not something you can do today? If so, can you guide me on how I would do that?

I can write a PR tomorrow.

If you want to fix it today, here are the code position that need to be updated:

H2:

g3/g3proxy/src/inspect/http/v2/connect/extended.rs

Line 142 in 1f215cc

async fn run_extended_websocket(

H1:

g3/g3proxy/src/inspect/http/v1/upgrade/mod.rs

Line 152 in 1f215cc

pub(super) async fn forward_icap<CW, UR, UW>(

g3/g3proxy/src/inspect/http/v1/upgrade/mod.rs

Line 177 in 1f215cc

pub(super) async fn forward_original<CW, UR, UW>(