Collection of interesting configs

[ydylla]

I have collected some of the configs that I have encountered during developing #192. These are mostly about testing the matching and routing logic of layer4. Maybe they can some day be used to setup an automated testing suite. For now this is just intended as reference, so they are easier to find and maybe help other developers.

prefetch

This tests basic functionality of multiple routes that are executed one after another. The interesting bit is that curl does not send the http2 request in one go. But rather first the http2 preface followed by the rest of the http2 request in a different flush (see #72 (comment)). So usually multiple prefetch calls are needed.

References:

• Non blocking matchers & matching timeout #192

prefetch.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "tls": {
      "certificates": {
        "automate": ["localhost"]
      },
      "automation": {
        "policies": [{
          "subjects": ["localhost"],
          "issuers": [{
            "module": "internal"
          }]
        }]
      }
    },
    "layer4": {
      "servers": {
        "https": {
          "listen": ["0.0.0.0:10443"],
          "routes": [
            {
              "match": [
                {"proxy_protocol": {}}
              ],
              "handle": [
                {
                  "handler": "proxy_protocol",
                  "allow": ["0.0.0.0/0"]
                }
              ]
            },
            {
              "match": [
                {"tls": {"sni": ["localhost"]}}
              ],
              "handle": [
                {"handler": "tls"}
              ]
            },
            {
              "match": [
                {"http": [{"host": ["localhost"]}]}
              ],
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [{"dial": ["127.0.0.1:10080"]}]
                }
              ]
            }
          ]
        }
      }
    },
    "http": {
      "servers": {
        "backend": {
          "protocols": ["h1","h2","h2c"],
          "listen": ["127.0.0.1:10080"],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello World\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        }
      }
    }
  }
}

Testing commands:

curl -v -k --http2 https://localhost:10443
curl -v -k --http2-prior-knowledge https://localhost:10443
curl -v http://localhost:10443
curl -v -k --haproxy-protocol https://localhost:10443

All should succeed and show "Hello World" in the response.

socks4 before socks5

This tests if matchers that are tried first and require more data than the client has sent will block all other matchers.
This was part of the initial motivation for #192.

References:

• Add socks5 support #68

socks4-before-socks5.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "layer4": {
      "servers": {
        "https": {
          "listen": ["0.0.0.0:1080"],
          "routes": [
            {
              "match": [
                {"socks4": {}}
              ],
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [{"dial": ["10.64.0.1:1080"]}]
                }
              ]
            },
            {
              "match": [
                {"socks5": {}}
              ],
              "handle": [
                {"handler": "socks5"}
              ]
            }
          ]
        }
      }
    }
  }
}

Testing commands:

curl -x socks4://127.0.0.1:1080 https://ipinfo.io
curl -x socks5://127.0.0.1:1080 https://ipinfo.io

The socks4 command will likely fail for you, except you are connected to Mullvad VPN.
But the socks5 one should print your external ip address.
What should not happen is that the socks5 command blocks indefinitely like before #192.

subroute

This tests that the subroute handler works after another matcher was already executed.
This is a reproducer for a bug introduced by #192.

References:

• fix: subroute blocking if all client data was already read #194
• Non blocking matchers & matching timeout #192 (comment)
• Non blocking matchers & matching timeout #192 (comment)

subroute.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "tls": {
      "certificates": {
        "automate": ["localhost"]
      },
      "automation": {
        "policies": [{
          "subjects": ["localhost"],
          "issuers": [{
            "module": "internal"
          }]
        }]
      }
    },
    "layer4": {
      "servers": {
        "https": {
          "listen": ["0.0.0.0:10443"],
          "routes": [
            {
              "match": [
                {"tls": {}}
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "match": [
                        {"tls": {"sni": ["localhost"]}}
                      ],
                      "handle": [
                        {"handler": "tls"},
                        {
                          "handler": "proxy",
                          "upstreams": [{"dial": ["127.0.0.1:10080"]}]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "http": {
      "servers": {
        "backend": {
          "protocols": ["h1","h2","h2c"],
          "listen": ["127.0.0.1:10080"],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello World\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        }
      }
    }
  }
}

Testing commands:

curl -v -k https://localhost:10443

Should contain "Hello World" in the response.

tls in tls

This tests if it is possible to use tls inside tls. Which is for example used by Signal proxies.
Or more generally if the route order in the config is still respected.
This is a reproducer for a bug introduced by #192.

References:

• fix: skip already seen routes after a match #196
• Non blocking matchers & matching timeout #192 (comment)
• Non blocking matchers & matching timeout #192 (comment)

tls-in-tls.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "tls": {
      "certificates": {
        "automate": ["outer", "inner"]
      },
      "automation": {
        "policies": [
          {
            "subjects": ["outer"],
            "issuers": [{
              "module": "internal"
            }]
          },
          {
            "subjects": ["inner"],
            "issuers": [{
              "module": "internal"
            }]
          }
        ]
      }
    },
    "layer4": {
      "servers": {
        "https": {
          "listen": ["0.0.0.0:10443"],
          "routes": [
            {
              "handle": [
                {"handler": "tls"}
              ]
            },
            {
              "match": [
                {"tls": {"sni": ["inner"]}}
              ],
              "handle": [
                {"handler": "tls"},
                {
                  "handler": "proxy",
                  "upstreams": [{"dial": ["127.0.0.1:10080"]}]
                }
              ]
            }
          ]
        }
      }
    },
    "http": {
      "servers": {
        "backend": {
          "listen": ["127.0.0.1:10080"],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello World\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        }
      }
    }
  }
}

Testing Go program

package main

import (
	"crypto/tls"
	"log"
)

func main() {
	outerConn, err := tls.Dial("tcp", "127.0.0.1:10443", &tls.Config{
		ServerName:         "outer",
		InsecureSkipVerify: true,
	})
	if err != nil {
		log.Fatal(err)
	}

	innerConn := tls.Client(outerConn, &tls.Config{
		ServerName:         "inner",
		InsecureSkipVerify: true,
	})

	_, err = innerConn.Write([]byte("GET / HTTP/1.0\r\nHost: localhost\r\n\r\n"))
	if err != nil {
		log.Fatal(err)
	}
	buf := make([]byte, 1024)
	n, err := innerConn.Read(buf)
	if err != nil {
		log.Fatal(err)
	}
	println(string(buf[:n]))
}

Should contain "Hello World" in the output.

endless loop bug

This is a reproducer for a bug that was introduced in #196.
It was triggered by a non terminal handler that did not consume its bytes and would thus be matched again in the next matching loop iteration. Normally a call to prefetch should have happened after the first loop iteration, which blocks the endless loop until timeout.
In this example config a proxy_protocol handler is used that does not accept the proxy protocol send by the test command and thus the matched bytes are not consumed.

References:

• fix: endless matching loop #199

endless-loop-bug.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "layer4": {
      "servers": {
        "loop": {
          "listen": ["0.0.0.0:8080"],
          "routes": [
            {
              "match": [
                {"proxy_protocol": {}}
              ],
              "handle": [
                {
                  "handler": "proxy_protocol",
                  "allow": ["1.1.1.1/32"]
                }
              ]
            }
          ]
        }
      }
    }
  }
}

Testing commands:

curl -v --haproxy-protocol http://localhost:8080

Should fail after the matching_timeout (by default 3s) with a "aborted matching according to timeout" error in the caddy log.
If the bug is triggered a caddy thread will run at 100% percent cpu load indefinitely and no error or connection stats are visible in the caddy log.

listener wrapper fallback

This tests if it is possible to use the layer4 listener wrapper with caddy as fallback. Meaning connections not matched by layer4 are passed back to the caddy http app.
This is a reproducer for a bug that was introduced with #192, that was fixed by #208.

References:

• Non blocking matchers & matching timeout breaking l4 as a listener wrapper #207
• remove unnecessary routes matching and restore old fallback next hand… #208

listener-wrapper-fallback.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "tls": {
      "certificates": {
        "automate": ["localhost"]
      },
      "automation": {
        "policies": [{
          "subjects": ["localhost"],
          "issuers": [{
            "module": "internal"
          }]
        }]
      }
    },
    "http": {
      "servers": {
        "testing": {
          "listen": ["127.0.0.1:8080"],
          "listener_wrappers": [
            {
              "wrapper": "layer4",
              "routes": [
                {
                  "match": [
                    {
                      "tls": {
                        "sni": ["localhost"]
                      }
                    }
                  ],
                  "handle": [
                    {"handler": "tls"},
                    {
                      "handler": "proxy",
                      "upstreams": [{"dial": ["127.0.0.1:10080"]}]
                    }
                  ]
                }
              ]
            }
          ],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello World from testing caddy\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        },
        "backend": {
          "protocols": ["h1","h2","h2c"],
          "listen": ["127.0.0.1:10080"],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello World from backend caddy\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        }
      }
    }
  }
}

Testing commands:

curl -v -k http://localhost:8080
curl -v -k https://localhost:8080

The first (http) request should contain "Hello World from testing caddy" in the response.
The second one (https) request should contain "Hello World from backend caddy" in the response.

listener wrapper matcher with no handler

This tests if it is possible to use a route with a matcher but without any handlers to workaround #207 in most cases.

References:

• Non blocking matchers & matching timeout breaking l4 as a listener wrapper #207
• Non blocking matchers & matching timeout breaking l4 as a listener wrapper #207 (comment)

listener-wrapper-matcher-with-no-handler.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "tls": {
      "certificates": {
        "automate": ["localhost"]
      },
      "automation": {
        "policies": [{
          "subjects": ["localhost"],
          "issuers": [{
            "module": "internal"
          }]
        }]
      }
    },
    "http": {
      "servers": {
        "testing": {
          "listen": ["127.0.0.1:8080"],
          "listener_wrappers": [
            {
              "wrapper": "layer4",
              "routes": [
                {
                  "match": [
                    {
                      "tls": {
                        "sni": ["localhost"]
                      }
                    }
                  ],
                  "handle": [
                    {"handler": "tls"},
                    {
                      "handler": "proxy",
                      "upstreams": [{"dial": ["127.0.0.1:10080"]}]
                    }
                  ]
                },
                {
                  "match": [
                    {"http": []}
                  ]
                }
              ]
            }
          ],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello World from testing caddy\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        },
        "backend": {
          "protocols": ["h1","h2","h2c"],
          "listen": ["127.0.0.1:10080"],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello World from backend caddy\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        }
      }
    }
  }
}

Testing commands:

curl -v -k http://localhost:8080
curl -v -k https://localhost:8080

The first (http) request should contain "Hello World from testing caddy" in the response.
The second one (https) request should contain "Hello World from backend caddy" in the response.
At the time of writing this works.

matcher without io

This tests if it is possible to use a route with a matcher that does not read from the connection.

matcher-without-io.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "layer4": {
      "servers": {
        "https": {
          "listen": ["0.0.0.0:10443"],
          "routes": [
            {
              "match": [
                {"remote_ip": {"ranges": ["0.0.0.0/0"]}}
              ],
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [{"dial": ["127.0.0.1:10080"]}]
                }
              ]
            }
          ]
        }
      }
    },
    "http": {
      "servers": {
        "backend": {
          "protocols": ["h1","h2","h2c"],
          "listen": ["127.0.0.1:10080"],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello World\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        }
      }
    }
  }
}

Testing commands:

curl -v -4 http://localhost:10443
curl -v -6 http://localhost:10443

The first (ipv4) request should contain "Hello World" in the response.
The second one (ipv6) request should fail with layer4 closing the connection.

server speaks first

This tests if it is possible to use layer4 as tcp proxy with protocols where the server speaks first. For example like SMTP or MariaDB.

References:

• Regression: PROXY protocol line sent only when first data arrives from client #212
• Cannot proxy to Mariadb instance #228

server-speaks-first.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "layer4": {
      "servers": {
        "https": {
          "listen": ["0.0.0.0:10443"],
          "routes": [
            {
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [{"dial": ["127.0.0.1:10080"]}]
                }
              ]
            }
          ]
        }
      }
    }
  }
}

Testing commands:
Simulating a server that sends data after connection establishment.

echo "Hello World" | nc -v -l 127.0.0.1 10080

Connecting to caddy as client

nc -v 127.0.0.1 10443

You should see "Hello World" being printed on the client shell. You have to restart the server nc command each time.

route without matchers after matcher

This tests if routes without matchers are not preferred when there exist matching routes before them.
This is a reproducer for a regression introduced in #208, that was fixed by #239.

References:

• afa78d7 update introduces a bug #233 (comment)
• prefetch immediately if matchers requires more data and if not alread… #239

route-without-matchers-after-matcher.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "layer4": {
      "servers": {
        "https": {
          "listen": ["0.0.0.0:10443"],
          "routes": [
            {
              "match": [
                {"http": [{"host": ["localhost"]}]}
              ],
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [{"dial": ["127.0.0.1:10080"]}]
                }
              ]
            },
            {
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [{"dial": ["127.0.0.1:10081"]}]
                }
              ]
            }
          ]
        }
      }
    },
    "http": {
      "servers": {
        "backend": {
          "protocols": ["h1"],
          "listen": ["127.0.0.1:10080"],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello from backend\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        },
        "fallback": {
          "protocols": ["h1"],
          "listen": ["127.0.0.1:10081"],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello from fallback\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        }
      }
    }
  }
}

Testing commands:

curl -v http://localhost:10443
curl -v --resolve fallback:10443:127.0.0.1 http://fallback:10443

The first curl should print "Hello from backend" and the second "Hello from fallback". During the regression both returned "Hello from fallback".

double matching not allowed

This tests if it is possible to match the same route multiple times. For example by recursively using tls in tls. This should not be possible, because it is confusing and often not what users want. But it was possible on some commits between #192 and #208.

References:

• remove unnecessary routes matching and restore old fallback next hand… #208 (comment)
• remove unnecessary routes matching and restore old fallback next hand… #208 (comment)

double-matching-not-allowed.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "tls": {
      "certificates": {
        "automate": ["localhost"]
      },
      "automation": {
        "policies": [{
          "subjects": ["localhost"],
          "issuers": [{
            "module": "internal"
          }]
        }]
      }
    },
    "layer4": {
      "servers": {
        "https": {
          "listen": ["0.0.0.0:10443"],
          "routes": [
            {
              "match": [
                {"tls": {"sni": ["localhost"]}}
              ],
              "handle": [
                {"handler": "tls"}
              ]
            },
            {
              "match": [
                {"http": [{"host": ["localhost"]}]}
              ],
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [{"dial": ["127.0.0.1:10080"]}]
                }
              ]
            }
          ]
        }
      }
    },
    "http": {
      "servers": {
        "backend": {
          "protocols": ["h1","h2","h2c"],
          "listen": ["127.0.0.1:10080"],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello World\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        }
      }
    }
  }
}

Testing Go program

package main

import (
	"crypto/tls"
	"log"
)

func main() {
	outerConn, err := tls.Dial("tcp", "127.0.0.1:10443", &tls.Config{
		ServerName:         "localhost",
		InsecureSkipVerify: true,
	})
	if err != nil {
		log.Fatal(err)
	}

	innerConn := tls.Client(outerConn, &tls.Config{
		ServerName:         "localhost",
		InsecureSkipVerify: true,
	})

	_, err = innerConn.Write([]byte("GET / HTTP/1.0\r\nHost: localhost\r\n\r\n"))
	if err != nil {
		log.Fatal(err)
	}
	buf := make([]byte, 1024)
	n, err := innerConn.Read(buf)
	if err != nil {
		log.Fatal(err)
	}
	println(string(buf[:n]))
}

The test program should print an EOF error since layer4 should immediately force close the connection after the first tls layer is processed. But on 6a8be7c for example it would print "Hello World".

out of order matching not allowed

This tests if out of order route matching is possible. It should not be possible, because it is confusing and often not what users want. Instead it should only be possible to execute routes in the order they are defined in the config. For example this config is using a proxy protocol handler as second route after tls.

out-of-order-matching-not-allowed.json

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {"level":"DEBUG", "encoder": {"format":"console"}}
    }
  },
  "apps": {
    "tls": {
      "certificates": {
        "automate": ["localhost"]
      },
      "automation": {
        "policies": [{
          "subjects": ["localhost"],
          "issuers": [{
            "module": "internal"
          }]
        }]
      }
    },
    "layer4": {
      "servers": {
        "https": {
          "listen": ["0.0.0.0:10443"],
          "routes": [
            {
              "match": [
                {"tls": {"sni": ["localhost"]}}
              ],
              "handle": [
                {"handler": "tls"}
              ]
            },
            {
              "match": [
                {"proxy_protocol": {}}
              ],
              "handle": [
                {
                  "handler": "proxy_protocol",
                  "allow": ["0.0.0.0/0"]
                }
              ]
            },
            {
              "match": [
                {"http": [{"host": ["localhost"]}]}
              ],
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [{"dial": ["127.0.0.1:10080"]}]
                }
              ]
            }
          ]
        }
      }
    },
    "http": {
      "servers": {
        "backend": {
          "protocols": ["h1","h2","h2c"],
          "listen": ["127.0.0.1:10080"],
          "routes": [
            {
              "handle": [{
                "handler": "static_response",
                "status_code": "200",
                "body": "Hello World\n",
                "headers": {
                  "Content-Type": ["text/plain"]
                }
              }]
            }
          ]
        }
      }
    }
  }
}

Testing commands:

curl -v -k --haproxy-protocol https://localhost:10443

This should immediately fail with layer4 closing the connection, because the proxy protocol route is not the first one. Or in other words the tls route should be unmatchable after the proxy protocol route was executed.

[mholt]

This is a great idea. Thank you for collecting these!

[mholt]

Related to #254, I have opened the Wiki tab for this project, where maybe we can make a collection of examples there, one per page.