Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch default encoding to gzip for now #1260

Merged
merged 2 commits into from
May 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

55 changes: 30 additions & 25 deletions docs/src/config-file.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,17 @@
# Configuration File

If you don't want to expose all of your tables and functions, you can list your sources in a configuration file. To start Martin with a configuration file you need to pass a path to a file with a `--config` argument. Config files may contain environment variables, which will be expanded before parsing. For example, to use `MY_DATABASE_URL` in your config file: `connection_string: ${MY_DATABASE_URL}`, or with a default `connection_string: ${MY_DATABASE_URL:-postgresql://postgres@localhost/db}`
If you don't want to expose all of your tables and functions, you can list your sources in a configuration file. To
start Martin with a configuration file you need to pass a path to a file with a `--config` argument. Config files may
contain environment variables, which will be expanded before parsing. For example, to use `MY_DATABASE_URL` in your
config file: `connection_string: ${MY_DATABASE_URL}`, or with a
default `connection_string: ${MY_DATABASE_URL:-postgresql://postgres@localhost/db}`

```shell
martin --config config.yaml
```

You may wish to auto-generate a config file with `--save-config` argument. This will generate a config yaml file with all of your configuration, which you can edit to remove any sources you don't want to expose.
You may wish to auto-generate a config file with `--save-config` argument. This will generate a config yaml file with
all of your configuration, which you can edit to remove any sources you don't want to expose.

```shell
martin ... ... ... --save-config config.yaml
Expand All @@ -31,7 +36,7 @@ worker_processes: 8
# Amount of memory (in MB) to use for caching tiles [default: 512, 0 to disable]
cache_size_mb: 1024

# If the client accepts multiple compression formats, and the tile source is not pre-compressed, which compression should be used. `gzip` is faster, but `brotli` is smaller, and may be faster with caching. Defaults to brotli.
# If the client accepts multiple compression formats, and the tile source is not pre-compressed, which compression should be used. `gzip` is faster, but `brotli` is smaller, and may be faster with caching. Default could be different depending on Martin version.
preferred_encoding: gzip

# Database configuration. This can also be a list of PG configs.
Expand Down Expand Up @@ -92,76 +97,76 @@ postgres:
functions:
# Optionally set how source ID should be generated based on the function's name and schema
source_id_format: '{schema}.{function}'

# Associative arrays of table sources
tables:
table_source_id:
# ID of the MVT layer (optional, defaults to table name)
layer_id: table_source

# Table schema (required)
schema: public

# Table name (required)
table: table_source

# Geometry SRID (required)
srid: 4326

# Geometry column name (required)
geometry_column: geom

# Feature id column name
id_column: ~

# An integer specifying the minimum zoom level
minzoom: 0

# An integer specifying the maximum zoom level. MUST be >= minzoom
maxzoom: 30

# The maximum extent of available map tiles. Bounds MUST define an area
# covered by all zoom levels. The bounds are represented in WGS:84
# latitude and longitude values, in the order left, bottom, right, top.
# Values may be integers or floating point numbers.
bounds: [-180.0, -90.0, 180.0, 90.0]
bounds: [ -180.0, -90.0, 180.0, 90.0 ]

# Tile extent in tile coordinate space
extent: 4096

# Buffer distance in tile coordinate space to optionally clip geometries
buffer: 64

# Boolean to control if geometries should be clipped or encoded as is
clip_geom: true

# Geometry type
geometry_type: GEOMETRY

# List of columns, that should be encoded as tile properties (required)
properties:
gid: int4

# Associative arrays of function sources
functions:
function_source_id:
# Schema name (required)
schema: public

# Function name (required)
function: function_zxy_query

# An integer specifying the minimum zoom level
minzoom: 0

# An integer specifying the maximum zoom level. MUST be >= minzoom
maxzoom: 30

# The maximum extent of available map tiles. Bounds MUST define an area
# covered by all zoom levels. The bounds are represented in WGS:84
# latitude and longitude values, in the order left, bottom, right, top.
# Values may be integers or floating point numbers.
bounds: [-180.0, -90.0, 180.0, 90.0]
bounds: [ -180.0, -90.0, 180.0, 90.0 ]

# Publish PMTiles files from local disk or proxy to a web server
pmtiles:
Expand All @@ -177,7 +182,7 @@ pmtiles:
pm-src1: /path/to/pmt.pmtiles
# A named source to a web server with a PMTiles file that supports range requests
pm-web2: https://example.org/path/tiles.pmtiles

# Publish MBTiles files
mbtiles:
paths:
Expand Down
5 changes: 3 additions & 2 deletions docs/src/run-with-cli.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
## Command-line Interface

You can configure Martin using command-line interface. See `martin --help` or `cargo run -- --help` for more information.
You can configure Martin using command-line interface. See `martin --help` or `cargo run -- --help` for more
information.

```shell
Usage: martin [OPTIONS] [CONNECTION]...
Expand Down Expand Up @@ -34,7 +35,7 @@ Options:
Number of web server workers

--preferred-encoding <PREFERRED_ENCODING>
Martin server preferred tile encoding. If the client accepts multiple compression formats, and the tile source is not pre-compressed, which compression should be used. `gzip` is faster, but `brotli` is smaller, and may be faster with caching. Defaults to brotli
Martin server preferred tile encoding. If the client accepts multiple compression formats, and the tile source is not pre-compressed, which compression should be used. `gzip` is faster, but `brotli` is smaller, and may be faster with caching. Default could be different depending on Martin version

[possible values: brotli, gzip]

Expand Down
1 change: 1 addition & 0 deletions martin/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -120,3 +120,4 @@ ctor.workspace = true
indoc.workspace = true
insta = { workspace = true, features = ["yaml"] }
pprof.workspace = true
rstest.workspace = true
5 changes: 2 additions & 3 deletions martin/src/args/srv.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,14 @@ pub struct SrvArgs {
/// Number of web server workers
#[arg(short = 'W', long)]
pub workers: Option<usize>,
/// Martin server preferred tile encoding. If the client accepts multiple compression formats, and the tile source is not pre-compressed, which compression should be used. `gzip` is faster, but `brotli` is smaller, and may be faster with caching. Defaults to brotli.
/// Martin server preferred tile encoding. If the client accepts multiple compression formats, and the tile source is not pre-compressed, which compression should be used. `gzip` is faster, but `brotli` is smaller, and may be faster with caching. Defaults to gzip.
#[arg(long)]
pub preferred_encoding: Option<PreferredEncoding>,
}

#[derive(PartialEq, Eq, Default, Debug, Clone, Copy, Serialize, Deserialize, ValueEnum)]
#[derive(PartialEq, Eq, Debug, Clone, Copy, Serialize, Deserialize, ValueEnum)]
#[serde(rename_all = "lowercase")]
pub enum PreferredEncoding {
#[default]
#[serde(alias = "br")]
#[clap(alias("br"))]
Brotli,
Expand Down
80 changes: 37 additions & 43 deletions martin/src/srv/tiles.rs
Original file line number Diff line number Diff line change
Expand Up @@ -199,8 +199,8 @@ impl<'a> DynTileSource<'a> {

if tile.info.encoding == Encoding::Uncompressed {
let ordered_encodings = match self.preferred_enc {
Some(PreferredEncoding::Gzip) => PREFER_GZIP_ENC,
Some(PreferredEncoding::Brotli) | None => PREFER_BROTLI_ENC,
Some(PreferredEncoding::Gzip) | None => PREFER_GZIP_ENC,
Some(PreferredEncoding::Brotli) => PREFER_BROTLI_ENC,
};

// only apply compression if the content supports it
Expand Down Expand Up @@ -264,58 +264,52 @@ pub fn to_encoding(val: ContentEncoding) -> Option<Encoding> {

#[cfg(test)]
mod tests {
use rstest::rstest;
use tilejson::tilejson;

use super::*;
use crate::srv::server::tests::TestSource;

#[rstest]
#[trace]
#[case(&["gzip", "deflate", "br", "zstd"], None, Encoding::Gzip)]
#[case(&["gzip", "deflate", "br", "zstd"], Some(PreferredEncoding::Brotli), Encoding::Brotli)]
#[case(&["gzip", "deflate", "br", "zstd"], Some(PreferredEncoding::Gzip), Encoding::Gzip)]
#[case(&["br;q=1", "gzip;q=1"], Some(PreferredEncoding::Gzip), Encoding::Gzip)]
#[case(&["gzip;q=1", "br;q=1"], Some(PreferredEncoding::Brotli), Encoding::Brotli)]
#[case(&["gzip;q=1", "br;q=0.5"], Some(PreferredEncoding::Brotli), Encoding::Gzip)]
#[actix_rt::test]
async fn test_encoding_preference() {
let source = TestSource {
async fn test_enc_preference(
#[case] accept_enc: &[&'static str],
#[case] preferred_enc: Option<PreferredEncoding>,
#[case] expected_enc: Encoding,
) {
let sources = TileSources::new(vec![vec![Box::new(TestSource {
id: "test_source",
tj: tilejson! { tiles: vec![] },
data: vec![1_u8, 2, 3],
};
let sources = TileSources::new(vec![vec![Box::new(source)]]);
})]]);

for (accept_encodings, prefered_encoding, result_encoding) in [
(
Some(AcceptEncoding(vec![
"gzip;q=1".parse().unwrap(),
"br;q=1".parse().unwrap(),
])),
Some(PreferredEncoding::Brotli),
Encoding::Brotli,
),
(
Some(AcceptEncoding(vec![
"gzip;q=1".parse().unwrap(),
"br;q=0.5".parse().unwrap(),
])),
Some(PreferredEncoding::Brotli),
Encoding::Gzip,
),
] {
let src = DynTileSource::new(
&sources,
"test_source",
None,
"",
accept_encodings,
prefered_encoding,
None,
)
.unwrap();
let xyz = TileCoord { z: 0, x: 0, y: 0 };
let data = &src.get_tile_content(xyz).await.unwrap().data;
let decoded = match result_encoding {
Encoding::Gzip => decode_gzip(data),
Encoding::Brotli => decode_brotli(data),
_ => panic!("Unexpected encoding"),
};
assert_eq!(vec![1_u8, 2, 3], decoded.unwrap());
}
let accept_enc = Some(AcceptEncoding(
accept_enc.iter().map(|s| s.parse().unwrap()).collect(),
));

let src = DynTileSource::new(
&sources,
"test_source",
None,
"",
accept_enc,
preferred_enc,
None,
)
.unwrap();

let xyz = TileCoord { z: 0, x: 0, y: 0 };
let tile = src.get_tile_content(xyz).await.unwrap();
assert_eq!(tile.info.encoding, expected_enc);
}

#[actix_rt::test]
async fn test_tile_content() {
let non_empty_source = TestSource {
Expand Down
Loading