SQL handlers for custom media types #2825

steve-chavez · 2023-06-15T06:40:49Z

Summary

Custom media types are now possible on RPC and tables by using scalar functions and aggregates respectively.

Docs in progress: PostgREST/postgrest-docs#689

Examples

Create custom text/csv handler that includes a BOM (byte order mark) plus Content-Disposition header:

create domain "text/csv" as text;

-- override CSV with BOM plus attachment
create or replace function test.bom_csv_trans (state text, next test.lines)
returns text as $$
  select state || next.id::text || ',' || next.name || ',' || next.geom::text || E'\n';
$$ language sql;

create or replace function test.bom_csv_final (data text)
returns "text/csv" as $$
  select set_config('response.headers', '[{"Content-Disposition": "attachment; filename=\"lines.csv\""}]', true);
  -- EFBBBF is the BOM in UTF8 https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8
  select convert_from (decode (E'EFBBBF', 'hex'),'UTF8') || (E'id,name,geom\n' || data));
$$ language sql;

drop aggregate if exists test.bom_csv_agg(test.lines);
create aggregate test.bom_csv_agg (test.lines) (
  initcond = ''
, stype = text
, sfunc = bom_csv_trans
, finalfunc = bom_csv_final
);

curl 'localhost:3000/lines?id=in.(1,2)' -H "Accept: text/csv"  -i

HTTP/1.1 200 OK
Content-Disposition: attachment; filename="lines.csv"

id,name,geom
1,line-1,0102000020E610000002000000000000000000F03F000000000000F03F00000000000014400000000000001440
2,line-2,0102000020E6100000020000000000000000000040000000000000004000000000000018400000000000001840

# works for RPC that returns a table type too
curl 'localhost:3000/rpc/get_lines?id=eq.1' -H "Accept: text/csv"  -i

HTTP/1.1 200 OK
Content-Disposition: attachment; filename="lines.csv"

id,name,geom
1,line-1,0102000020E610000002000000000000000000F03F000000000000F03F00000000000014400000000000001440

Create a custom application/geo+json generic handler (anyelement can work thanks to ST_AsGeoJSON being generic):

create domain "application/geo+json" as jsonb;

create or replace function test.lines_geojson_trans (state jsonb, next anyelement)
returns jsonb as $$
  select state || extensions.ST_AsGeoJSON(next)::jsonb;
$$ language sql;

create or replace function test.lines_geojson_final (data jsonb)
returns "application/geo+json" as $$
  select jsonb_build_object(
    'type', 'FeatureCollection',
    'crs',  json_build_object(
        'type',      'name',
        'properties', json_build_object(
            'name', 'EPSG:4326'
         )
     ),
    'features', data
  );
$$ language sql;

drop aggregate if exists test.lines_geojson_agg(anyelement);
create aggregate test.lines_geojson_agg (anyelement) (
  initcond = '[]'
, stype = jsonb
, sfunc = lines_geojson_trans
, finalfunc = lines_geojson_final
);

curl 'localhost:3000/lines?id=in.(1,2)' -H "Accept: application/geo+json" 

{"crs": {"type": "name", "properties": {"name": "EPSG:4326"}}, "type": "FeatureCollection", "features": [{"type": "Feature", "geometry": {"type": "LineString", "coordinates": [[1, 1], [5, 
5]]}, "properties": {"id": 1, "name": "line-1"}}, {"type": "Feature", "geometry": {"type": "LineString", "coordinates": [[2, 2], [6, 6]]}, "properties": {"id": 2, "name": "line-2"}}]}

Create a custom html handler response

create domain "text/html" as text;

create or replace function welcome() returns "text/html" as $_$
select $$
<html>
  <head>
    <title>Custom handler for HTML</title>
  </head>
  <body>
    <h1>Welcome!</h1>
  </body>
</html>
$$;
$_$ language sql;

curl 'localhost:3000/rpc/welcome' -H "Accept: text/html"

<html>
  <head>
    <title>Custom handler for HTML</title>
  </head>
  <body>
    <h1>Welcome!</h1>
  </body>
</html>

Breaks

Removed raw-media-types config
Removed application/octet-stream, text/plain, text/xml builtin support for scalar results
Removed default application/openapi+json media type for db-root-spec
All can be migrated without impacting API clients by using custom handlers.

Closes

Closes Feature request: allow user-defined function to handle application-specific Accept headers #1548
Closes geojson structure: crs inside each feature instead on level type and features #2699
Closes Add a HTML table response type #2763
Closes Don't fail on RPC withAccept: text/plain, application/json #2170
Closes Binary Endpoints #1462
Closes Feature request: option to change delimiter character for CSV output #1102
Closes CSV output issues #1374
Closes Refactor module MediaType.hs to use Network.HTTP.Media #2901. No longer think it's necessary to use an external module.

Pending

Both feat: support XML? #2188 and Asymetric CSV encoding/decoding #1043 would need Custom media types + conversion for Content-Type #2826

test/spec/Feature/Query/PlanSpec.hs

steve-chavez · 2023-06-21T03:59:26Z

There's a problem in solving #2699 with the current aggregate interface.

So having a replacement for our geojson is simple:

create domain "application/geo+json" as jsonb;

create or replace function test.geoson_trans (state jsonb, next anyelement)
returns jsonb as $$
  select state || extensions.ST_AsGeoJSON(next)::jsonb;
$$ language sql;

create or replace function test.geojson_final (data jsonb)
returns "application/geo+json" as $$
  select jsonb_build_object('type', 'FeatureCollection', 'features', data);
$$ language sql;

create or replace aggregate test.geojson_agg (anyelement) (
  initcond = '[]'
, stype = jsonb
, sfunc = geoson_trans
, finalfunc = geojson_final
);

curl "localhost:3000/shops" -H "Accept: application/geo+json"

{"type": "FeatureCollection", "features": [{"type": "Feature", "geometry": {"type": "Point", "coordinates": [-71.10044, 42.373695]}, "properties": {"id": 1, "address": "1369 Cambridge St"}}, {
"type": "Feature", "geometry": {"type": "Point", "coordinates": [-71.10543, 42.366432]}, "properties": {"id": 2, "address": "757 Massachusetts Ave"}}, {"type": "Feature", "geometry": {"type":
"Point", "coordinates": [-71.081924, 42.36437]}, "properties": {"id": 3, "address": "605 W Kendall St"}}]}

If we want to add a crs to the final geojson, we could do it like:

create or replace function test.geojson_final (data jsonb)
returns "application/geo+json" as $$
  select jsonb_build_object(
    'type', 'FeatureCollection', 
    'crs',  json_build_object(
        'type',      'name', 
        'properties', json_build_object(
            'name', 'EPSG:4326'  
         )
     ),
    'features', data
  );
$$ language sql;

That's not a generic way though, ideally we use Find_SRID as recommend on #2699.

For this, we could use FINALFUNC_EXTRA:

create or replace function test.geojson_final (data jsonb, typ anyelement)
returns "application/geo+json" as $$
  -- here, assume we enhance pg_typeof with what's required for find_SRID to work, i.e. getting the schema, name and column. Which is possible.
  select jsonb_build_object('type', 'FeatureCollection', 'type', pg_typeof(typ), 'features', data);
$$ language sql;

create or replace aggregate test.geojson_agg (anyelement) (
  initcond = '[]'
, stype = jsonb
, sfunc = geoson_trans
, finalfunc = geojson_final
, finalfunc_extra = true
);

curl "localhost:3000/shops" -H "Accept: application/geo+json"
{"type": "FeatureCollection", "type": "record", "features": [{"type": "Feature", "geometry": {"type": "Point", "coordinates": [-71.10044, 42.373695]}, "properties": {"id": 1, "address": "1369
 Cambridge St"}}, {"type": "Feature", "geometry": {"type": "Point", "coordinates": [-71.10543, 42.366432]}, "properties": {"id": 2, "address": "757 Massachusetts Ave"}}, {"type": "Feature", "g
eometry": {"type": "Point", "coordinates": [-71.081924, 42.36437]}, "properties": {"id": 3, "address": "605 W Kendall St"}}]}

Notice there's a type: record. This is bc of how we call the aggregation:

customAggF :: QualifiedIdentifier -> SqlFragment
customAggF qi = fromQi qi <> "(_postgrest_t)"

If we casted _postgrest_t::test.shops, then the correct type: shops would show.

The casting won't work if resource embedding is done.

curl "localhost:3000/shops?select=*,shop_bles(*)" -H "Accept: application/geo+json"
{"code":"42846","details":"Input has too many columns.","hint":null,"message":"cannot cast type record to shops"}

So maybe we can cast it only when embedding is not used? That would make the aggregate function unreliable though.

Or maybe we should disallow resource embedding on non application/json media types?

I believe this is something it should be solved in PostGIS itself. Maybe it's not worth it. The user can define different types of geojson aggs manually.

src/PostgREST/SchemaCache.hs

steve-chavez · 2023-06-21T06:29:30Z

I was under the wrong impression that we could do everything with aggregates. For instance, creating an aggregate that also strips nulls #1601. For this we would only need to apply a function to the final function of the aggregate.

json_agg uses json_agg_finalfn and json_agg_transfn. So:

create or replace function test.appjson_agg_final (data json)
returns "application/json" as $$
  select json_strip_nulls(pg_catalog.json_agg_finalfn($1));
$$ language sql;

-- ERROR:  function pg_catalog.json_agg_finalfn(json) does not exist
-- LINE 3:   select json_strip_nulls(pg_catalog.json_agg_finalfn($1));
-- HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

create or replace aggregate test.appjson_agg (anyelement) (
  initcond = '[]'
, stype = json
, sfunc = pg_catalog.json_agg_transfn
, finalfunc = appjson_agg_final
);

The above won't work bc json_agg_finalfn argument is internal:

postgres=# \df+ json_agg_finalfn
List of functions
-[ RECORD 1 ]-------+------------------------------
Schema              | pg_catalog
Name                | json_agg_finalfn
Result data type    | json
Argument data types | internal

Wolfgang idea with CASTs would solve this case #1582 (comment). So looks we'll need CASTs anyway.

CREATE CAST ("application/json" AS json) WITH FUNCTION  json_strip_nulls;

In general, CASTs allows us to have function composition.

I'm thinking we should merge #2523 before going further on this feature.

steve-chavez · 2023-10-09T17:54:12Z

I was under the wrong impression that we could do everything with aggregates. For instance, creating an aggregate that also strips nulls #1601

Alternatively, I think this could be solved later with the postgrest-contrib extension. We might be able to define custom aggregates there. Even some that could integrate with our tx settings (request.method, etc). With this we could avoid the need for CASTs too (though I won't preclude the possibility of implementing that in the future), which were deemed complicated by a user recently (ref) .

steve-chavez · 2023-10-10T18:00:25Z

I've just found out that thanks the new GRANT SET ON PARAMETER on pg 15, we can implement the original proposal with the custom GUC #1582. It works like this:

create role manager login password 'secret';
grant all on schema public to manager;

set pgrst.bar to ''; -- this must be done otherwise the next statement fails with permission denied. I believe what happens is that the GUC is now registered as a placeholder.
grant set on parameter pgrst.bar to manager;
GRANT

begin;
set local role to manager;

create or replace function a_bar() returns text as $$
  select 'hello';
$$ language sql
set pgrst.bar='foo';
CREATE FUNCTION

This even works on RDS.

The original proposal had the advantage of being able to define a default handler for */* in a more simple way.

The only thing that worries me is that the above could be considered a bug somehow and then it could be patched but it seems unlikely since it kinda makes sense (check the comment above).

~~I'll proceed with the pgrst.handler implementation.~~

cc @wolfgangwalther

wolfgangwalther · 2023-10-10T18:14:29Z

I've just found out that thanks the new GRANT SET ON PARAMETER on pg 15, we can implement the original proposal with the custom GUC #1582. It works like this:

[...]

This even works on RDS.

While this solves your concern about the availability of this feature on cloud providers, it does not solve the inlining problem mentioned in #1582 (comment).

The idea of using SET pgrst.xxx on the function definition should be generally abandoned. The approach via CASTs and "representations" is a very sound concept, so I don't see a reason to not do that.

test/spec/fixtures/schema.sql

test/spec/Feature/Query/CustomMediaSpec.hs

steve-chavez · 2023-10-24T18:42:09Z

As a consequence of #2894, application/vnd.pgrst.object and application/vnd.pgrst.object are no longer overridable per our original design, since they take parameters (nulls=stripped) now.

Maybe it's not so bad to make vendored media types non-overridable. After all users cannot override vnd.pgrst.plan too, this one has a lot of special parameters.'

We could choose to drop nulls=stripped but as mentioned above the same effect can't be achieved right now with just one single handler function. We would need the CASTs thing.

steve-chavez · 2023-10-25T19:23:11Z

test/spec/Feature/Query/CustomMediaSpec.hs

+    it "will match image/png according to q values" $ do
+      r1 <- request methodGet "/rpc/ret_image" (acceptHdrs "image/png, */*") ""
+      liftIO $ do
+        simpleBody r1 `shouldBe` readFixtureFile "A.png"
+        simpleHeaders r1 `shouldContain` [("Content-Type", "image/png")]
+
+      r2 <- request methodGet "/rpc/ret_image" (acceptHdrs "text/html,application/xhtml+xml,application/xml;q=0.9,image/png,*/*;q=0.8") ""
+      liftIO $ do
+        simpleBody r2 `shouldBe` readFixtureFile "A.png"
+        simpleHeaders r2 `shouldContain` [("Content-Type", "image/png")]


Clears the case in #1462, replace image/png for image/webp. This is more about correct content negotiation.

steve-chavez · 2023-10-25T19:24:05Z

test/spec/Feature/Query/CustomMediaSpec.hs

+    -- https://github.com/PostgREST/postgrest/issues/1371#issuecomment-519248984
+    it "will match a custom text/csv with BOM" $ do
+      r <- request methodGet "/lines" (acceptHdrs "text/csv") ""
+      liftIO $ do
+        simpleBody r `shouldBe` readFixtureFile "lines.csv"
+        simpleHeaders r `shouldContain` [("Content-Type", "text/csv; charset=utf-8")]
+        simpleHeaders r `shouldContain` [("Content-Disposition", "attachment; filename=\"lines.csv\"")]


Clears #1371 (comment), includes a BOM CSV plus attachment header. Had to read from a file otherwise couldn't match the BOM on Haskell

BREAKING CHANGE Can be done later with custom media types

* test text/html and drop HtmlRawOutputSpec.hs * all tests passing, removed all pendingWith * make functions compatible with pg <= 12 * move custom media types tests to own spec * anyelement aggregate * apply aggregates without a final function * overriding works * overriding anyelement with particular agg * cannot override vendored media types * plan spec works with custom aggregate * renamed media types to make clear which ones are overridable * correct content negotiation with same weight * text/tab-separated-values media type * text/csv with BOM plus content-disposition header

steve-chavez · 2023-10-25T21:35:57Z

Ok this should be ready to go. It implements the first stage idea discussed on #1582 (comment). This means that the CASTs idea can be built on top.

The code supports the idea of "handlers", not only aggregates, meaning a regular function can be the final one applied.

handlerF :: Maybe Routine -> QualifiedIdentifier -> MediaHandler -> SQL.Snippet
handlerF rout target = \case
  BuiltinAggArrayJsonStrip   -> asJsonF rout True
  BuiltinAggSingleJson strip -> asJsonSingleF rout strip
  -- these ones are overridable
  BuiltinOvAggJson           -> asJsonF rout False
  BuiltinOvAggGeoJson        -> asGeoJsonF
  BuiltinOvAggCsv            -> asCsvF
  CustomFunc funcQi          -> customFuncF rout funcQi target
  NoAgg                      -> "''::text"

customFuncF :: Maybe Routine -> QualifiedIdentifier -> QualifiedIdentifier -> SQL.Snippet
customFuncF rout funcQi target
  | (funcReturnsScalar <$> rout) == Just True = fromQi funcQi <> "(_postgrest_t.pgrst_scalar)"
  | otherwise                                 = fromQi funcQi <> "(_postgrest_t::" <> fromQi target <> ")"

There might be some rough edges, but it already solves a lot of issues. Merging it later will cost a lot (it already took a lot of rebases), so I'd rather merge it at this stage and improve it later.

steve-chavez · 2023-10-26T14:03:44Z

Merging. Will continue with #2826 to solve the remaining issues.

steve-chavez · 2023-11-28T15:19:42Z

With raw-media-types gone, https://postgrest.org/en/stable/how-tos/providing-images-for-img.html is not possible now. We'd need a way to dynamically set the accepted media type inside the function.

This was removed in #2825.

This was referenced Jun 15, 2023

Custom media types + conversion for Content-Type #2826

Open

Add pgrst.accept setting to RPC #1582

Closed

refactor: clarify content negotiation #2827

Merged

refactor: remove MTPlanAttrs to doctest #2829

Merged

steve-chavez force-pushed the domain-media branch 3 times, most recently from 4243c4a to ee3d182 Compare June 20, 2023 21:37

steve-chavez commented Jun 20, 2023

View reviewed changes

test/spec/Feature/Query/PlanSpec.hs Outdated Show resolved Hide resolved

steve-chavez mentioned this pull request Jun 20, 2023

refactor: stricter plan media type #2833

Merged

steve-chavez force-pushed the domain-media branch from ee3d182 to 7ec4ee7 Compare June 21, 2023 02:06

steve-chavez commented Jun 21, 2023

View reviewed changes

src/PostgREST/SchemaCache.hs Outdated Show resolved Hide resolved

This was referenced Jul 7, 2023

HEAD request unnecessarily sends query data back to PostgREST #2849

Closed

fix: HEAD unnecessarily executing aggregates #2857

Merged

steve-chavez mentioned this pull request Jul 19, 2023

Support for PGroonga operators #2028

Open

steve-chavez mentioned this pull request Sep 11, 2023

refactor: move media type logic to Plan module #2934

Merged

steve-chavez force-pushed the domain-media branch from 7ec4ee7 to 5505f64 Compare October 8, 2023 02:21

steve-chavez mentioned this pull request Oct 18, 2023

Ordering embedded resources for PATCH request is broken #3013

Open

steve-chavez force-pushed the domain-media branch from 5505f64 to 71830ec Compare October 20, 2023 04:27

steve-chavez commented Oct 20, 2023

View reviewed changes

test/spec/fixtures/schema.sql Outdated Show resolved Hide resolved

steve-chavez force-pushed the domain-media branch from 71830ec to 45c05c5 Compare October 20, 2023 05:09

steve-chavez commented Oct 20, 2023

View reviewed changes

test/spec/fixtures/schema.sql Outdated Show resolved Hide resolved

steve-chavez force-pushed the domain-media branch from 6462f64 to 8ac8067 Compare October 24, 2023 16:11

steve-chavez commented Oct 24, 2023

View reviewed changes

test/spec/Feature/Query/CustomMediaSpec.hs Show resolved Hide resolved

steve-chavez force-pushed the domain-media branch from 8ac8067 to 61da62b Compare October 24, 2023 21:17

steve-chavez force-pushed the domain-media branch from 61da62b to 7ce199c Compare October 25, 2023 19:20

steve-chavez commented Oct 25, 2023

View reviewed changes

steve-chavez force-pushed the domain-media branch from 7ce199c to 707b100 Compare October 25, 2023 19:42

steve-chavez added 2 commits October 25, 2023 15:37

break: remove binary field logic, raw-media-types

602cc8d

BREAKING CHANGE Can be done later with custom media types

break: db-root-spec default app/openapi+json media

d767398

BREAKING CHANGE Can be done later with custom media types

steve-chavez changed the title ~~Custom media types for Accept~~ SQL handlers for custom media types Oct 25, 2023

steve-chavez changed the title ~~SQL handlers for custom media types~~ SQL handlers for custom media types on Accept Oct 25, 2023

steve-chavez changed the title ~~SQL handlers for custom media types on Accept~~ SQL handlers for custom media types Oct 25, 2023

steve-chavez force-pushed the domain-media branch from 707b100 to ee842eb Compare October 25, 2023 21:16

steve-chavez marked this pull request as ready for review October 25, 2023 21:16

steve-chavez force-pushed the domain-media branch from ee842eb to cb87772 Compare October 25, 2023 21:23

steve-chavez force-pushed the domain-media branch from cb87772 to 33c008b Compare October 25, 2023 21:35

steve-chavez mentioned this pull request Oct 26, 2023

doc: media type handlers PostgREST/postgrest-docs#689

Merged

steve-chavez merged commit 82b3834 into PostgREST:main Oct 26, 2023
30 of 31 checks passed

fjf2002 mentioned this pull request Oct 31, 2023

feat: support XML? #2188

Open

This was referenced Nov 3, 2023

More flexible SQL media type handlers #3037

Closed

Update providing-html-content-using-htmx.rst PostgREST/postgrest-docs#691

Closed

This was referenced Nov 29, 2023

feat: custom SQL handler for the "*/*" media type #3076

Merged

reference: media type handlers PostgREST/postgrest-docs#704

Merged

wolfgangwalther added a commit that referenced this pull request May 12, 2024

refactor: Remove left-over raw_media_types from Database.hs

c6f152f

This was removed in #2825.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SQL handlers for custom media types #2825

SQL handlers for custom media types #2825

steve-chavez commented Jun 15, 2023 •

edited

Loading

steve-chavez commented Jun 21, 2023 •

edited

Loading

steve-chavez commented Jun 21, 2023 •

edited

Loading

steve-chavez commented Oct 9, 2023

steve-chavez commented Oct 10, 2023 •

edited

Loading

wolfgangwalther commented Oct 10, 2023 •

edited

Loading

steve-chavez commented Oct 24, 2023 •

edited

Loading

steve-chavez Oct 25, 2023

steve-chavez Oct 25, 2023 •

edited

Loading

steve-chavez commented Oct 25, 2023 •

edited

Loading

steve-chavez commented Oct 26, 2023

steve-chavez commented Nov 28, 2023

SQL handlers for custom media types #2825

SQL handlers for custom media types #2825

Conversation

steve-chavez commented Jun 15, 2023 • edited Loading

Summary

Examples

Breaks

Closes

Pending

steve-chavez commented Jun 21, 2023 • edited Loading

steve-chavez commented Jun 21, 2023 • edited Loading

steve-chavez commented Oct 9, 2023

steve-chavez commented Oct 10, 2023 • edited Loading

wolfgangwalther commented Oct 10, 2023 • edited Loading

steve-chavez commented Oct 24, 2023 • edited Loading

steve-chavez Oct 25, 2023

Choose a reason for hiding this comment

steve-chavez Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

steve-chavez commented Oct 25, 2023 • edited Loading

steve-chavez commented Oct 26, 2023

steve-chavez commented Nov 28, 2023

steve-chavez commented Jun 15, 2023 •

edited

Loading

steve-chavez commented Jun 21, 2023 •

edited

Loading

steve-chavez commented Jun 21, 2023 •

edited

Loading

steve-chavez commented Oct 10, 2023 •

edited

Loading

wolfgangwalther commented Oct 10, 2023 •

edited

Loading

steve-chavez commented Oct 24, 2023 •

edited

Loading

steve-chavez Oct 25, 2023 •

edited

Loading

steve-chavez commented Oct 25, 2023 •

edited

Loading