add gzip brotli compression to http responses (close #2674) #2751

rakeshkky · 2019-08-19T05:46:45Z

Description

The server compresses the responses (success only) from /v1/query and /v1/graphql endpoints. The server chooses either brotli or gzip based on Accept-Encoding client header. If both are requested then the server picks brotli.

Affected components

Server
Docs
Build System (tests)
Tests

Related Issues

close #2674

Solution and Design

Add a new Hasura.Server.Compression module which exposes the compression type and function to compress.
Compress the end response (if required) and log the details of compression.

New things

Compression details are added to the logs. In the start-up log, the new field enable_compression is introduced which is a boolean value.

A new content_encoding field is added to http_info of http-log. It is either brotli or gzip and null if the encoding is not enabled or not requested by the client. See the example below.

{
  "timestamp": "2019-08-23T02:05:28.059-05:00",
  "level": "info",
  "type": "http-log",
  "detail": {
    "operation": {
      "query_execution_time": 0.018795933,
      "user_vars": {
        "x-hasura-role": "admin"
      },
      "error": null,
      "request_id": "6283add2-db7b-4eec-9b1e-d14781893b2f",
      "response_size": 45,
      "query": null
    },
    "http_info": {
      "status": 200,
      "http_version": "HTTP/1.1",
      "url": "/v1/query",
      "ip": "127.0.0.1",
      "method": "POST",
      "content_encoding": "brotli"
    }
  }
}

Steps to test and verify

In the console, make any query with the network tab opened in developer options. Check the size of response received when compression enabled and disabled.

Limitations, known bugs & workarounds

netlify · 2019-08-19T05:47:53Z

Deploy preview for hasura-docs ready!

Built with commit 15fe9eb

https://deploy-preview-2751--hasura-docs.netlify.com

ecthiender · 2019-08-19T06:39:48Z

There is a wai middleware which does gzip https://hackage.haskell.org/package/wai-extra-3.0.28/docs/Network-Wai-Middleware-Gzip.html . Shouldn't we use this directly?

server/src-lib/Hasura/Server/App.hs

ecthiender · 2019-08-20T12:45:25Z

There is a wai middleware which does gzip https://hackage.haskell.org/package/wai-extra-3.0.28/docs/Network-Wai-Middleware-Gzip.html . Shouldn't we use this directly?

@rakeshkky pointed out that we wanted to support Brotli compression as well (the wai middleware only does gzip), hence we rolled our own implementation.

marionschleifer

I left some comments 🙂

docs/graphql/manual/deployment/graphql-engine-flags/config-examples.rst

@marionschleifer

Suggested by @marionschleifer

Resolve Conflicts: .circleci/config.yml server/stack.yaml.lock

marionschleifer

💯

hasura-bot · 2019-08-26T07:17:21Z

Review app for commit d66c417 deployed to Heroku: https://hge-ci-pull-2751.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2751-d66c4176

Resolve Conflicts: server/src-lib/Hasura/Server/App.hs server/src-lib/Hasura/Server/Init.hs

hasura-bot · 2019-09-17T13:57:34Z

Review app for commit 7d4ac90 deployed to Heroku: https://hge-ci-pull-2751.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2751-7d4ac906

hasura-bot · 2019-09-18T20:19:34Z

Review app for commit befabf8 deployed to Heroku: https://hge-ci-pull-2751.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2751-befabf84

shahidhk

I have a feeling that ENABLE_COMPRESSION can be confusing with data compression in the database etc. Should we call it ENABLE_API_RESPONSE_COMPRESSION or ENABLE_API_COMPRESSION to be as explicit as we can?

lexi-lambda · 2019-09-19T04:58:09Z

I don’t have strong feelings about it either way, but how about ENABLE_HTTP_COMPRESSION? That seems both terse and unambiguous.

shahidhk · 2019-09-19T05:31:04Z

Also, why do we need a flag to enable this? The response is only compressed if the right Accept-Encoding header is present in the request, right? Are there any issues if this is enabled silently?

lexi-lambda · 2019-09-19T05:36:42Z

Are there any issues if this is enabled silently?

Not that I know of. I’d have no problem enabling it by default, but I’ll defer to @rakeshkky’s judgement on that.

The server compresses the response based on `Accept-Encoding` header value

rakeshkky · 2019-09-19T07:00:44Z

@shahidhk @lexi-lambda I've removed the --enable-compression flag in the latest commit. The server shouldn't have control over compression. It is the client which requests the compression based on Accept-Encoding header.

hasura-bot · 2019-09-19T07:10:55Z

Review app for commit 15fe9eb deployed to Heroku: https://hge-ci-pull-2751.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2751-15fe9eba

shahidhk

LGTM

hasura-bot · 2019-09-19T12:54:43Z

Review app https://hge-ci-pull-2751.herokuapp.com is deleted

Add brotli library dependencies to the server docker image. The compression feature introduced in #2751 requires brotli shared libraries at runtime. In original PR, adding them to server packager image was missing.

…sura#2751)

Add brotli library dependencies to the server docker image. The compression feature introduced in hasura#2751 requires brotli shared libraries at runtime. In original PR, adding them to server packager image was missing.

## Description ✍️ This PR fixes the config status update when the `Service configured successfully` message is written before the server is actually spawned. Now the status is updated only when the server is spawned successfully. To be specific, this change posts the status closer to where we log `starting API server`. ### Related Issues ✍ #2751 ### Solution and Design ✍ We update the status inside `runHGEServer` function. This helps in adding the message only when the server is started. If any exception is thrown before the server is spawned, only that message is written to `config_status` table instead of the `Service configured successfully` message. ## Affected components ✍️ - ✅ Server PR-URL: hasura/graphql-engine-mono#5179 Co-authored-by: Naveen Naidu <30195193+Naveenaidu@users.noreply.github.com> Co-authored-by: Anon Ray <616387+ecthiender@users.noreply.github.com> GitOrigin-RevId: 7860008403aa0645583e26915f620b66a5bbc531

allow compressing responses, close hasura#2674

bc9eb9c

rakeshkky added c/server Related to server c/docs Related to docs labels Aug 19, 2019

rakeshkky self-assigned this Aug 19, 2019

Merge branch 'master' into issue-2674-compressing

dfafeb7

rakeshkky added 2 commits August 20, 2019 12:49

update docs

6e6a9e1

Merge branch 'master' into issue-2674-compressing

bcdb852

ecthiender reviewed Aug 20, 2019

View reviewed changes

server/src-lib/Hasura/Server/App.hs Outdated Show resolved Hide resolved

Merge branch 'master' into issue-2674-compressing

f30ca5a

rakeshkky changed the title ~~allow compressing responses, close #2674~~ allow compressing responses (close #2674) Aug 21, 2019

rakeshkky added 3 commits August 22, 2019 12:13

compression details in logs

b0bdb33

add tests for compression

f53900f

Merge branch 'master' into issue-2674-compressing

79f5700

rakeshkky marked this pull request as ready for review August 23, 2019 07:20

rakeshkky requested review from lexi-lambda, marionschleifer and shahidhk as code owners August 23, 2019 07:20

rakeshkky added 2 commits August 23, 2019 12:52

Merge branch 'master' into issue-2674-compressing

7ec2ede

update server builder image to have latest stack version

c7aa0b8

marionschleifer requested changes Aug 26, 2019

View reviewed changes

rakeshkky added 3 commits August 26, 2019 12:09

add brotli dependencies docker images

c4a77b7

improve docs

510073d

Suggested by @marionschleifer

Merge branch 'master' into issue-2674-compressing

d66c417

Resolve Conflicts: .circleci/config.yml server/stack.yaml.lock

marionschleifer previously approved these changes Aug 26, 2019

View reviewed changes

use hge_key in compression tests

64ed551

rakeshkky dismissed marionschleifer’s stale review via 64ed551 August 26, 2019 07:35

Merge branch 'master' into issue-2674-compressing

7d4ac90

Resolve Conflicts: server/src-lib/Hasura/Server/App.hs server/src-lib/Hasura/Server/Init.hs

marionschleifer previously approved these changes Sep 18, 2019

View reviewed changes

lexi-lambda previously approved these changes Sep 18, 2019

View reviewed changes

shahidhk reviewed Sep 19, 2019

View reviewed changes

rakeshkky added 2 commits September 19, 2019 11:41

Merge branch 'master' into issue-2674-compressing

4800579

remove --enable-compression flag

15fe9eb

The server compresses the response based on `Accept-Encoding` header value

rakeshkky dismissed stale reviews from lexi-lambda and marionschleifer via 15fe9eb September 19, 2019 06:57

rakeshkky force-pushed the issue-2674-compressing branch from befabf8 to 15fe9eb Compare September 19, 2019 06:57

lexi-lambda approved these changes Sep 19, 2019

View reviewed changes

marionschleifer approved these changes Sep 19, 2019

View reviewed changes

shahidhk changed the title ~~allow compressing responses (close #2674)~~ allow compressing http responses (close #2674) Sep 19, 2019

shahidhk changed the title ~~allow compressing http responses (close #2674)~~ add gzip brotli compression to http responses (close #2674) Sep 19, 2019

shahidhk approved these changes Sep 19, 2019

View reviewed changes

shahidhk merged commit 8a0615f into hasura:master Sep 19, 2019

rakeshkky mentioned this pull request Sep 23, 2019

add brotli library dependencies to graphql-engine-packager image #2924

Merged

Toub mentioned this pull request Oct 10, 2019

add gzip compression #1253

Closed

polRk pushed a commit to polRk/graphql-engine that referenced this pull request Feb 12, 2020

add gzip brotli compression to http responses (close hasura#2674) (ha…

e607593

…sura#2751)

eaf-jtanneh mentioned this pull request Jan 25, 2021

Hasura returns content encoded in gzip instead of brotli #6484

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add gzip brotli compression to http responses (close #2674) #2751

add gzip brotli compression to http responses (close #2674) #2751

rakeshkky commented Aug 19, 2019 •

edited

netlify bot commented Aug 19, 2019 •

edited

ecthiender commented Aug 19, 2019

ecthiender commented Aug 20, 2019

marionschleifer left a comment

marionschleifer left a comment

hasura-bot commented Aug 26, 2019

hasura-bot commented Sep 17, 2019

hasura-bot commented Sep 18, 2019

shahidhk left a comment

lexi-lambda commented Sep 19, 2019

shahidhk commented Sep 19, 2019

lexi-lambda commented Sep 19, 2019

rakeshkky commented Sep 19, 2019

hasura-bot commented Sep 19, 2019

shahidhk left a comment

hasura-bot commented Sep 19, 2019

add gzip brotli compression to http responses (close #2674) #2751

add gzip brotli compression to http responses (close #2674) #2751

Conversation

rakeshkky commented Aug 19, 2019 • edited

Description

Affected components

Related Issues

Solution and Design

New things

Steps to test and verify

Limitations, known bugs & workarounds

netlify bot commented Aug 19, 2019 • edited

ecthiender commented Aug 19, 2019

ecthiender commented Aug 20, 2019

marionschleifer left a comment

Choose a reason for hiding this comment

marionschleifer left a comment

Choose a reason for hiding this comment

hasura-bot commented Aug 26, 2019

hasura-bot commented Sep 17, 2019

hasura-bot commented Sep 18, 2019

shahidhk left a comment

Choose a reason for hiding this comment

lexi-lambda commented Sep 19, 2019

shahidhk commented Sep 19, 2019

lexi-lambda commented Sep 19, 2019

rakeshkky commented Sep 19, 2019

hasura-bot commented Sep 19, 2019

shahidhk left a comment

Choose a reason for hiding this comment

hasura-bot commented Sep 19, 2019

rakeshkky commented Aug 19, 2019 •

edited

netlify bot commented Aug 19, 2019 •

edited