New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "Accept-Encoding: gzip" header to mapd.py to reduce data consumption #525

Merged
merged 92 commits into from Feb 13, 2019

Conversation

Projects
None yet
3 participants
@eFiniLan
Copy link
Contributor

eFiniLan commented Feb 7, 2019

after a few tests, I noticed that overpass.kumi.systems server does not support any kind of response data compression.

The response data size greatly reduced when we use overpass-api.de server with "Accept-Encoding: gzip" header added.

With the same request, overpass-api.de: 271k vs overpass.kumi.systems: 2227k

image

We can potentially increase the radius to 9000 on overpass-api.de and consume around the same amount of data on overpass.kumi.systems with radius of 3000

image

Rick Lan and others added some commits Aug 30, 2018

Rick Lan
Merge branch 'devel-lexus-ish-ave30r' of https://github.com/EternityL…
…an/openpilot into devel-lexus-ish-ave30r

# Conflicts:
#	selfdrive/car/toyota/interface.py
Rick Lan
Merge branch 'devel' of https://github.com/commaai/openpilot into dev…
…el-lexus-ish-ave30r

# Conflicts:
#	selfdrive/boardd/boardd.cc
#	selfdrive/car/toyota/interface.py
#	selfdrive/car/toyota/values.py
Rick Lan
Rick Lan
Merge branch 'devel' of https://github.com/commaai/openpilot into dev…
…el-lexus-ish-ave30r

# Conflicts:
#	selfdrive/car/toyota/interface.py
Rick Lan
Rick Lan

eFiniLan added some commits Nov 19, 2018

use overpass-api.de instead of overpass.kumi.systems, as overpass-api…
….de support "Accept-Encoding: gzip" header which greatly reduce the response data size.

blender0 pushed a commit to blender0/openpilot that referenced this pull request Feb 9, 2019

@pd0wm

This comment has been minimized.

Copy link
Contributor

pd0wm commented Feb 11, 2019

I really like the idea of using compression to reduce the download size. Unfortunately all the openpilot users combined generate quite a large number of requests, which this endpoint might not be able to handle.

Currently we have permission by the owners of the current endpoint to hit the endpoint as much as we like. I'll contact them and ask if they can support gzip support. Otherwise we will host our own endpoint.

@eFiniLan

This comment has been minimized.

Copy link
Contributor Author

eFiniLan commented Feb 11, 2019

I really like the idea of using compression to reduce the download size. Unfortunately all the openpilot users combined generate quite a large number of requests, which this endpoint might not be able to handle.

Currently we have permission by the owners of the current endpoint to hit the endpoint as much as we like. I'll contact them and ask if they can support gzip support. Otherwise we will host our own endpoint.

no worries that's understandable 👍 hopefully kumi system will support gzip soon. :)

@arne182

This comment has been minimized.

Copy link
Contributor

arne182 commented Feb 12, 2019

@pd0wm
What about downloading the open street maps statewise and run the server locally on the eon?

@pd0wm

This comment has been minimized.

Copy link
Contributor

pd0wm commented Feb 12, 2019

@eFiniLan according to Kumi Systems they have enabled gzip. Can you verify that this is the case?

@arne182, that would be nice, but that would require a nice UI to select and sync selected areas. We definitely want to switch to that in the future.

@eFiniLan

This comment has been minimized.

Copy link
Contributor Author

eFiniLan commented Feb 13, 2019

@eFiniLan according to Kumi Systems they have enabled gzip. Can you verify that this is the case?

@pd0wm looks like it's working now, I've changed the server back to kumi :)
image

@eFiniLan eFiniLan changed the title Use Overpass API server that support "Accept-Encoding: gzip" header to reduce data consumption Add "Accept-Encoding: gzip" header to mapd.py to reduce data consumption Feb 13, 2019

@pd0wm pd0wm merged commit 03f13e6 into commaai:devel Feb 13, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment