Added possibility to add a custom http request user agent header #115

sebastianneubauer · 2015-03-21T14:53:01Z

My application that uses geopy was blocked by Nominatim as geopy uses the standard user agent header for requests provided by urllib (Python-urllib/2.7). Nominatim requires to add an identifying user agent header per application in order to be able to track and block requests if needed.

I now added in this PR a default header in the base geocoder class:
User-Agent: geopy/#version
where #version is the current geopy version (e.g "geopy/1.9.1")

This is now added to every request and every geocoder inheriting from the 'Geocoder' base class.
This should be much better, as it is much easier for services to be able to track which request come from where. Now every service is able to distinguish geopy requests from non geopy requests.

In addition I added the possibility to pass in a custom user agent string to the Nominatim geocoder class:

        geocoder = Nominatim(
            user_agent='my_user_agent/1.0'
        )
        self.assertEqual(geocoder.headers['User-Agent'], 'my_user_agent/1.0')

This should help to prevent quite some blockings`and bans from Nominatim, as now they are able to distinguish bad requests from good request based on the user agent headers and only block those, which have e.g a standard user agent like "geopy/1.9.1" or even "Python-urllib/2.7" and spare those requests identifying with a specific user agent like "GoodApp/0.1" as long as all "GoodApp/0.1" requests respect the Nominatim terms of usage.

I also added tests for it and thereby fixed the tests for ignfrance which were not able to pass tests without api key credentials even they should be skipped in that case.

ijl · 2015-03-29T20:08:00Z

I've cherry-picked 8b38ebb for the moment, thanks.

ijl · 2015-03-29T20:08:22Z

geopy/util.py

@@ -80,3 +80,10 @@ def decode_page(page):
        else: # requests?
            encoding = page.headers.get("charset") or "utf-8"
            return str(page.content, encoding=encoding)
+
+
+def get_version():


get_version() should be replaced with use of a constant, to me.

You mean really an arbitrary constant like "1.0" or the version number of geopy? In my opinion for traceability and debugging it makes sense that it is the actual geopy package version? But in this case a constant would have to be held in sync manually...

Just that it can be "geopy/%s" % geopy.version.GEOPY_VERSION or the like, determined at interpreter start. No need for a function call.

ijl · 2015-03-29T20:11:41Z

@sebastianneubauer it certainly makes sense, and thanks for working on it. Some comments. One significant thing is that every geocoder would need to accept and pass down a user_agent parameter.

sebastianneubauer · 2015-03-29T20:31:05Z

As it it implemented now, if a geocoder has not implemented the acceptance and passing of a custom http header, the default value "geopy/VERSION" which is still much better than the urllib default value. As said, at the moment I implemented it only for the Nominatim geocoder, as it was requested by them. Of course it would make sense also for all or many other geocoders to implement this feature, but I fear, I certainly would need help for this...Until then, I believe it improves also with this current implementation and maybe one could write an issue to implement this in other geocoders later on?

ijl · 2015-03-29T20:41:04Z

Ok, I'll add it to other geocoders, then.

sebastianneubauer · 2015-04-08T20:38:36Z

Ok, I think I managed to incorporate the suggestions, the default value of the user agent is now a constant and the circular import is removed...

sebastianneubauer · 2015-04-08T22:27:54Z

Now added customizable user agents as optional parameter to all geocoders!
From my side this is finished for now...

ijl · 2015-04-10T14:24:21Z

@sebastianneubauer ok thanks, I'll merge later.

ijl · 2015-04-10T22:29:58Z

geopy/geocoders/osm.py

@@ -156,8 +156,7 @@ def geocode(
            params = {'q': self.format_string % query}

        params.update({
-            # `viewbox` apparently replaces `view_box`
-            'viewbox': self.view_box,
+            'view_box': self.view_box,


Bad merge, I'll fix.

ijl · 2015-06-09T23:03:21Z

@sebastianneubauer sorry, this is stalled because it breaks Yahoo Placefinder, which uses requests. Would you be willing to fix that?

sebastianneubauer · 2015-06-11T19:19:37Z

Oh, ok, but why are all tests passing? Would be much easier for me to have a breaking test for this bug....I will have a look....

sebastianneubauer · 2015-07-01T13:30:41Z

@ijl : sorry, could you elaborate a bit further, what does "break" mean in detail. How can I reproduce this? Would it be an option to remove the user agent option from yahoo placefinder?

pnorman · 2015-07-09T18:28:31Z

My application that uses geopy was blocked by Nominatim as geopy uses the standard user agent header for requests provided by urllib (Python-urllib/2.7). Nominatim requires to add an identifying user agent header per application in order to be able to track and block requests if needed.

Just to note that the Nominatim usage policy requires a valid HTTP Referer or User-Agent identifying the application and stock User-Agents as set by http libraries are explicitly not okay.

sebastianneubauer · 2015-08-31T12:30:01Z

@ijl : what is the status here? this library violates the Nominatim usage policy without this PR! So for us it is really urgent to have this included!

sebastianneubauer · 2015-08-31T12:46:31Z

@ijl : Now I see, apparently I removed accidentally some code in a bad merge, I'll fix that...

…le for Nominatim

sebastianneubauer · 2015-08-31T14:18:09Z

ok, so I updated it and fixed the bad merges...still any open issues?

ijl · 2015-08-31T14:24:25Z

@sebastianneubauer, Placefinder needed a fix in Base, but I added that. I'll release in 1.11.0 shortly. Thanks.

ijl · 2015-08-31T14:31:21Z

@sebastianneubauer on PyPI.

sebastianneubauer · 2015-09-01T08:19:51Z

wohooo, thanks!!

sebastianneubauer mentioned this pull request Mar 24, 2015

Add customized http request User-Agent header #117

Closed

ijl reviewed Mar 29, 2015
View reviewed changes

ijl reviewed Apr 10, 2015
View reviewed changes

sebastianneubauer added 11 commits August 31, 2015 15:09

added fixed user agent to headers for every geocoder call, configurab…

e387ae7

…le for Nominatim

added tests for custom and default user agents

2ad1d99

removed caveman debugging print artifact ;-)

2bb6d4b

moved __version__ to own module in order to avoid circular imports

73799f8

refactored the default user agent into a constant

03ca0f1

added customizable user agent to arcgis

4a3de16

added customizable user agent to baidu

ba997dd

added customizable user agent to bing

7301eb3

added customizable user agent to databc

128a7f5

added customizable user agent to dot_us

eee555a

added customizable user agent to geocodefarm

ef95e07

sebastianneubauer and others added 12 commits August 31, 2015 15:15

added customizable user agent to geonames

5afdb7a

added customizable user agent to googleV3

538f67a

added customizable user agent to ignfrance

e091bab

added customizable user agent to navidata

607bc1b

added customizable user agent to opencage

5ade117

added customizable user agent to openmapquest

568b907

added customizable user agent to placefinder

d701f4c

added customizable user agent to smartystreets

8308372

added customizable user agent to what3words

3f69e81

added customizable user agent to yandex

020887a

a pep improvement mostly a dummy commit for a travis retriggering...

e0dcb74

fixed some bad merges

71e4819

sebastianneubauer force-pushed the master branch from 14a343c to 71e4819 Compare August 31, 2015 13:38

ijl closed this Aug 31, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added possibility to add a custom http request user agent header #115

Added possibility to add a custom http request user agent header #115

sebastianneubauer commented Mar 21, 2015

ijl commented Mar 29, 2015

ijl Mar 29, 2015

sebastianneubauer Mar 29, 2015

ijl Mar 29, 2015

ijl commented Mar 29, 2015

sebastianneubauer commented Mar 29, 2015

ijl commented Mar 29, 2015

sebastianneubauer commented Apr 8, 2015

sebastianneubauer commented Apr 8, 2015

ijl commented Apr 10, 2015

ijl Apr 10, 2015

ijl commented Jun 9, 2015

sebastianneubauer commented Jun 11, 2015

sebastianneubauer commented Jul 1, 2015

pnorman commented Jul 9, 2015

sebastianneubauer commented Aug 31, 2015

sebastianneubauer commented Aug 31, 2015

sebastianneubauer commented Aug 31, 2015

ijl commented Aug 31, 2015

ijl commented Aug 31, 2015

sebastianneubauer commented Sep 1, 2015

Added possibility to add a custom http request user agent header #115

Added possibility to add a custom http request user agent header #115

Conversation

sebastianneubauer commented Mar 21, 2015

ijl commented Mar 29, 2015

ijl Mar 29, 2015

Choose a reason for hiding this comment

sebastianneubauer Mar 29, 2015

Choose a reason for hiding this comment

ijl Mar 29, 2015

Choose a reason for hiding this comment

ijl commented Mar 29, 2015

sebastianneubauer commented Mar 29, 2015

ijl commented Mar 29, 2015

sebastianneubauer commented Apr 8, 2015

sebastianneubauer commented Apr 8, 2015

ijl commented Apr 10, 2015

ijl Apr 10, 2015

Choose a reason for hiding this comment

ijl commented Jun 9, 2015

sebastianneubauer commented Jun 11, 2015

sebastianneubauer commented Jul 1, 2015

pnorman commented Jul 9, 2015

sebastianneubauer commented Aug 31, 2015

sebastianneubauer commented Aug 31, 2015

sebastianneubauer commented Aug 31, 2015

ijl commented Aug 31, 2015

ijl commented Aug 31, 2015

sebastianneubauer commented Sep 1, 2015