Script works, but returns low number of listings

[tjukanovt]

First of all, thanks for the great project!

I have been able to get some data, but I'm just wondering what's the main issue as I'm only getting a few listings from Helsinki. When I am running python airbnb.py -sb 1 I get the following result

INFO Retrieved logged progress: None, None guests, price None-None
INFO quadtree node []
INFO median node []
INFO Bounding box: [60.297839, 25.254485, 59.922489, 24.782876]
INFO ======================================================================
INFO Survey 1, for helsinki
INFO Searching by bounding box, max_zoom=10
INFO ----------------------------------------------------------------------
INFO Rectangle calculated: [60.297839, 25.254485, 59.922489, 24.782876]
INFO Searching rectangle: zoom factor = 0, node = []
INFO Page 01 returned 00 listings
INFO Results: 1 pages, 0 new rooms
INFO Finishing survey 1, for helsinki

I ran this manually a few times and it mostly just returns nothing like above and at most something like 40 results. This seems a bit odd, as the data should contain thousands of listings. Bounding box is correct, as I am getting correct data in PostGIS, but not much.

Did I just read the docs badly, or is there something that's not working at AirBNB side?

[tanjavollmer]

Hey tjukanovt,

I got the same issue, running a search for Munich. However I also got the information that it's due to a connection error. Yet I don't know how to solve it. But I will research myself and also here are very helpful people around :-) Good luck!

INFO Retrieved logged progress: None, None guests, price None-None
INFO quadtree node []
INFO median node []
INFO Bounding box: [48.24822, 11.722875, 48.061602, 11.360796]
INFO ======================================================================
INFO Survey 2, for Muenchen
INFO Searching by bounding box, max_zoom=6
INFO ----------------------------------------------------------------------
INFO Rectangle calculated: [48.24822, 11.722875, 48.061602, 11.360796]
INFO Searching rectangle: zoom factor = 0, node = []
WARNING Network request exception 0: connectionError
WARNING Network request exception 1: connectionError
INFO Page 01 returned 00 listings
INFO Results: 1 pages, 0 new rooms
INFO Finishing survey 2, for Muenchen

[tomslee]

Hi tjukanovt and tanjavollmer: I think I broke something for the case where there is no set of proxy addresses in use. Unfortunately I doubt that I will be able to look at the code again for a week or so. If I get time tomorrow I will see if I can put a fix out, but it seems unlikely right now...

[tjukanovt]

Hi! Thanks both for the replies. I will also try to sort this out when I have a moment. Tom you suggest that a workaround would be to just use a proxy address?

[tjukanovt]

Update: I added two proxy addresses to the config, but the results are even more weird. I run the same command as above, with the same bounding box, but get ten results from around the world and none of them are in Helsinki.

[tanjavollmer]

Hey tjukanovt.
I got the same issue here. When I run the survey without a proxy, I get no results. When I run the survey with a Proxy (-list) I get either ConnectionErrors or a mere result of one page with about 10 listings max. from everywhere but Munich.

Any ideas out there to solve this? Thanks :-)

[tomslee]

At work right now, but I'll respond this evening (Canada time).

[tomslee]

I've posted a new version, which needs a bit more user work to get started. I'm using an API that returns JSON instead of HTML and it seems more consistent (I cannot get the HTML code working at all), but you need to collect a key from the site and enter it in your config file -- there are instructions in example.config. See the README for more.

There may still be annoying bugs: as data collection is time sensitive I thought I would post the new version, but it is not well tested.

Not sure how long I'm going to continue this project...