IndexError with bounding box search

[tjukanovt]

Hi,

Thanks a lot for the latest fixes. Now the bounding box search seems to be working pretty well. Only thing is that when running the survey, I get the following error after a while:

2018-05-01 15:42:24,933 INFO Retrieved logged progress: quadtree node [[0, 0]] 2018-05-01 15:42:24,933 INFO median node [[60.17343, 24.94219]] 2018-05-01 15:42:24,933 INFO Bounding box: [60.297839, 25.254485, 59.922489, 24.782876] 2018-05-01 15:42:24,933 INFO ====================================================================== 2018-05-01 15:42:24,933 INFO Survey 1, for helsinki 2018-05-01 15:42:24,933 INFO Searching by bounding box, max_zoom=8 2018-05-01 15:42:24,933 INFO ---------------------------------------------------------------------- 2018-05-01 15:42:24,933 INFO Rectangle calculated: [60.29784, 25.25448, 60.11016, 25.01868] 2018-05-01 15:42:24,933 INFO Searching rectangle: zoom factor = 1, node = [[0, 0]] 2018-05-01 15:42:29,585 INFO Page 01 returned 18 listings 2018-05-01 15:42:34,308 INFO Page 02 returned 18 listings 2018-05-01 15:42:38,998 INFO Page 03 returned 18 listings 2018-05-01 15:42:41,766 INFO Page 04 returned 18 listings 2018-05-01 15:42:47,011 INFO Page 05 returned 18 listings 2018-05-01 15:42:53,118 INFO Page 06 returned 18 listings 2018-05-01 15:42:57,597 INFO Page 07 returned 18 listings 2018-05-01 15:43:01,262 INFO Page 08 returned 18 listings 2018-05-01 15:43:05,579 INFO Page 09 returned 18 listings 2018-05-01 15:43:07,565 INFO Page 10 returned 18 listings 2018-05-01 15:43:07,565 INFO Results: 10 pages, 0 new rooms 2018-05-01 15:43:07,580 ERROR Error in recurse_quadtree Traceback (most recent call last): File ",python\airbnb-data-collection-master\airbnb_survey.py", line 421, in recurse_quadtree if self.subtree_previously_completed(quadtree_node): File ",python\airbnb-data-collection-master\airbnb_survey.py", line 787, in subtree_previously_completed for j in range(0, 2) File ",python\airbnb-data-collection-master\airbnb_survey.py", line 788, in <genexpr> for i in range(0, len(quadtree_node))) IndexError: list index out of range 2018-05-01 15:43:07,580 ERROR Error in recurse_quadtree Traceback (most recent call last): File ",python\airbnb-data-collection-master\airbnb_survey.py", line 454, in recurse_quadtree self.recurse_quadtree(quadtree_node, median_node, flag) File ",python\airbnb-data-collection-master\airbnb_survey.py", line 421, in recurse_quadtree if self.subtree_previously_completed(quadtree_node): File ",python\airbnb-data-collection-master\airbnb_survey.py", line 787, in subtree_previously_completed for j in range(0, 2) File ",python\airbnb-data-collection-master\airbnb_survey.py", line 788, in <genexpr> for i in range(0, len(quadtree_node))) IndexError: list index out of range 2018-05-01 15:43:07,580 ERROR Error in recurse_quadtree Traceback (most recent call last): File ",python\airbnb-data-collection-master\airbnb_survey.py", line 454, in recurse_quadtree self.recurse_quadtree(quadtree_node, median_node, flag) File ",python\airbnb-data-collection-master\airbnb_survey.py", line 454, in recurse_quadtree self.recurse_quadtree(quadtree_node, median_node, flag) File ",python\airbnb-data-collection-master\airbnb_survey.py", line 421, in recurse_quadtree if self.subtree_previously_completed(quadtree_node): File ",python\airbnb-data-collection-master\airbnb_survey.py", line 787, in subtree_previously_completed for j in range(0, 2) File ",python\airbnb-data-collection-master\airbnb_survey.py", line 788, in <genexpr> for i in range(0, len(quadtree_node))) IndexError: list index out of range 2018-05-01 15:43:07,580 ERROR Error Traceback (most recent call last): File ",python\airbnb-data-collection-master\airbnb_survey.py", line 395, in search self.recurse_quadtree(quadtree_node, median_node, flag) File ",python\airbnb-data-collection-master\airbnb_survey.py", line 454, in recurse_quadtree self.recurse_quadtree(quadtree_node, median_node, flag) File ",python\airbnb-data-collection-master\airbnb_survey.py", line 454, in recurse_quadtree self.recurse_quadtree(quadtree_node, median_node, flag) File ",python\airbnb-data-collection-master\airbnb_survey.py", line 421, in recurse_quadtree if self.subtree_previously_completed(quadtree_node): File ",python\airbnb-data-collection-master\airbnb_survey.py", line 787, in subtree_previously_completed for j in range(0, 2) File ",python\airbnb-data-collection-master\airbnb_survey.py", line 788, in <genexpr> for i in range(0, len(quadtree_node))) IndexError: list index out of range

So this is probably just a bug in the code this time and not dependent on the AirBNB site changes? I will also look in to this myself if I could fix it.

[feferri]

Does someone is getting this error?

INFO    Bounding box: [38.795854, -9.090571, 38.691399, -9.229836]
INFO    ======================================================================
INFO    Survey 17, for Lisbon
INFO    Searching by bounding box, max_zoom=8
INFO    ----------------------------------------------------------------------
INFO    Rectangle calculated: [38.795854, -9.090571, 38.691399, -9.229836]
INFO    Searching rectangle: zoom factor = 0, node = []
ERROR   Exception in get_search_page_info_rectangle
Traceback (most recent call last):
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_listing.py", line 138, in save
    self.__insert()
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_listing.py", line 291, in __insert
    cur.execute(sql, insert_args)
psycopg2.ProgrammingError: column "coworker_hosted" of relation "room" does not exist
LINE 7:                     coworker_hosted, extra_host_languages, n...
                            ^


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_survey.py", line 679, in search_node
    if listing.save(self.config.FLAGS_INSERT_NO_REPLACE):
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_listing.py", line 148, in save
    self.config.connection.conn.rollback()
AttributeError: 'psycopg2.extensions.connection' object has no attribute 'conn'
ERROR   Error in recurse_quadtree
Traceback (most recent call last):
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_listing.py", line 138, in save
    self.__insert()
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_listing.py", line 291, in __insert
    cur.execute(sql, insert_args)
psycopg2.ProgrammingError: column "coworker_hosted" of relation "room" does not exist
LINE 7:                     coworker_hosted, extra_host_languages, n...
                            ^


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_survey.py", line 434, in recurse_quadtree
    (zoomable, median_leaf) = self.search_node(quadtree_node, median_node, flag)
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_survey.py", line 679, in search_node
    if listing.save(self.config.FLAGS_INSERT_NO_REPLACE):
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_listing.py", line 148, in save
    self.config.connection.conn.rollback()
AttributeError: 'psycopg2.extensions.connection' object has no attribute 'conn'
ERROR   Error
Traceback (most recent call last):
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_listing.py", line 138, in save
    self.__insert()
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_listing.py", line 291, in __insert
    cur.execute(sql, insert_args)
psycopg2.ProgrammingError: column "coworker_hosted" of relation "room" does not exist
LINE 7:                     coworker_hosted, extra_host_languages, n...
                            ^


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_survey.py", line 395, in search
    self.recurse_quadtree(quadtree_node, median_node, flag)
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_survey.py", line 434, in recurse_quadtree
    (zoomable, median_leaf) = self.search_node(quadtree_node, median_node, flag)
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_survey.py", line 679, in search_node
    if listing.save(self.config.FLAGS_INSERT_NO_REPLACE):
  File "C:\Users\fe_fe\OneDrive\Documentos\tomslee airbnb\airbnb_listing.py", line 148, in save
    self.config.connection.conn.rollback()
AttributeError: 'psycopg2.extensions.connection' object has no attribute 'conn'

[tomslee]

@tjukanovt : this problem should now be fixed.

@feferri: it seems that you have an old version of the SQL scripts. I don't have an automatic way to update that, but take a look in postgres/schema_current.sql for what it should be. Here is the table definition for the table "room" which holds most of the listings:

CREATE TABLE public.room
(
  room_id integer NOT NULL,
  host_id integer,
  room_type character varying(255),
  country character varying(255),
  city character varying(255),
  neighborhood character varying(255),
  address character varying(1023),
  reviews integer,
  overall_satisfaction double precision,
  accommodates integer,
  bedrooms numeric(5,2),
  bathrooms numeric(5,2),
  price double precision,
  deleted integer,
  minstay integer,
  last_modified timestamp without time zone DEFAULT now(),
  latitude numeric(30,6),
  longitude numeric(30,6),
  survey_id integer NOT NULL DEFAULT 999999,
  location geometry,
  coworker_hosted integer,
  extra_host_languages character varying(255),
  name character varying(255),
  property_type character varying(255),
  currency character varying(20),
  rate_type character varying(20),
  CONSTRAINT room_pkey PRIMARY KEY (room_id, survey_id)
)
WITH (
  OIDS=FALSE
);

[luisencalada]

Hello to all! I am getting this problem when searching by bbox... Warning HTTP Status 400 from web site: IP address blocked.Waiting 1.0 minutes... It seems my university IP is blocked... Do you have any recommendation to overpass this issue? When searching by zipcode or neighborhood, it finishes the process but no data is on DB.. Thanks in advance! I am looking for data within Lisbon boundaries