Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Filter on maps pages not working as expected #74

Open
rtburg opened this Issue · 5 comments

4 participants

@rtburg
Owner

For example: http://columbusco-staging.openrural.org/maps/corporations/filter/?end_date=2012-07-05&start_date=2012-03-15

I can't figure out how the map filters the items below it. The default start date is 03/15/2012, yet it displays news items from Feb and early March as well.

In fact, the news items displayed on the map or in the list below don't change until you get to a 6/20 start date, when which all news items disappear. ... until you zoom out on the map, when they start to appear again.

The simplest thing here in the short term my be simply to de-link this page and continue to work on it post-launch.

(Also, there are no breadcrumbs on this page... which may be more of a feature than a bug.)

@sarahdooley sarahdooley was assigned
@sarahdooley

Let's just de-link for now, unless there's a clear solution...

@rtburg
Owner

I'm ok with that, Sarah. Let's table it and flag for later.

@sarahdooley

Based on the team's research, it looks like this is a bug that only turns up when the most recent item from the scraper is more than 7 days old.
Post about this to see whether other OpenBlock users have encountered / solved it....but otherwise don't spend a lot of time on it for now.

@rebecca-caktus
Collaborator

@rtburg With more investigation, I think that I've discovered the actual issue at hand:

All NewsItem objects have two dates:

  • item_date: "Date (without time) this Item occurred, or failing that, the date of publication on the original source site; default today." For example, "McDonald's was inspected on June 01, 2012."
  • pub_date: "Date/time this Item was added to the OpenBlock site; default now." For example, "McDonald's new inspection showed up when restaurants were scraped on July 03, 2012."

We see weird behavior on the filter because it uses pub_date to drill down map items. Because we're still developing, most of the pub_dates are quite recent even if the item_dates aren't. The good part of this is that users always see the latest things added to the site- e.g., even though restaurant inspections will be on a month's lag, the users will see them when they're added. The bad part of this is that users won't be able to search for NewsItems on a particular date- e.g., someone can't search for the property transactions that occurred in March.

Interestingly, this is not the way that the homepage map works. The homepage map shows NewsItems whose item_date is in the last 14 days (as set in the settings as DEFAULT_DAYS).

@kmtracey
Collaborator

Rebecca found an OpenBlock ticket that possibly explains why pub_date is being used in one place and item_date in another. The last comment on that ticket says "We decided to leave pub_date used in the REST API and feeds, since for those use cases, it actually matters when the data was added to our system." I'm guesing the REST API referred to here is the one that the bigmap is using to fetch items. At any rate ultimately it seems that the map code is using this method:

https://github.com/openplans/openblock/blob/master/ebpub/ebpub/openblockapi/itemquery.py#L113

to filter by the start/end dates in the queries, and that's using pub_date rather than item_date. Further pub_date is a datetime field rather than a date field so the end date winds up being exclusive (excepting things published at exactly midnight on the specified end day) rather than inclusive, since times are not included in the filter and defaulting to 00:00:00. Possibly we should raise this issue on the openblock list...I am not sure the decision to use pub_date in this api and not allow for using item_date is correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.