Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid dates on a "_date" ending field will crash the indexing #267

Merged
merged 4 commits into from Mar 13, 2013
Merged

Invalid dates on a "_date" ending field will crash the indexing #267

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9cc1aee
[#267] Fixes issue where incorrectly formated dates broke the indexing
tobes Mar 11, 2013
4a9d9ed
[#267] Add dateutils requirement
tobes Mar 11, 2013
39b059d
[#267] Stop new error related to organisation
tobes Mar 11, 2013
b928adc
[#267] change dependency
tobes Mar 12, 2013
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
19 changes: 17 additions & 2 deletions ckan/lib/search/index.py
View file
Open in desktop
Expand Up @@ -3,6 +3,7 @@
import logging
import collections
import json
from dateutil.parser import parse

import re

Expand Down Expand Up @@ -156,7 +157,7 @@ def index_package(self, pkg_dict, defer_commit=False):

# if there is an owner_org we want to add this to groups for index
# purposes
if pkg_dict['owner_org']:
if pkg_dict.get('organization'):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this line good to go to 2.0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sure it is but I will double check on 2.0 branch with these commits added and ping you

pkg_dict['groups'].append(pkg_dict['organization']['name'])


Expand Down Expand Up @@ -192,7 +193,21 @@ def index_package(self, pkg_dict, defer_commit=False):
# Save dataset type
pkg_dict['dataset_type'] = pkg_dict['type']

pkg_dict = dict([(k.encode('ascii', 'ignore'), v) for (k, v) in pkg_dict.items()])
# clean the dict fixing keys and dates
# FIXME where are we getting these dirty keys from? can we not just
# fix them in the correct place or is this something that always will
# be needed? For my data not changing the keys seems to not cause a
# problem.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tobes let's not pollute the code with long FIXME comments like this one. Ping the list, open an issue or better yet comment on source on GitHub. 👮
I'm not sure where would we get the dirty keys from to be honest.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I sort of think that adding FIXMEs is a good thing to do. Adding this as an issue on github seems silly as this is unlikely to ever get fixed unless someone is actually cleaning up the code around it.

maybe we should decide on this sort of thing as a team. In this particular instance this looks like we are doing unneeded work key = key.encode('ascii', 'ignore') does nothing at all on the data I have. Maybe it is needed but I cannot see where this happens. This just looks like a dirty fix that was made at some point but there is no comment explaining why we are doing this, and it just adds complexity and slowness to indexing.

Personally I think adding FIXMEs is good as it will eventually lead to better code I hope.

new_dict = {}
for key, value in pkg_dict.items():
key = key.encode('ascii', 'ignore')
if key.endswith('_date'):
try:
value = parse(value).isoformat() + 'Z'
except ValueError:
continue
new_dict[key] = value
pkg_dict = new_dict

for k in ('title', 'notes', 'title_string'):
if k in pkg_dict and pkg_dict[k]:
Expand Down
1 change: 1 addition & 0 deletions pip-requirements.txt
View file
Open in desktop
Expand Up @@ -29,3 +29,4 @@ Jinja2==2.6
fanstatic==0.12
requests==1.1.0
WebTest==1.4.3
dateutils==0.6.5
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure you want dateutils and not the plain python-dateutil>=1.5.0,<2.0.0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the difference? All I want is parse()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand it, dateutils is a library that uses python-dateutil. If you only want the parse, use python-dateutil

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I'm lost here what would you do to make this how you want?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:-) Just replace the line 32 with python-dateutil>=1.5.0,<2.0.0. It will reduce the number of dependencies since only python-dateutil is required but not dateutils.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool let's see what travis thinks