[MRG+1] Added JmesSelect #1016

SudShekhar · 2015-01-19T09:29:39Z

Created the first implementation of Json processor, any comments/edits are welcome.

I am currently using the jmespath (https://github.com/boto/jmespath) module to search for paths in the given list of values. I have added some sample test cases too.
The processor will return the list of values unchanged in case the jmespath module isn't installed.
I wasn't sure how to show a warning message in such a case and thus, have used the python print statement for now.

nramirezuy · 2015-01-19T16:12:30Z

scrapy/contrib/loader/processor.py

+        try:
+            import jmespath,ast,json
+        except:
+            print "JsonProcessor module requires the jmespath library to function. No processing will happen in its abscence"


When I said required, I meant for the Scrapy library. So it should raise the exception on the import of this Processor.

nramirezuy · 2015-01-19T16:13:29Z

scrapy/contrib/loader/processor.py

+        except:
+            print "JsonProcessor module requires the jmespath library to function. No processing will happen in its abscence"
+            return values
+        compiledPath = jmespath.compile(self.json_path)    


This compilation should happen on __init__, it will improve performance.

SudShekhar · 2015-01-22T17:04:37Z

Regarding the storage of ast module as self.astModule in the JsonProcessor class, I looked up the following document : https://wiki.python.org/moin/PythonSpeed/PerformanceTips (Import statement overhead topic). It seems that the current method should be acceptable. Kindly let me know if there are any other issues to be handled.

nramirezuy · 2015-01-23T14:41:36Z

scrapy/contrib/loader/processor.py

+        try:
+            import jmespath
+            import ast
+            self.astModule = ast


Just save the function you are going to use. self.ast_literal_eval = ast.literal_eval.
Also fix naming, we use camel case just for classes.

nramirezuy · 2015-01-23T14:42:11Z

scrapy/contrib/loader/processor.py

+            raise ImportError("You need to have jmespath(https://github.com/boto/jmespath)"
+                              + " and AST modules before being able to use this processor")
+
+    def __call__(self, currentString):


Rename currentString to value

nramirezuy · 2015-01-23T14:43:13Z

scrapy/contrib/loader/processor.py

+                              + " and AST modules before being able to use this processor")
+
+    def __call__(self, currentString):
+        if currentString != "":  # Empty strings will return None


Be prepared for None values. I think if value: is enough.

nramirezuy · 2015-01-23T14:43:34Z

scrapy/contrib/loader/processor.py

+                self.astModule.literal_eval(currentString)
+                )
+            return jsonValue
+        return None


return None line can be removed

nramirezuy · 2015-01-23T14:44:17Z

scrapy/contrib/loader/processor.py

@@ -63,6 +63,31 @@ def __call__(self, values):
        return values


+class JsonProcessor(object):
+    """


Declare that it expect a string in __call__

nramirezuy · 2015-01-23T14:47:21Z

We are also missing documentation.

nramirezuy · 2015-01-30T17:03:11Z

tests/test_contrib_loader.py

+                    self.assertEqual(test, expected,
+                                     msg='test "{}" got {} expected {}'.format(l, test, expected)
+                                     )
+            except:  # in case jmespath isn't installed


If jmespath isn't installed just let it fail.
You can add dependencies for tests: https://github.com/scrapy/scrapy/blob/master/tests/requirements.txt

nramirezuy · 2015-01-30T17:08:45Z

tests/test_contrib_loader.py

+            except:  # in case jmespath isn't installed
+                pass
+
+        def test_errors(self):


Not sure what this test.

nramirezuy · 2015-01-30T17:09:29Z

tests/test_contrib_loader.py

+
+        def test_equals(self):
+            try:
+                import jmespath


Why are you importing jmespath ?

nramirezuy · 2015-02-02T16:31:40Z

scrapy/contrib/loader/processor.py

+        self.json_path = json_path
+        try:
+            import jmespath
+            import ast


ast is from the standard library so we can import it at the top. So storing literal_eval in an attribute won't be needed.

nramirezuy · 2015-02-02T16:49:00Z

docs/topics/loaders.rst

@@ -675,3 +675,18 @@ Here is a list of all built-in processors:
    constructor keyword arguments are used as default context values. See
    :class:`Compose` processor for more info.

+.. class:: JsonProcessor(json_path)


Let's rename it to JmesProcessor.

nramirezuy · 2015-02-02T17:28:24Z

@kmike Can you review this? I'm looking for py3 and documentation feedback.

dangra · 2015-02-02T17:28:50Z

+1 LGTM. It needs to be rebased into a single commit before merging.

kmike · 2015-02-02T17:44:03Z

docs/topics/loaders.rst

+    Queries the value using the json path provided to the constructor and returns the output. 
+    Requires jmespath (https://github.com/boto/jmespath) to run.
+
+    This processor takes only one string at a time and will return a python string/dictionary/None as answer


Can it return a list? A number? A boolean?

What is a "python string"? Does this processor return bytes or unicode?

kmike · 2015-02-02T17:44:15Z

docs/topics/loaders.rst

+.. class:: JmesProcessor(json_path)
+
+    Queries the value using the json path provided to the constructor and returns the output. 
+    Requires jmespath (https://github.com/boto/jmespath) to run.


the new link is https://github.com/jmespath/jmespath

kmike · 2015-02-02T17:46:08Z

scrapy/contrib/loader/processor.py

+        """
+        if value:  # Empty strings will return None
+            return_value = self.compiled_path.search(
+                literal_eval(value)


Why is literal_eval needed?

I mean, why not use json?

kmike · 2015-02-02T17:47:56Z

scrapy/contrib/loader/processor.py

+    def __call__(self, value):
+        """Query value for the jmespath query and return answer
+        Input : string
+        Output : string / dict / None


What about other Python types?
Could you please also use Sphinx-compatible docstrings, like

:param str value: a string with JSON data to extract from

kmike · 2015-02-02T17:51:23Z

scrapy/contrib/loader/processor.py

+        Query the input string for the jmespath (given at instantiation), and return the answer
+        Requires : jmespath(https://github.com/boto/jmespath), ast
+        Note: JmesProcessor accepts only a single string at a time and returns string/dict/None based on the jmespath query.
+    """


Could you please fix the indentation?

According to PEP8 docstrings should be wrapped at 72th column.

It would be nice for it to follow http://sphinx-doc.org/domains.html#info-field-lists

Thanks for your feedback. I will update the things you have mentioned.

Thanks!
The Sphinx link I posted here is not relevant, but it is relevant for __call__ docstring.

kmike · 2015-02-02T17:56:13Z

scrapy/contrib/loader/processor.py

+            self.compiled_path = jmespath.compile(self.json_path)
+        except:
+            raise ImportError("You need to have jmespath(https://github.com/boto/jmespath)"
+                              + " and AST modules before being able to use this processor")


This is bad for several reasons:

If "import jmespath" fails, it hides the original exception which can provide more information about why isn't jmespath available. See e.g. #902.

If jmespath.compile fails the code raises an obscure ImportError, hiding the original exception again.

except: shouldn't be used; it catches e.g. KeyboardInterrupt and SystemExit exceptions.

I suggest to remove try-except entirely. It will fail with "ImportError" pointing to jmespath, this is self-explanatory.

kmike · 2015-02-02T17:56:43Z

tests/test_contrib_loader.py

@@ -3,7 +3,7 @@

 from scrapy.contrib.loader import ItemLoader
 from scrapy.contrib.loader.processor import Join, Identity, TakeFirst, \
-    Compose, MapCompose
+    Compose, MapCompose,JmesProcessor


a nitpick - space is missing after a comma

nramirezuy · 2015-02-03T22:32:50Z

class JmesProcessor(object):
    def __init__(self, path, process=None):
        self.process = process or self._process

    def _process(self, data):
         if isinstance(data, basestring):
              return json.loads(data)
         return data

But I guess doing the json.loads outside is not that hard.

# Load JSON and process
Compose(json.loads, JmesProcessor('foo'))
# Process several JSON objects: '[{"foo":"bar"}, {"baz":"tar"}]'
Compose(json.loads, MapCompose(JmesProcessor('foo')))
# Just process
JmesProcessor('foo')
# Chained processors with load
Compose(json.loads, JmesProcessor('foo'), JmesProcessor('bar'))

I like more the second option.
@kmike @SudShekhar What do you think?

EDIT: We should also add examples to the doc.

kmike · 2015-02-03T22:45:02Z

I like not doing json.loads in the JmesProcessor, and the Compose / MapCompose examples are good.

@nramirezuy you should really create a few nice images/diagrams which show how Compose and MapCompose work :) What are the inputs, what are the outputs, how they can be used together. And maybe we should rename the processors to verbs - instead of JmesProcessor write SelectJmes - Compose(json.loads, SelectJmes('foo')) reads very well.

kmike · 2015-02-03T22:50:49Z

@nramirezuy a realted question: is it possible to use the processors without the item loaders?

get_foo = Compose(json.loads, SelectJmes('foo'))
# ...
value = get_foo(response.body_as_unicode())

is rather nice. If it works then I think it worths documenting. An maybe splitting this micro-framework from the item loaders, if it is not only about item loaders.

nramirezuy · 2015-02-04T14:18:16Z

Well every Processor works outside ItemLoaders; and that is completely valid.
I'm fine with the processor rename there are no processors named Processor 👅

SudShekhar · 2015-02-04T16:42:31Z

@nramirezuy : The second options looks much better and it gives users more control. So, should I remove the json.loads from inside the processor? Regarding the documentation, the examples you have given look pretty comprehensive to me 😄 .I will add them and some others to the list.
+1 to the renaming too.

EDIT: In the documentation, I didn't add the chaining example because I felt that it fit better in the Compose examples list (same can be said for the processor handling a list of json strings I guess). Do let me know your views on this.

nramirezuy · 2015-02-13T15:36:41Z

docs/topics/loaders.rst

+        >>> proc({'foo':{'bar':'baz'}})
+        {'bar': 'baz'}
+	>>> import json
+        >>> procSingleJsonStr = Compose(json.loads,SelectJmes("foo"))


We just use we just use CamelCase for classes; blame pep8 👅
procSingleJsonStr -> proc_single_json_str

Space after commas please 😄

nramirezuy · 2015-02-13T15:39:42Z

docs/topics/loaders.rst

+        >>> procSingleJsonStr = Compose(json.loads,SelectJmes("foo"))
+        >>> procSingleJsonStr('{"foo":"bar"}')
+        u'bar'
+        >>> procJsonList = Compose(json.loads, MapCompose(SelectJmes('foo')))


nramirezuy · 2015-02-13T15:40:06Z

scrapy/contrib/loader/processor.py

+        :return: Element extracted according to jmespath query
+        """
+        return_value = self.compiled_path.search(value)
+        return return_value


make it one line

nramirezuy · 2015-02-13T15:41:39Z

tests/test_contrib_loader.py

@@ -579,5 +579,44 @@ def test_replace_css_re(self):
        self.assertEqual(l.get_output_value('url'), [u'scrapy.org'])


+class SelectJmesTestCase(unittest.TestCase):


Gotta test those python types. Json isn't relevant here.

nramirezuy · 2015-02-13T15:42:19Z

docs/topics/loaders.rst

+        'bar'
+        >>> proc({'foo':{'bar':'baz'}})
+        {'bar': 'baz'}
+	>>> import json


Make it 2 blocks, little title in between Working with Json.

nramirezuy · 2015-02-13T15:43:07Z

tests/test_contrib_loader.py

+            'simple': ('foo.bar', '{"foo": {"bar": "baz"}}', "baz"),
+            'invalid': ('foo.bar.baz', '{"foo": {"bar": "baz"}}', None),
+            'top_level': ('foo', '{"foo": {"bar": "baz"}}', {"bar": "baz"}),
+            'double_vs_single_quoteString': ('foo.bar', '{"foo":{"bar":"baz"}}', "baz"),


That camel case 👅

nramirezuy · 2015-02-13T15:46:23Z

tests/test_contrib_loader.py

+                    msg='test "{}" got {} expected {}'.format(l, test, expected)
+                )
+
+        def test_dict(self):


Use the tests configuration above.

Can you please clarify what you mean by this? Json's used to load the input. Do you just want me to directly use the dict/list/simple strings to check?

Yes, use python types and forget about json, since we don't load json anymore inside the class. Also use the test_list_equals method, so it is easy to add more comparisons if needed.

nramirezuy · 2015-02-17T14:54:07Z

There is something with the markup but I don't know what it is.

/cc @kmike

SudShekhar · 2015-02-24T11:03:32Z

I created the documentation locally but was unable to figure out the error. Can you please point out the issue?

Thanks.

nramirezuy · 2015-02-24T15:06:02Z

I think you are missing :: in Working with Json: http://pasteboard.co/1mkMnIj3.png

Utilizes jmespath. Also, added tests and documentation for the same.

SudShekhar · 2015-03-08T07:10:48Z

Hi,
Is there anything else that needs to be changed in this commit? Thanks for all your feedback.
/cc @nramirezuy @kmike

kmike · 2015-03-13T21:52:58Z

I think this PR is fine, +1 to merge it.

[MRG+1] Added JmesSelect

SudShekhar · 2015-03-18T11:54:32Z

Thanks 😄

pablohoffman · 2015-03-18T17:55:26Z

Sorry I couldn't chime in before merge but, if we're gonna add jmespath as requirement (to requirements.txt) we should import it at module-level, not in the class constructor, don't you think?

kmike · 2015-03-18T17:59:35Z

@pablohoffman why does it matter? jmespath is not added to install_requires, it is optional.

Imports are very fast after the first successful import (a lookup in a dict) - by moving import to module level we won't get any speed benefits, but the exception may become less clear is jmespath is absent.

nramirezuy · 2015-03-20T15:07:57Z

@pablohoffman @kmike jmespath was added to tests/requirements.txt, not Scrapy ones.

nramirezuy reviewed Jan 19, 2015
View changes

nramirezuy mentioned this pull request Jan 21, 2015

loader.add_json? #1005

Closed

nramirezuy reviewed Jan 23, 2015
View changes

scrapy/contrib/loader/processor.py Outdated

self.astModule.literal_eval(currentString)

)

return jsonValue

return None

This comment has been minimized.

Sign in to view

nramirezuy Jan 23, 2015
Contributor

return None line can be removed

nramirezuy reviewed Jan 23, 2015
View changes

scrapy/contrib/loader/processor.py Outdated

@@ -63,6 +63,31 @@ def __call__(self, values):

return values

class JsonProcessor(object):

"""

This comment has been minimized.

Sign in to view

nramirezuy Jan 23, 2015
Contributor

Declare that it expect a string in __call__

nramirezuy reviewed Jan 30, 2015
View changes

tests/test_contrib_loader.py Outdated

except: # in case jmespath isn't installed

pass

def test_errors(self):

This comment has been minimized.

Sign in to view

nramirezuy Jan 30, 2015
Contributor

Not sure what this test.

nramirezuy reviewed Jan 30, 2015
View changes

tests/test_contrib_loader.py Outdated

def test_equals(self):

try:

import jmespath

This comment has been minimized.

Sign in to view

nramirezuy Jan 30, 2015
Contributor

Why are you importing jmespath ?

SudShekhar force-pushed the SudShekhar:jsonProcessor branch 4 times, most recently from 2bdf817 to d096792 Jan 30, 2015

nramirezuy reviewed Feb 2, 2015
View changes

SudShekhar force-pushed the SudShekhar:jsonProcessor branch 2 times, most recently from d0a1a2e to 5c64b3a Feb 2, 2015

kmike reviewed Feb 2, 2015
View changes

SudShekhar force-pushed the SudShekhar:jsonProcessor branch from bf02b31 to f975f27 Feb 4, 2015

nramirezuy reviewed Feb 13, 2015
View changes

scrapy/contrib/loader/processor.py Outdated

:return: Element extracted according to jmespath query

"""

return_value = self.compiled_path.search(value)

return return_value

This comment has been minimized.

Sign in to view

nramirezuy Feb 13, 2015
Contributor

make it one line

nramirezuy reviewed Feb 13, 2015
View changes

docs/topics/loaders.rst Outdated

'bar'

>>> proc({'foo':{'bar':'baz'}})

{'bar': 'baz'}

>>> import json

This comment has been minimized.

Sign in to view

nramirezuy Feb 13, 2015
Contributor

Make it 2 blocks, little title in between Working with Json.

nramirezuy reviewed Feb 13, 2015
View changes

SudShekhar force-pushed the SudShekhar:jsonProcessor branch from f975f27 to d430dd7 Feb 13, 2015

Added the first version of SelectJmes

Loading status checks…

839ffba

Utilizes jmespath. Also, added tests and documentation for the same.

SudShekhar force-pushed the SudShekhar:jsonProcessor branch from d430dd7 to 839ffba Feb 24, 2015

kmike changed the title ~~Added a json processor~~ [MRG+1] Added a json processor Mar 13, 2015

nramirezuy changed the title ~~[MRG+1] Added a json processor~~ [MRG+1] Added JmesSelect Mar 18, 2015

nramirezuy added a commit that referenced this pull request Mar 18, 2015

Merge pull request #1016 from SudShekhar/jsonProcessor

Loading status checks…

ee82fe0

[MRG+1] Added JmesSelect

nramirezuy merged commit ee82fe0 into scrapy:master Mar 18, 2015
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

SudShekhar deleted the SudShekhar:jsonProcessor branch Mar 24, 2015

		@@ -579,5 +579,44 @@ def test_replace_css_re(self):
		self.assertEqual(l.get_output_value('url'), [u'scrapy.org'])


		class SelectJmesTestCase(unittest.TestCase):

scrapy / scrapy

[MRG+1] Added JmesSelect #1016

[MRG+1] Added JmesSelect #1016

Conversation

SudShekhar commented Jan 19, 2015

This comment has been minimized.

nramirezuy Jan 19, 2015 Contributor

This comment has been minimized.

nramirezuy Jan 19, 2015 Contributor

SudShekhar commented Jan 22, 2015

This comment has been minimized.

nramirezuy Jan 23, 2015 Contributor

This comment has been minimized.

nramirezuy Jan 23, 2015 Contributor

This comment has been minimized.

nramirezuy Jan 23, 2015 Contributor

This comment has been minimized.

nramirezuy Jan 23, 2015 Contributor

This comment has been minimized.

nramirezuy Jan 23, 2015 Contributor

nramirezuy commented Jan 23, 2015

This comment has been minimized.

nramirezuy Jan 30, 2015 Contributor

This comment has been minimized.

nramirezuy Jan 30, 2015 Contributor

This comment has been minimized.

nramirezuy Jan 30, 2015 Contributor

This comment has been minimized.

nramirezuy Feb 2, 2015 Contributor

This comment has been minimized.

nramirezuy Feb 2, 2015 Contributor

nramirezuy commented Feb 2, 2015

dangra commented Feb 2, 2015

This comment has been minimized.

kmike Feb 2, 2015 Member

This comment has been minimized.

kmike Feb 2, 2015 Member

This comment has been minimized.

kmike Feb 2, 2015 Member

This comment has been minimized.

kmike Feb 2, 2015 Member

This comment has been minimized.

kmike Feb 2, 2015 Member

This comment has been minimized.

kmike Feb 2, 2015 Member

This comment has been minimized.

SudShekhar Feb 2, 2015 Author Contributor

This comment has been minimized.

kmike Feb 2, 2015 Member

This comment has been minimized.

kmike Feb 2, 2015 Member

This comment has been minimized.

kmike Feb 2, 2015 Member

nramirezuy commented Feb 3, 2015

kmike commented Feb 3, 2015

kmike commented Feb 3, 2015

nramirezuy commented Feb 4, 2015

SudShekhar commented Feb 4, 2015

This comment has been minimized.

nramirezuy Feb 13, 2015 Contributor

This comment has been minimized.

nramirezuy Feb 13, 2015 Contributor

This comment has been minimized.

nramirezuy Feb 13, 2015 Contributor

This comment has been minimized.

nramirezuy Feb 13, 2015 Contributor

This comment has been minimized.

nramirezuy Feb 13, 2015 Contributor

This comment has been minimized.

nramirezuy Feb 13, 2015 Contributor

This comment has been minimized.

nramirezuy Feb 13, 2015 Contributor

This comment has been minimized.

SudShekhar Feb 13, 2015 Author Contributor

This comment has been minimized.

nramirezuy Feb 13, 2015 Contributor

nramirezuy commented Feb 17, 2015

SudShekhar commented Feb 24, 2015

nramirezuy commented Feb 24, 2015

SudShekhar commented Mar 8, 2015

nramirezuy Jan 19, 2015
Contributor

nramirezuy Jan 19, 2015
Contributor

nramirezuy Jan 23, 2015
Contributor

nramirezuy Jan 23, 2015
Contributor

nramirezuy Jan 23, 2015
Contributor

nramirezuy Jan 23, 2015
Contributor

nramirezuy Jan 23, 2015
Contributor

nramirezuy Jan 30, 2015
Contributor

nramirezuy Jan 30, 2015
Contributor

nramirezuy Jan 30, 2015
Contributor

nramirezuy Feb 2, 2015
Contributor

nramirezuy Feb 2, 2015
Contributor

kmike Feb 2, 2015
Member

kmike Feb 2, 2015
Member

kmike Feb 2, 2015
Member

kmike Feb 2, 2015
Member

kmike Feb 2, 2015
Member

kmike Feb 2, 2015
Member

SudShekhar Feb 2, 2015
Author Contributor

kmike Feb 2, 2015
Member

kmike Feb 2, 2015
Member

kmike Feb 2, 2015
Member

nramirezuy Feb 13, 2015
Contributor

nramirezuy Feb 13, 2015
Contributor

nramirezuy Feb 13, 2015
Contributor

nramirezuy Feb 13, 2015
Contributor

nramirezuy Feb 13, 2015
Contributor

nramirezuy Feb 13, 2015
Contributor

nramirezuy Feb 13, 2015
Contributor

SudShekhar Feb 13, 2015
Author Contributor

nramirezuy Feb 13, 2015
Contributor