New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed broken extract function #104

Merged
merged 2 commits into from Dec 6, 2013

Conversation

Projects
None yet
2 participants
@tongwang
Contributor

tongwang commented Dec 6, 2013

fixed both issues #96 and #90 and updated test solr sever from 4.1.0 to 4.6.0. All tests pass.

Show outdated Hide outdated tests/client.py
@@ -425,7 +425,7 @@ def test_optimize(self):
self.solr.optimize()
self.assertEqual(len(self.solr.search('doc')), 4)
@unittest.expectedFailure
#@unittest.expectedFailure

This comment has been minimized.

@acdha

acdha Dec 6, 2013

Collaborator

This line should be removed outright since it was only added until #90 & #96 were fixed

@acdha

acdha Dec 6, 2013

Collaborator

This line should be removed outright since it was only added until #90 & #96 were fixed

@tongwang

This comment has been minimized.

Show comment
Hide comment
@tongwang

tongwang Dec 6, 2013

Contributor

Absolutely. Forgot to remove it.

Contributor

tongwang commented Dec 6, 2013

Absolutely. Forgot to remove it.

@acdha

This comment has been minimized.

Show comment
Hide comment
@acdha

acdha Dec 6, 2013

Collaborator

This looks good to me – have you tested it on anything older than Solr 4.6? I'm wondering whether this is a compatibility issue for the 3.x series & will test that shortly

Collaborator

acdha commented Dec 6, 2013

This looks good to me – have you tested it on anything older than Solr 4.6? I'm wondering whether this is a compatibility issue for the 3.x series & will test that shortly

@tongwang

This comment has been minimized.

Show comment
Hide comment
@tongwang

tongwang Dec 6, 2013

Contributor

I suspect the test started to break 5 months ago after commit 1322bff
I doubt it has anything to do with Solr releases. But will verify that.

Contributor

tongwang commented Dec 6, 2013

I suspect the test started to break 5 months ago after commit 1322bff
I doubt it has anything to do with Solr releases. But will verify that.

@tongwang

This comment has been minimized.

Show comment
Hide comment
@tongwang

tongwang Dec 6, 2013

Contributor

I take it back. Assuming people ran unit tests after the commit 1322bff, something must have been changed since then. It could be Solr 3.x vs 4.x. I will run some testing.

Contributor

tongwang commented Dec 6, 2013

I take it back. Assuming people ran unit tests after the commit 1322bff, something must have been changed since then. It could be Solr 3.x vs 4.x. I will run some testing.

@tongwang

This comment has been minimized.

Show comment
Hide comment
@tongwang

tongwang Dec 6, 2013

Contributor

Tested against 4 releases of Solr (4.6.0, 4.1.0, 3.6.2, 3.5.0), the result is the same. Without this patch, test_extract fails. With this patch, it passes.

Contributor

tongwang commented Dec 6, 2013

Tested against 4 releases of Solr (4.6.0, 4.1.0, 3.6.2, 3.5.0), the result is the same. Without this patch, test_extract fails. With this patch, it passes.

@acdha

This comment has been minimized.

Show comment
Hide comment
@acdha

acdha Dec 6, 2013

Collaborator

Did you run the full test suite to confirm that nothing else stopped working? It looks like Solr 3.6 is also fine in my admittedly modest testing

Collaborator

acdha commented Dec 6, 2013

Did you run the full test suite to confirm that nothing else stopped working? It looks like Solr 3.6 is also fine in my admittedly modest testing

@tongwang

This comment has been minimized.

Show comment
Hide comment
@tongwang

tongwang Dec 6, 2013

Contributor

All tests in the test suite pass against 4.6.0 and 4.1.0. However when I tested against 3.6.2 and 3.5.0, some tests were failing with or without my patch. So I assume it is my Solr server configuration and has nothing to do with the code change.

Here is the test result for 3.5.0 with my patch (one more error without my patch though):

.......F.........sFFE..FFFF....FF.......
======================================================================
ERROR: test__suggest_terms (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 220, in test__suggest_terms
    resp_body = self.solr._select({'terms.fl': 'title'})
  File "pysolr.py", line 333, in _select
    return self._send_request('get', path)
  File "pysolr.py", line 321, in _send_request
    raise SolrError(error_message)
SolrError: [Reason: Error 500 null

java.lang.NullPointerException
    at java.io.StringReader.<init>(StringReader.java:33)
    at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:203)
    at org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:80)
    at org.apache.solr.search.QParser.getQuery(QParser.java:142)
    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:101)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1372)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
]

======================================================================
FAIL: test_status (tests.admin.SolrCoreAdminTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/admin.py", line 18, in test_status
    self.assertTrue('name="defaultCoreName"' in self.solr_admin.status())
AssertionError: False is not true

======================================================================
FAIL: test__select (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 206, in test__select
    self.assertEqual(resp_data['response']['numFound'], 3)
AssertionError: 0 != 3

======================================================================
FAIL: test__send_request (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 187, in test__send_request
    self.assertTrue('"numFound":3' in resp_body)
AssertionError: False is not true

======================================================================
FAIL: test_add (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 357, in test_add
    self.assertEqual(len(self.solr.search('doc')), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_add_with_boost (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 375, in test_add_with_boost
    self.assertEqual(len(self.solr.search('doc')), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_commit (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 404, in test_commit
    self.assertEqual(len(self.solr.search('doc')), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_delete (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 388, in test_delete
    self.assertEqual(len(self.solr.search('doc')), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_optimize (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 417, in test_optimize
    self.assertEqual(len(self.solr.search('doc')), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_search (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 304, in test_search
    self.assertEqual(len(results), 3)
AssertionError: 0 != 3

----------------------------------------------------------------------
Ran 40 tests in 2.166s

FAILED (failures=9, errors=1, skipped=1)
Contributor

tongwang commented Dec 6, 2013

All tests in the test suite pass against 4.6.0 and 4.1.0. However when I tested against 3.6.2 and 3.5.0, some tests were failing with or without my patch. So I assume it is my Solr server configuration and has nothing to do with the code change.

Here is the test result for 3.5.0 with my patch (one more error without my patch though):

.......F.........sFFE..FFFF....FF.......
======================================================================
ERROR: test__suggest_terms (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 220, in test__suggest_terms
    resp_body = self.solr._select({'terms.fl': 'title'})
  File "pysolr.py", line 333, in _select
    return self._send_request('get', path)
  File "pysolr.py", line 321, in _send_request
    raise SolrError(error_message)
SolrError: [Reason: Error 500 null

java.lang.NullPointerException
    at java.io.StringReader.<init>(StringReader.java:33)
    at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:203)
    at org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:80)
    at org.apache.solr.search.QParser.getQuery(QParser.java:142)
    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:101)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1372)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
]

======================================================================
FAIL: test_status (tests.admin.SolrCoreAdminTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/admin.py", line 18, in test_status
    self.assertTrue('name="defaultCoreName"' in self.solr_admin.status())
AssertionError: False is not true

======================================================================
FAIL: test__select (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 206, in test__select
    self.assertEqual(resp_data['response']['numFound'], 3)
AssertionError: 0 != 3

======================================================================
FAIL: test__send_request (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 187, in test__send_request
    self.assertTrue('"numFound":3' in resp_body)
AssertionError: False is not true

======================================================================
FAIL: test_add (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 357, in test_add
    self.assertEqual(len(self.solr.search('doc')), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_add_with_boost (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 375, in test_add_with_boost
    self.assertEqual(len(self.solr.search('doc')), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_commit (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 404, in test_commit
    self.assertEqual(len(self.solr.search('doc')), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_delete (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 388, in test_delete
    self.assertEqual(len(self.solr.search('doc')), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_optimize (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 417, in test_optimize
    self.assertEqual(len(self.solr.search('doc')), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_search (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 304, in test_search
    self.assertEqual(len(results), 3)
AssertionError: 0 != 3

----------------------------------------------------------------------
Ran 40 tests in 2.166s

FAILED (failures=9, errors=1, skipped=1)
@tongwang

This comment has been minimized.

Show comment
Hide comment
@tongwang

tongwang Dec 6, 2013

Contributor

Without my patch, test_extract fails too on Solr 3.x:

======================================================================
ERROR: test_extract (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 440, in test_extract
    extracted = self.solr.extract(fake_f)
  File "pysolr.py", line 899, in extract
    files={'file': (file_obj.name, file_obj)})
  File "pysolr.py", line 324, in _send_request
    raise SolrError(error_message)
SolrError: [Reason: Error 400 [doc=null] missing required field: id]
Contributor

tongwang commented Dec 6, 2013

Without my patch, test_extract fails too on Solr 3.x:

======================================================================
ERROR: test_extract (tests.client.SolrTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/client.py", line 440, in test_extract
    extracted = self.solr.extract(fake_f)
  File "pysolr.py", line 899, in extract
    files={'file': (file_obj.name, file_obj)})
  File "pysolr.py", line 324, in _send_request
    raise SolrError(error_message)
SolrError: [Reason: Error 400 [doc=null] missing required field: id]

acdha added a commit that referenced this pull request Dec 6, 2013

Merge pull request #104 from tongwang/master
Fix content extraction (thanks @tongwang)

@acdha acdha merged commit 65ac0ab into django-haystack:master Dec 6, 2013

@acdha acdha referenced this pull request Dec 18, 2013

Closed

3.1.1 release #106

acdha added a commit to acdha/django-haystack that referenced this pull request Jan 26, 2014

Tests: skip Solr content extraction with old PySolr
Until pysolr 3.1.1 ships there's no point in running the Solr content
extraction tests because they'll fail:

django-haystack/pysolr#104
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment