Invenio v3.0.0a1

[lnielsen]

Signed-off-by: Lars Holm Nielsen lars.holm.nielsen@cern.ch

See https://gist.github.com/lnielsen/982d3ea51d2d69983bce

[nharraud]

Works for me. I installed it twice got both time a slow answer on the first REST requests. Do we have something loading when the first request is received?

[nharraud]

Wait I can reproduce the problem in fact. This is not just slowing but stopping on chrome.
SCENARIO:
1/ open http://localhost:5000/reset in chrome
2/ curl -XGET http://localhost:5000/api/records/1 -H "Accept: application/json"
=> the GET blocks and does not appear in the server's log. Every other API request will be stuck in the same way.
3/ reload the page http://localhost:5000/reset
=> the API requests are executed at last.

When using firefox, the API requests run after one or two seconds instead of being stopped.
In chrome, API requests are also unstuck when you open the UI page again.
This behavior can be reproduced with every combination of API request (PATCH, PUT, GET...) and UI page (loggin, reset, record...).

It's as if the process was still serving the UI and stopped processing the API requests. It is still listening for them though as the server log has Broken pipe errors if I kill the curl before reloading the page (step 3).

[nharraud]

The API is not blocked when running in multi-threaded mode:

from mysite.factory import create_app

create_app().run(threaded=True)

Also if I open a UI page in chrome, wait for it to load and then kill the server, I get this error:

Exception happened during processing of request from ('127.0.0.1', 49696)
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 655, in __init__
    self.handle()
  File "/Users/nharraud/.virtualenvs/invenio3test/lib/python2.7/site-packages/werkzeug/serving.py", line 203, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/Users/nharraud/.virtualenvs/invenio3test/lib/python2.7/site-packages/werkzeug/serving.py", line 234, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 451, in readline
    data = self._sock.recv(self._rbufsize)

[tiborsimko]

When using firefox, the API requests run after one or two seconds instead of being stopped.

FWIW I don't seem to observe any delay. If I open a web page as in your scenario step 1, the subsequent API query is fast and immediate:

$ time curl -XGET http://localhost:5000/api/records/1 -H "Accept: application/json"
{
  "data": {
    "software": "Invenioyyy", 
    "title": "Invenio 3 Rocks"
  }, 
  "id": 1, 
  "links": {
    "self": "http://localhost:5000/api/records/1"
  }
}
real    0m0.043s
user    0m0.008s
sys     0m0.016s

[nharraud]

Yes I saw that sometime this not happen. But when it does it blocks each time. I used lsof and saw that a connection was established and never closing. In chrome://net-internals/ the sockets where marked as idle. If I close them forcefully the API requests get unblocked.

I have been trying to force a keep-alive socket with curl to see if this was the problem but the macos version seems cripled (or at least that what curl says in the output). If the issue is still here next week I will try to work on it again, I spent already too much time on it with B2Share release comming.

[nharraud]

Forget it, @jirikuncar explained that it's just werkzeug accepting only one connection in single threaded process.

[otron]

I got to the fifth line of Lars' gist where the process is halted by the inveniomanage command not being found.

[nharraud]

@otron Did you use the exact same lines? As Invenio is not released yet they have to be changed a little in order to use this pull request.

[otron]

@nharraud I did use the exact same lines. Which ones require which changes?

[nharraud]

see checking-out-pull-requests-locally. Instead of installing Invenio you fetch this pull request and make pip install -e .[all] after that it's all the same.

[otron]

Hahaha. Right, yeah, makes sense it won't work if it assumes the PR branch of the repository is residing in ~/src/invenio. That was not clear to me from the provided instructions.

Thanks!

Suggest adding the following between lines 3 and 4:

git clone git@github.com:inveniosoftware/invenio.git
cd invenio
git fetch origin pull/3550/head:pr3550
git checkout pr3550

[otron]

Guide assumes redis is running on the system, although it doesn't mention this.
If redis isn't running then celery worker -A mysite.celery -l INFO will complain about not being able to connect and the next command (where you create a record) fails.

[tiborsimko]

@otron WRT testing the branch, the explicit addition of upstream/pr/3550 isn't really necessary to the outward audience, since the PR will be merged soon. Until then, people could also try something like:

pip install -e git+git://github.com/lnielsen/invenio.git@122e918bfb1b238a0c78ef7089cfde49a8a48792#egg=invenio[full]

if they don't check out the sources.

WRT Redis, it is briefly mentioned in the INSTALL.rst, but we should indeed improve the new installation documentation, e.g. the INSTALL.rst file in the PR refers to a tutorial that does not exist, and docs/installation/installation-detailed.rst had been simply reading the same INSTALL.rst that has now been much simplified... so the OS steps are kinda missing. This could be improved also after merge.

[otron]

@tiborsimko Thanks for the clarification. I assumed the instructions were for this specific PR/branch so that people could verify that it worked or check it out or whatever. It wasn't clear to me that they were actually meant as a suggestion for the install procedure instructions post-merge/release on PyPi.