Use a requests session to limit opened sessions #270
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We have some issues with a very high throughput of jobs and Odoo's anonymous sessions. Every job pushed to Odoo uses a new anonymous sessions, which creates a tremendous amount of session files. Using a requests.session means that the session will keep the same cookie once it receives the first response. There is 2 problems with this implementation though: * requests.Session is not proved to be threadsafe [0], but maybe we can manually retrieve the session id and craft the next messages with it. Anyway as they are all anonymous, if we lose a session at some point there is no big deal. * the main issue is that we don't wait for an answer so until we have a requests responding before the timeout we won't have any session id [0] psf/requests#2766
guewen
force-pushed
the
test-anonymous-session
branch
from
e974d70
to
d9c7d32
Compare
|
@guewen thanks for tackling this issue! Code looks good. I'm a bit scared of the shared session object. But from reading around it looks like it should work. So not sure if it would be worth spinning a ThreadPool. |
|
About the session shared between threads, my feeling is that with the usage we have here, it should not be an issue. |
simahawk
approved these changes
Jan 2, 2018
| @@ -68,6 +68,10 @@ def _try_perform_job(self, session_hdl, job): | |||
| self.job_storage_class(session).store(job) | |||
| _logger.debug('%s done', job) | |||
|
|
|||
| @http.route('/connector/session', type='http', auth="none") | |||
| def session(self): | |||
This fix the problem explained in the previous commit: When we run a job, we don't wait for the answer. So when we don't have any cookie yet, we send a GET on a very fast controller route which will return a cookie, used for all the subsequents calls to runjob. This effectively prevent to create one session file per job. It still needs intensive testing.
guewen
force-pushed
the
test-anonymous-session
branch
from
d9c7d32
to
1dec701
Compare
This was referenced Feb 8, 2018
leemannd
approved these changes
Mar 27, 2018
lmignon
approved these changes
May 25, 2018
anikeenko-viktor
pushed a commit
to anikeenko-viktor/queue
that referenced
this pull request
Oct 27, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have some issues with a very high throughput of jobs and Odoo's
anonymous sessions. Every job pushed to Odoo uses a new anonymous
sessions, which creates a tremendous amount of session files.
Using a requests.session means that the session will keep the same
cookie once it receives the first response.
There is 2 problems with the usage of a session though:
Anyway as they are all anonymous, if we lose or mix up odoo sessions at some point it should work as well.
requests responding before the timeout we won't have any session id
The second issue is resolved in the last commit with a pre-request to obtain a cookie.
When we run a job, we don't wait for the answer. So when we don't have
any cookie yet, we send a GET on a very fast controller route which will
return a cookie, used for all the subsequents calls to runjob.
This effectively prevent to create one session file per job.
I tested it on my side on a large amount of jobs, but extra testing could be helpful.
What I did not test: