-
-
Notifications
You must be signed in to change notification settings - Fork 442
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use a requests session to limit opened sessions #270
Conversation
We have some issues with a very high throughput of jobs and Odoo's anonymous sessions. Every job pushed to Odoo uses a new anonymous sessions, which creates a tremendous amount of session files. Using a requests.session means that the session will keep the same cookie once it receives the first response. There is 2 problems with this implementation though: * requests.Session is not proved to be threadsafe [0], but maybe we can manually retrieve the session id and craft the next messages with it. Anyway as they are all anonymous, if we lose a session at some point there is no big deal. * the main issue is that we don't wait for an answer so until we have a requests responding before the timeout we won't have any session id [0] psf/requests#2766
e974d70
to
d9c7d32
Compare
@guewen thanks for tackling this issue! Code looks good. I'm a bit scared of the shared session object. But from reading around it looks like it should work. So not sure if it would be worth spinning a ThreadPool. |
About the session shared between threads, my feeling is that with the usage we have here, it should not be an issue. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I'd add a docstring just to mention why that empty method is there ;)
@@ -68,6 +68,10 @@ def _try_perform_job(self, session_hdl, job): | |||
self.job_storage_class(session).store(job) | |||
_logger.debug('%s done', job) | |||
|
|||
@http.route('/connector/session', type='http', auth="none") | |||
def session(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
docstring pls ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good catch, added
This fix the problem explained in the previous commit: When we run a job, we don't wait for the answer. So when we don't have any cookie yet, we send a GET on a very fast controller route which will return a cookie, used for all the subsequents calls to runjob. This effectively prevent to create one session file per job. It still needs intensive testing.
d9c7d32
to
1dec701
Compare
We have some issues with a very high throughput of jobs and Odoo's
anonymous sessions. Every job pushed to Odoo uses a new anonymous
sessions, which creates a tremendous amount of session files.
Using a requests.session means that the session will keep the same
cookie once it receives the first response.
There is 2 problems with the usage of a session though:
Anyway as they are all anonymous, if we lose or mix up odoo sessions at some point it should work as well.
requests responding before the timeout we won't have any session id
The second issue is resolved in the last commit with a pre-request to obtain a cookie.
When we run a job, we don't wait for the answer. So when we don't have
any cookie yet, we send a GET on a very fast controller route which will
return a cookie, used for all the subsequents calls to runjob.
This effectively prevent to create one session file per job.
I tested it on my side on a large amount of jobs, but extra testing could be helpful.
What I did not test: