[11.0] Concurrent Update Issue when number of server is more than 1

[Gokulakannanaj]

In my production, i have 4 servers[CPU] and capacity of channel is 1.When i run the job i got the concurrent update issue. But In local machine i'm not getting any issues. It works perfectly.
What went wrong ? It works perfectly in my local but in production getting concurrent update issue.

[guewen]

Can you show the definition of your job function with concurrent update issues?

[Gokulakannanaj]

Job Function

def send_approve_action(self):
self.with_delay().do_process_in_single_stage

@api.multi
@job
def do_process_in_single_stage(self):
self.env['my.plan'].create_records(user_id)

Another model like my.plan

@api.multi
def create_records(self, user_id):
self.create({'user_id': user_id, 'login_date': datetime.date.today()})

Getting Below Error:

Traceback (most recent call last):
File "/mnt/addons/extra/core/enterprise/models/customers.py", line 2430, in do_process_in_single_stage
jobs.send_approve_action()
File "/mnt/addons/extra/core/enterprise/models/customers.py", line 1736, in send_approve_action
self.env['standard.call.plan'].create_records(self.id, stp_plan, 'master', today)
File "/mnt/addons/extra/core/field/models/plan.py", line 1076, in create_records
standard_plan.write({'plan': [(0, 0, values)], 'destroy': False})
File "/mnt/addons/extra/core/field/models/plan.py", line 332, in write
return super(StandardPlan, self).write(vals)
File "/mnt/odoo-source/odoo/models.py", line 3024, in write
self._write(old_vals)
File "/mnt/odoo-source/odoo/models.py", line 3125, in _write
cr.execute(query, params + (sub_ids,))
File "/mnt/odoo-source/odoo/sql_db.py", line 155, in wrapper
return f(self, *args, **kwargs)
File "/mnt/odoo-source/odoo/sql_db.py", line 232, in execute
res = self._obj.execute(query, params)
psycopg2.extensions.TransactionRollbackError: could not serialize access due to concurrent update

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/mnt/addons/extra/web/queue_job/controllers/main.py", line 97, in runjob
self._try_perform_job(env, job)
File "/mnt/addons/extra/web/queue_job/controllers/main.py", line 61, in _try_perform_job
job.perform()
File "/mnt/addons/extra/web/queue_job/job.py", line 364, in perform
self.result = self.func(*tuple(self.args), **self.kwargs)
File "/mnt/addons/extra/core/enterprise/models/customers.py", line 2434, in do_process_in_single_stage
jobs.write({'status': 'submit', 'called_from': False})
File "/mnt/addons/extra/core/enterprise/models/customers.py", line 1573, in write
return super(CustomerMaster, self).write(vals)
File "/mnt/odoo-source/odoo/addons/base/res/res_partner.py", line 514, in write
result = result and super(Partner, self).write(vals)
File "/mnt/odoo-source/addons/mail/models/mail_thread.py", line 277, in write
result = super(MailThread, self).write(values)
File "/mnt/odoo-source/addons/mail/models/mail_activity.py", line 327, in write
return super(MailActivityMixin, self).write(vals)
File "/mnt/odoo-source/odoo/models.py", line 3024, in write
self._write(old_vals)
File "/mnt/odoo-source/odoo/models.py", line 3125, in _write
cr.execute(query, params + (sub_ids,))
File "/mnt/odoo-source/odoo/sql_db.py", line 155, in wrapper
return f(self, *args, **kwargs)
File "/mnt/odoo-source/odoo/sql_db.py", line 232, in execute
res = self._obj.execute(query, params)
psycopg2.InternalError: current transaction is aborted, commands ignored until end of transaction block

I'm getting this error when i run my database with more than 2 CPU(Nodes).
But no error with single CPU and also my Channel capacity 1 and throttle 2 seconds like (channels = root:1:throttle=10)

Thanks.

[guewen]

This seems to be unrelated to the job queue, if you look at the traceback, you'll see that your function writes to a bunch of models, partner, mail thread, mail activity, ... if you have 2 concurrent jobs that write to the same record, you'll have a concurrent transaction error. The queue job automatically retries the jobs when there is such error, but only a certain amount of times after what it sets the job as failed.

I mean, if you were running the same function on the same record using xml-rpc on several concurrent requests, you would have the same issue.

What you should track here is why you have several jobs calling the function for the same record. If this is unavoidable, then you should probably set the job function in a channel with a capacity of 1 so you won't have more than 1 job running at once.

[Gokulakannanaj]

I'm not getting any error when my no.of CPU is 1. If changed my no.of CPU is 2 i'm getting this concurrent error and also load-balancer not worked in Job Queue.so this related to load-balancing issue.
If process 1 started via CPU 1, that time CPU 2 also trying to perform the process 1. so only i'm getting this issue.
Here how can i control to perform the process 1 by CPU 1 only ?

Error:
Traceback (most recent call last):
File "/mnt/addons/extra/web/queue_job/controllers/main.py", line 105, in runjob
seconds=PG_RETRY)
File "/mnt/addons/extra/web/queue_job/controllers/main.py", line 87, in retry_postpone
job.store()
File "/mnt/addons/extra/web/queue_job/job.py", line 409, in store
db_record = self.db_record()
File "/mnt/addons/extra/web/queue_job/job.py", line 433, in db_record
return self.db_record_from_uuid(self.env, self.uuid)
File "/mnt/addons/extra/web/queue_job/job.py", line 262, in db_record_from_uuid
record = model.search([('uuid', '=', job_uuid)], limit=1)
File "/mnt/odoo-source/odoo/models.py", line 1422, in search
res = self._search(args, offset=offset, limit=limit, order=order, count=count)
File "/mnt/odoo-source/odoo/models.py", line 3721, in _search
self._cr.execute(query_str, where_clause_params)
File "/mnt/odoo-source/odoo/sql_db.py", line 155, in wrapper
return f(self, *args, **kwargs)
File "/mnt/odoo-source/odoo/sql_db.py", line 232, in execute
res = self._obj.execute(query, params)
psycopg2.InternalError: current transaction is aborted, commands ignored until end of transaction block

[guewen]

The last traceback is different, it's not inside the job but an error in the jobrunner, it could use some investigation.

[guewen]

The errors in #103 (comment) were not the same, so you are not repeating anything. Having concurrent issues is of course more likely to happen when you actually run things concurrently, it doesn't really help to repeat it. The useful information is more what you actually get, like the traceback in #103 (comment).

I'm not sure to understand the relation between the number of CPUs (CPU cores?) and a load balancer? When you say CPU aren't you speaking about different hosts/different odoo processes? If so, ensure that you don't run 2 jobrunner concurrently because this is not supported.

Edit: reread the first post and indeed by CPU you mean servers... that got me confused. Run the jobrunner on only one of them.

[Gokulakannanaj]

My intension.

Consider CPU 1 and CPU 2 in idle stage.
Process 1 started. Sending request to CPU 1 and CPU 2. Both are trying to perform same task. so only concurrent update produced.
Here how can i tell all jobs are should performed via CPU 1 only ? May be it resolved when jobs are performed via single CPU

[guewen]

As said before, you should start the jobrunner on only one server, precisely, you must have queue_job in --load on only one server, not the others. This way only one of them will process the jobs.

[Gokulakannanaj]

I have 4 servers but my code is common to all.
How can i tell this(server) is for Job Queue (I think its configuration) ?

My configuration
[options]
server_wide_modules = web,queue_job

[queue_job]
channels = root:1

Thanks

[guewen]

precisely, you must have queue_job in --load on only one server, not the others

Yes, server_wide_modules is the equivalent to --load, so put this only on one server. The other servers must have:

[options]
server_wide_modules = web

[guewen]

Your code can be the same, but you have to find a solution to have a different odoo configuration. I can't help you more...

[oliverzgy]

@guewen Hi Guewen, regarding this, can we add an option running job this way?
Say 4 channels, we load the jobs into the channels with unique job function and user id. That is, all the time the running jobs are with unique job function and user id.