Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

generateorders: "order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!" #828

Open
kousu opened this issue Mar 5, 2018 · 9 comments
Open

generateorders: "order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!" #828

kousu opened this issue Mar 5, 2018 · 9 comments

Comments

@kousu
Copy link

kousu commented Mar 5, 2018 •
edited

EDIT: sometimes it returns 3 as well. It could be any number. 1 is the most likely, 2 is the next most likely.

On our production server, at on Fri, Mar 2, 2018 at 9:00 PM, running the cronjob

/usr/bin/docker exec souschef_web_1 python src/manage.py generateorders $(date +%Y-%m-%d) --days 100

gave

Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

and the next night, Sat, Mar 3, 2018 at 9:00 PM

Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

We only just thought to make the cronjobs log what they're doing to email, so I don't know how long this has been going on.

The output was back to normal tonight (Sun, Mar 4, 2018 at 9:00 PM):

0 orders created on 2018-03-04: to be delivered on 2018-03-04.
81 orders created on 2018-03-04: to be delivered on 2018-03-05.
76 orders created on 2018-03-04: to be delivered on 2018-03-06.
77 orders created on 2018-03-04: to be delivered on 2018-03-07.
0 orders created on 2018-03-04: to be delivered on 2018-03-08.
81 orders created on 2018-03-04: to be delivered on 2018-03-09.
63 orders created on 2018-03-04: to be delivered on 2018-03-10.
0 orders created on 2018-03-04: to be delivered on 2018-03-11.
81 orders created on 2018-03-04: to be delivered on 2018-03-12.
76 orders created on 2018-03-04: to be delivered on 2018-03-13.
77 orders created on 2018-03-04: to be delivered on 2018-03-14.
0 orders created on 2018-03-04: to be delivered on 2018-03-15.
81 orders created on 2018-03-04: to be delivered on 2018-03-16.
63 orders created on 2018-03-04: to be delivered on 2018-03-17.
0 orders created on 2018-03-04: to be delivered on 2018-03-18.
81 orders created on 2018-03-04: to be delivered on 2018-03-19.
76 orders created on 2018-03-04: to be delivered on 2018-03-20.
77 orders created on 2018-03-04: to be delivered on 2018-03-21.
0 orders created on 2018-03-04: to be delivered on 2018-03-22.
81 orders created on 2018-03-04: to be delivered on 2018-03-23.
63 orders created on 2018-03-04: to be delivered on 2018-03-24.
0 orders created on 2018-03-04: to be delivered on 2018-03-25.
81 orders created on 2018-03-04: to be delivered on 2018-03-26.
76 orders created on 2018-03-04: to be delivered on 2018-03-27.
77 orders created on 2018-03-04: to be delivered on 2018-03-28.
0 orders created on 2018-03-04: to be delivered on 2018-03-29.
81 orders created on 2018-03-04: to be delivered on 2018-03-30.
63 orders created on 2018-03-04: to be delivered on 2018-03-31.
0 orders created on 2018-03-04: to be delivered on 2018-04-01.
81 orders created on 2018-03-04: to be delivered on 2018-04-02.
76 orders created on 2018-03-04: to be delivered on 2018-04-03.
77 orders created on 2018-03-04: to be delivered on 2018-04-04.
0 orders created on 2018-03-04: to be delivered on 2018-04-05.
81 orders created on 2018-03-04: to be delivered on 2018-04-06.
63 orders created on 2018-03-04: to be delivered on 2018-04-07.
0 orders created on 2018-03-04: to be delivered on 2018-04-08.
81 orders created on 2018-03-04: to be delivered on 2018-04-09.
76 orders created on 2018-03-04: to be delivered on 2018-04-10.
77 orders created on 2018-03-04: to be delivered on 2018-04-11.
0 orders created on 2018-03-04: to be delivered on 2018-04-12.
81 orders created on 2018-03-04: to be delivered on 2018-04-13.
63 orders created on 2018-03-04: to be delivered on 2018-04-14.
0 orders created on 2018-03-04: to be delivered on 2018-04-15.
81 orders created on 2018-03-04: to be delivered on 2018-04-16.
76 orders created on 2018-03-04: to be delivered on 2018-04-17.
77 orders created on 2018-03-04: to be delivered on 2018-04-18.
0 orders created on 2018-03-04: to be delivered on 2018-04-19.
81 orders created on 2018-03-04: to be delivered on 2018-04-20.
63 orders created on 2018-03-04: to be delivered on 2018-04-21.
0 orders created on 2018-03-04: to be delivered on 2018-04-22.
81 orders created on 2018-03-04: to be delivered on 2018-04-23.
76 orders created on 2018-03-04: to be delivered on 2018-04-24.
77 orders created on 2018-03-04: to be delivered on 2018-04-25.
0 orders created on 2018-03-04: to be delivered on 2018-04-26.
81 orders created on 2018-03-04: to be delivered on 2018-04-27.
63 orders created on 2018-03-04: to be delivered on 2018-04-28.
0 orders created on 2018-03-04: to be delivered on 2018-04-29.
81 orders created on 2018-03-04: to be delivered on 2018-04-30.
76 orders created on 2018-03-04: to be delivered on 2018-05-01.
77 orders created on 2018-03-04: to be delivered on 2018-05-02.
0 orders created on 2018-03-04: to be delivered on 2018-05-03.
81 orders created on 2018-03-04: to be delivered on 2018-05-04.
63 orders created on 2018-03-04: to be delivered on 2018-05-05.
0 orders created on 2018-03-04: to be delivered on 2018-05-06.
81 orders created on 2018-03-04: to be delivered on 2018-05-07.
76 orders created on 2018-03-04: to be delivered on 2018-05-08.
76 orders created on 2018-03-04: to be delivered on 2018-05-09.
0 orders created on 2018-03-04: to be delivered on 2018-05-10.
81 orders created on 2018-03-04: to be delivered on 2018-05-11.
62 orders created on 2018-03-04: to be delivered on 2018-05-12.
0 orders created on 2018-03-04: to be delivered on 2018-05-13.
81 orders created on 2018-03-04: to be delivered on 2018-05-14.
75 orders created on 2018-03-04: to be delivered on 2018-05-15.
76 orders created on 2018-03-04: to be delivered on 2018-05-16.
0 orders created on 2018-03-04: to be delivered on 2018-05-17.
81 orders created on 2018-03-04: to be delivered on 2018-05-18.
62 orders created on 2018-03-04: to be delivered on 2018-05-19.
0 orders created on 2018-03-04: to be delivered on 2018-05-20.
81 orders created on 2018-03-04: to be delivered on 2018-05-21.
75 orders created on 2018-03-04: to be delivered on 2018-05-22.
76 orders created on 2018-03-04: to be delivered on 2018-05-23.
0 orders created on 2018-03-04: to be delivered on 2018-05-24.
81 orders created on 2018-03-04: to be delivered on 2018-05-25.
62 orders created on 2018-03-04: to be delivered on 2018-05-26.
0 orders created on 2018-03-04: to be delivered on 2018-05-27.
81 orders created on 2018-03-04: to be delivered on 2018-05-28.
75 orders created on 2018-03-04: to be delivered on 2018-05-29.
76 orders created on 2018-03-04: to be delivered on 2018-05-30.
0 orders created on 2018-03-04: to be delivered on 2018-05-31.
81 orders created on 2018-03-04: to be delivered on 2018-06-01.
62 orders created on 2018-03-04: to be delivered on 2018-06-02.
0 orders created on 2018-03-04: to be delivered on 2018-06-03.
81 orders created on 2018-03-04: to be delivered on 2018-06-04.
75 orders created on 2018-03-04: to be delivered on 2018-06-05.
76 orders created on 2018-03-04: to be delivered on 2018-06-06.
0 orders created on 2018-03-04: to be delivered on 2018-06-07.
81 orders created on 2018-03-04: to be delivered on 2018-06-08.
62 orders created on 2018-03-04: to be delivered on 2018-06-09.
0 orders created on 2018-03-04: to be delivered on 2018-06-10.
81 orders created on 2018-03-04: to be delivered on 2018-06-11.
@kousu
Copy link
Author

kousu commented Mar 5, 2018

I'm able to reproduce it with manually running

$ /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders 2018-03-03 --days 100
 
Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

@kousu
Copy link
Author

kousu commented Mar 5, 2018

Going further back in time, sometimes one day will get generated and then the crash will happen:

$ /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders 2018-03-02 --days 100
 
Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

$ /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders 2018-03-01 --days 100
 
0 orders created on 2018-03-01: to be delivered on 2018-03-01.
Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

$ /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders 2018-02-28 --days 100
 
Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

$ /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders 2018-02-27 --days 100
 
75 orders created on 2018-02-27: to be delivered on 2018-02-27.
Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

@kousu kousu mentioned this issue Mar 5, 2018
Open
@kousu
Copy link
Author

kousu commented Mar 5, 2018

I've traced the code; what's happening is that in auto_create_orders there's

            try:
                order = Order.objects.get(client=client,
                                          delivery_date=delivery_date)
                created_orders.append(order)
                continue
            except Order.DoesNotExist:
                # If no order for this client/date, create it and attach items
                ...

like the exception says, that get() is choking on multiple results where it's only expecting one. I found a client ("xxxx") that triggers this and poked around the database. I found, for example:

MariaDB [feast]> select * from order_order where delivery_date = "2017-12-06" and client_id=xxxx;                  
+-------+---------------+---------------+--------+-----------+
| id    | creation_date | delivery_date | status | client_id |
+-------+---------------+---------------+--------+-----------+
| yyyyy | 2017-12-06    | 2017-12-06    | C      |      xxxx |
| yyyyy | 2017-12-06    | 2017-12-06    | C      |      xxxx |
| yyyyy | 2017-12-06    | 2017-12-06    | D      |      xxxx |
+-------+---------------+---------------+--------+-----------+

There's quite a few of these duplicate orders:

select count(*) from (select *, count(*) as c from order_order group by client_id, delivery_date) 
as x where x.c > 1;                                                                                                
+----------+
| count(*) |
+----------+
|      217 |
+----------+

Here they all our (censored for privacy):

root@juniorroulant:/var/www/sous-chef# cat a
id	creation_date	delivery_date	status	cliend_id	c
yyyyy	2017-12-25	2017-12-25	C	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-12-06	2017-12-06	C	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2017-03-30	2017-05-10	C	xxx	2
yyyyy	2017-08-16	2017-10-31	C	xxx	2
yyyyy	2017-11-03	2017-11-03	C	xxx	2
yyyyy	2017-11-04	2017-11-04	C	xxx	2
yyyyy	2017-11-06	2017-11-06	C	xxx	2
yyyyy	2017-11-08	2017-11-08	C	xxx	2
yyyyy	2017-11-10	2017-11-10	C	xxx	2
yyyyy	2017-11-11	2017-11-11	C	xxx	2
yyyyy	2017-11-13	2017-11-13	C	xxx	2
yyyyy	2017-11-15	2017-11-15	C	xxx	3
yyyyy	2017-11-17	2017-11-17	C	xxx	2
yyyyy	2017-11-18	2017-11-18	C	xxx	3
yyyyy	2017-11-20	2017-11-20	C	xxx	2
yyyyy	2017-11-21	2017-11-21	C	xxx	2
yyyyy	2017-11-22	2017-11-22	C	xxx	2
yyyyy	2017-11-24	2017-11-24	C	xxx	2
yyyyy	2017-11-28	2017-11-28	C	xxx	2
yyyyy	2017-11-29	2017-11-29	C	xxx	2
yyyyy	2017-12-02	2017-12-02	C	xxx	2
yyyyy	2017-12-04	2017-12-04	C	xxx	2
yyyyy	2017-12-05	2017-12-05	C	xxx	3
yyyyy	2017-12-06	2017-12-06	C	xxx	2
yyyyy	2017-12-08	2017-12-08	C	xxx	2
yyyyy	2017-12-09	2017-12-09	C	xxx	2
yyyyy	2017-12-11	2017-12-11	C	xxx	2
yyyyy	2017-12-12	2017-12-12	C	xxx	2
yyyyy	2017-12-15	2017-12-15	C	xxx	2
yyyyy	2017-12-16	2017-12-16	C	xxx	2
yyyyy	2017-12-18	2017-12-18	C	xxx	2
yyyyy	2017-12-19	2017-12-19	C	xxx	2
yyyyy	2017-12-20	2017-12-20	C	xxx	2
yyyyy	2017-12-23	2017-12-23	C	xxx	2
yyyyy	2017-12-25	2017-12-25	C	xxx	2
yyyyy	2017-12-27	2017-12-27	C	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2018-01-01	2018-01-01	C	xxx	3
yyyyy	2018-01-02	2018-01-02	C	xxx	2
yyyyy	2018-01-03	2018-01-03	C	xxx	2
yyyyy	2018-01-05	2018-01-05	C	xxx	2
yyyyy	2018-01-06	2018-01-06	C	xxx	2
yyyyy	2018-01-08	2018-01-08	C	xxx	2
yyyyy	2018-01-09	2018-01-09	C	xxx	2
yyyyy	2018-01-10	2018-01-10	C	xxx	2
yyyyy	2018-01-12	2018-01-12	C	xxx	2
yyyyy	2018-01-13	2018-01-13	C	xxx	2
yyyyy	2018-01-16	2018-01-16	C	xxx	2
yyyyy	2018-01-19	2018-01-19	C	xxx	2
yyyyy	2018-01-20	2018-01-20	C	xxx	2
yyyyy	2018-01-22	2018-01-22	C	xxx	2
yyyyy	2018-01-23	2018-01-23	C	xxx	2
yyyyy	2018-01-24	2018-01-24	C	xxx	2
yyyyy	2018-01-26	2018-01-26	C	xxx	2
yyyyy	2018-01-27	2018-01-27	C	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2018-01-31	2018-01-31	C	xxx	2
yyyyy	2018-02-02	2018-02-02	C	xxx	2
yyyyy	2018-02-03	2018-02-03	C	xxx	2
yyyyy	2018-02-05	2018-02-05	C	xxx	2
yyyyy	2018-02-06	2018-02-06	C	xxx	2
yyyyy	2018-02-07	2018-02-07	C	xxx	2
yyyyy	2018-02-10	2018-02-10	C	xxx	2
yyyyy	2018-02-12	2018-02-12	C	xxx	2
yyyyy	2018-02-16	2018-02-16	C	xxx	2
yyyyy	2018-02-19	2018-02-19	C	xxx	2
yyyyy	2018-02-20	2018-02-20	C	xxx	2
yyyyy	2018-02-21	2018-02-21	C	xxx	2
yyyyy	2018-02-23	2018-02-23	C	xxx	2
yyyyy	2018-02-24	2018-02-24	C	xxx	2
yyyyy	2018-02-26	2018-02-26	C	xxx	2
yyyyy	2018-02-28	2018-02-28	C	xxx	3
yyyyy	2018-03-02	2018-03-02	C	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-11-24	2017-11-24	C	xxx	2
yyyyy	2017-11-27	2017-11-27	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-08-16	2017-10-31	C	xxx	2
yyyyy	2017-11-04	2017-11-04	C	xxx	2
yyyyy	2017-11-24	2017-11-24	C	xxx	2
yyyyy	2017-11-28	2017-11-28	C	xxx	2
yyyyy	2017-12-02	2017-12-02	C	xxx	2
yyyyy	2017-12-08	2017-12-08	C	xxx	2
yyyyy	2017-12-13	2017-12-13	C	xxx	2
yyyyy	2017-12-19	2017-12-19	C	xxx	2
yyyyy	2017-12-26	2017-12-26	C	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2018-01-03	2018-01-03	C	xxx	2
yyyyy	2018-01-06	2018-01-06	C	xxx	2
yyyyy	2018-01-12	2018-01-12	D	xxx	2
yyyyy	2018-01-16	2018-01-16	C	xxx	2
yyyyy	2018-01-19	2018-01-19	C	xxx	2
yyyyy	2018-01-27	2018-01-27	C	xxx	2
yyyyy	2018-01-31	2018-01-31	C	xxx	2
yyyyy	2018-02-13	2018-02-13	C	xxx	2
yyyyy	2018-02-26	2018-02-26	C	xxx	2
yyyyy	2018-03-03	2018-03-03	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-12-05	2017-12-05	C	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2018-01-23	2018-01-23	C	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-11-28	2017-11-28	C	xxx	2
yyyyy	2017-12-15	2017-12-15	C	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-11-08	2017-11-08	C	xxx	2
yyyyy	2017-12-08	2017-12-08	C	xxx	2
yyyyy	2017-12-18	2017-12-18	C	xxx	2
yyyyy	2017-12-22	2017-12-22	C	xxx	2
yyyyy	2017-12-29	2017-12-29	C	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2018-02-03	2018-02-03	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-11-14	2017-11-14	C	xxx	2
yyyyy	2017-11-15	2017-11-15	C	xxx	2
yyyyy	2017-11-17	2017-11-17	C	xxx	2
yyyyy	2018-02-26	2018-02-26	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-12-22	2017-12-22	C	xxx	2
yyyyy	2018-01-02	2018-01-02	C	xxx	2
yyyyy	2018-01-13	2018-01-13	C	xxx	2
yyyyy	2018-01-19	2018-01-19	C	xxx	2
yyyyy	2018-01-20	2018-01-20	C	xxx	2
yyyyy	2018-01-26	2018-01-26	C	xxx	2
yyyyy	2017-01-24	2016-03-29	D	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2018-03-03	2018-03-03	C	xxx	2
yyyyy	2017-12-11	2017-12-11	C	xxx	2
yyyyy	2017-12-12	2017-12-12	C	xxx	2
yyyyy	2017-12-20	2017-12-20	C	xxx	2
yyyyy	2017-11-06	2017-11-06	C	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2017-11-04	2017-11-04	C	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2017-12-08	2017-12-08	C	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2
yyyyy	2017-12-16	2017-12-16	C	xxx	2
yyyyy	2017-12-06	2017-12-06	C	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2018-02-03	2018-02-03	C	xxx	2
yyyyy	2018-02-05	2018-02-05	C	xxx	2
yyyyy	2018-02-09	2018-02-09	C	xxx	2
yyyyy	2018-02-16	2018-02-16	C	xxx	3
yyyyy	2018-02-19	2018-02-19	C	xxx	2
yyyyy	2018-02-20	2018-02-20	C	xxx	2
yyyyy	2017-11-04	2017-11-04	C	xxx	2
yyyyy	2017-11-06	2017-11-06	C	xxx	2
yyyyy	2017-11-08	2017-11-08	C	xxx	2
yyyyy	2017-11-15	2017-11-15	C	xxx	2
yyyyy	2017-12-06	2017-12-06	C	xxx	3
yyyyy	2017-12-18	2017-12-18	C	xxx	2
yyyyy	2018-02-02	2018-02-02	C	xxx	2
yyyyy	2018-02-03	2018-02-03	C	xxx	2
yyyyy	2018-02-10	2018-02-10	C	xxx	2
yyyyy	2018-02-26	2018-02-26	C	xxx	2
yyyyy	2018-02-28	2018-02-28	C	xxx	2
yyyyy	2017-12-30	2017-12-30	C	xxx	2
yyyyy	2018-01-29	2018-01-29	C	xxx	2

@kousu
Copy link
Author

kousu commented Mar 5, 2018

I've spot checked a few, and most cases contain a cancellation (state=C, but some contain only two or three Delivered (state=D). My theory is that orders are getting call-in cancellations, but then on the day of the delivery or soon before a re-order came in, the coordinator @JeanFrancoisFournier or whoever went in to manually add one, and bam now we've got a consistency problem. The UI probably doesn't highlight when, and there's no UNIQUE of (client_id, delivery_date) on order_order that would at least crash an attempt to do such a thing.

We never want to have more than one order per client per day, that would break our business logic, so I think adding a UNIQUE constraint is the proper first step, and then we have to work backwards from that to every create on this table, turning them all into upserts that flips existing order into state=O if it exists. Or maybe add a bunch of error UI that reports the problem to the user and directs them to manually edit the order status.

Also generateorders should be more tolerant of exceptions. If the orders fail to generate for a single day that shouldn't break all the other days. Probably.

@erozqba
Copy link
Contributor

erozqba commented Mar 5, 2018

@kousu Great analysis! Could you provide a patch to fix this?

@JeanFrancoisFournier
Copy link
Contributor

Before hacking away at this issue, I'd love to check in with you Nick about that. Whenever you're in the office

@kousu
Copy link
Author

kousu commented Mar 7, 2018

To be clear, this causes generateorders to fail to run (usually) completely. This means that orders in the future may be missing. The current configuration generates 100 days in the future, so failing on one day may not be a problem because orders will have already been generated by a previous run -- for example, I saw this fail Friday and Saturday, but work again on Sunday -- but it might bite us by surprise later.

@erozqba
Copy link
Contributor

erozqba commented Apr 8, 2018

@kousu @JeanFrancoisFournier any information in this issue? It's not clear to me what should be the solution.

@kousu
Copy link
Author

kousu commented Apr 16, 2018 •
edited

@erozqba this got down-prioritized on our end because the daily workflow involves regenerated all the orders for the day, fresh, so we never run into an order made by generateorders except when looking ahead in the calendar for prediction.

The immediate solution here would be, I think, adding some other try-except clauses in judicious places. I haven't sat down to do this but I would like to, unless someone beats me to it. I kind of doubt that will happen because reproducing this is tricky without access to our production database; it's really not clear to me what the root cause here is; some combination of cancelling some portion of repeated orders and then regenerating them, maybe? Without working that out the best I can think to do is to try to patch this in production, which I'm not suuuuper keen on, and then extract the patch and push it up here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kousu @erozqba @JeanFrancoisFournier