Subbasin Timing Out on Staging

[rajadain]

Doesn't time out locally. Upon inspecting the Celery logs, I see the last entry in the log is:

[2018-04-30 16:45:19,173: INFO/MainProcess] Received task: apps.modeling.tasks.run_subbasin_gwlfe_chunks[9c3ab6c8-1da0-42b7-917e-f964c82cf1a7]

After which there is silence. Restarting Celery did not help. gwlf-e 0.6.3 is installed on the machine.

Find out what's wrong and make it work again.

[rajadain]

Okay here’s the issue (as far as I can tell today):

1. stream_lengths here:

   model-my-watershed/src/mmw/apps/modeling/views.py

   Line 256 in 88b838f

   stream_lengths = sum_subbasin_stream_lengths(model_input)

   is a typed class that looks like this:

StreamLengthSummary(ag=21255.097623364967, urban=416954.90237663506)

2. When it is used here:

   model-my-watershed/src/mmw/apps/modeling/calcs.py

   Line 95 in 88b838f

   ag_pct_total_stream_length = gms['AgLength'] / total_stream_lengths['ag']

   , on my local, it appears as a dict:

{'urban': 416954.90237663506, 'ag': 21255.097623364967}

3. When it is used in the same spot on staging, it appears as a list:

[21255.097623364967, 416954.90237663506]

which causes the named lookup to fail.

Now I’m investigating at what point this mistranslation happens, so we can fix it. Notably, the class is composed in the app VM, and parsed in the worker VM. Celery has had trouble deserializing classes in the past, but I don’t know why we’re seeing different behaviors in the two environments.