Funnel returns TaskLog instead of ExecutorLog

[uniqueg]

Description

Trying the example on py-tes, version 0.4.0 with a local installation of the latest commit of Funnel (b906ba8) running on http://localhost:8000 results in a TypeError when calling cli.get_task(task_id), suggesting that the response provided by Funnel does not conform to TES specs.

This assumes that py-tes is basing its validation/unmarshalling on the latest version of the specs. At a cursory glance, it does. If it doesn't, the issue applies to py-tes instead.

Expected outcome

For the example call, .get_task(task_id) should return a response object corresponding to the TesTask model in the TES specification without throwing an error.

Actual outcome

First error in stack:

TypeError: ("'logs' must be a list of <class 'tes.models.ExecutorLog'> (got [TaskLog(start_time=datetime.datetime(2020, 7, 8, 21, 17, 36, 222942, tzinfo=tzoffset(None, 7200)), end_time=None, metadata=None, logs=None, outputs=None, system_logs=None)] that is a list of <class 'tes.models.TaskLog'>).", Attribute(name='logs', default=None, validator=<optional validator for _ListOfValidator(type=<class 'tes.models.ExecutorLog'>) or None>, repr=True, eq=True, order=True, hash=None, init=True, metadata=mappingproxy({}), type=None, converter=None, kw_only=False), <class 'tes.models.ExecutorLog'>, [TaskLog(start_time=datetime.datetime(2020, 7, 8, 21, 17, 36, 222942, tzinfo=tzoffset(None, 7200)), end_time=None, metadata=None, logs=None, outputs=None, system_logs=None)])

Entire traceback:

Traceback (most recent call last):
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/tes/utils.py", line 80, in unmarshal
    output = o(**r)
  File "<attrs generated init tes.models.TaskLog>", line 12, in __init__
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/attr/validators.py", line 191, in __call__
    self.validator(inst, attr, value)
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/tes/models.py", line 30, in __call__
    attr, self.type, value,
TypeError: ("'logs' must be a list of <class 'tes.models.ExecutorLog'> (got [TaskLog(start_time=datetime.datetime(2020, 7, 8, 21, 17, 36, 222942, tzinfo=tzoffset(None, 7200)), end_time=None, metadata=None, logs=None, outputs=None, system_logs=None)] that is a list of <class 'tes.models.TaskLog'>).", Attribute(name='logs', default=None, validator=<optional validator for _ListOfValidator(type=<class 'tes.models.ExecutorLog'>) or None>, repr=True, eq=True, order=True, hash=None, init=True, metadata=mappingproxy({}), type=None, converter=None, kw_only=False), <class 'tes.models.ExecutorLog'>, [TaskLog(start_time=datetime.datetime(2020, 7, 8, 21, 17, 36, 222942, tzinfo=tzoffset(None, 7200)), end_time=None, metadata=None, logs=None, outputs=None, system_logs=None)])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/tes/utils.py", line 70, in unmarshal
    field = _unmarshal(v, obj)
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/tes/utils.py", line 58, in _unmarshal
    field.append(unmarshal(item, obj))
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/tes/utils.py", line 85, in unmarshal
    raise UnmarshalError(msg)
tes.utils.UnmarshalError: {'logs': [{'startTime': '2020-07-08T21:17:36.222942471+02:00'}], 'metadata': {'hostname': 'dell-7390'}, 'startTime': '2020-07-08T21:17:36.21283682+02:00'} could not be unmarshalled to type: TaskLog
TypeError: ("'logs' must be a list of <class 'tes.models.ExecutorLog'> (got [TaskLog(start_time=datetime.datetime(2020, 7, 8, 21, 17, 36, 222942, tzinfo=tzoffset(None, 7200)), end_time=None, metadata=None, logs=None, outputs=None, system_logs=None)] that is a list of <class 'tes.models.TaskLog'>).", Attribute(name='logs', default=None, validator=<optional validator for _ListOfValidator(type=<class 'tes.models.ExecutorLog'>) or None>, repr=True, eq=True, order=True, hash=None, init=True, metadata=mappingproxy({}), type=None, converter=None, kw_only=False), <class 'tes.models.ExecutorLog'>, [TaskLog(start_time=datetime.datetime(2020, 7, 8, 21, 17, 36, 222942, tzinfo=tzoffset(None, 7200)), end_time=None, metadata=None, logs=None, outputs=None, system_logs=None)])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/tes/utils.py", line 80, in unmarshal
    output = o(**r)
TypeError: __init__() got an unexpected keyword argument 'logs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/tes/client.py", line 75, in get_task
    return unmarshal(response.json(), Task)
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/tes/utils.py", line 73, in unmarshal
    field = _unmarshal(v, obj)
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/tes/utils.py", line 58, in _unmarshal
    field.append(unmarshal(item, obj))
  File "/home/uniqueg/Software/miniconda3/envs/snakemake/lib/python3.6/site-packages/tes/utils.py", line 85, in unmarshal
    raise UnmarshalError(msg)
tes.utils.UnmarshalError: {'logs': [{'startTime': '2020-07-08T21:17:36.222942471+02:00'}], 'metadata': {'hostname': 'dell-7390'}, 'startTime': '2020-07-08T21:17:36.21283682+02:00'} could not be unmarshalled to type: ExecutorLog
TypeError: __init__() got an unexpected keyword argument 'logs'

Steps to reproduce

Start Funnel:

funnel server run

Send task and check status:

import tes

task = tes.Task(
    executors=[
        tes.Executor(
            image="alpine",
            command=["echo", "hello"]
        )
    ]
)

cli = tes.HTTPClient("http://localhost:8000")
task_id = cli.create_task(task)
res = cli.get_task(task_id)  # Here the error occurs

Note that the error does not occur if view='MINIMAL' is passed to .get_task(); only views BASIC and FULL cause the error to be raised.

Environment

• Ubuntu 20.04
• Funnel; installed from commit b906ba8 as per instructions
• py-tes v0.4.0; installed via pip

[kellrott]

This may be related to an issue py-tes was having is misidentifying message types
Can you see if ohsu-comp-bio/py-tes#29 helps?

[uniqueg]

Thanks a lot Kyle, I'll have a look and report back.

[uniqueg]

Can confirm that ohsu-comp-bio/py-tes#29 fixes this issue.

[kellrott]

Thank you for the report. I'll work on getting that PR merged and pushed into a new PyPI release