500 Internal Server Error

[beukueb]

Running the tutorial wdl hello raises a 500 Internal Server Error
This is quite uniformative for someone trying out the tool, I tried it both on a mac and from a sagemaker instance.
In both cases runnning on an already existing vpc. workflow and context are defined

agc workflow run hello --context myContext
2021-09-29T15:38:49+02:00 𝒊 Running workflow. Workflow name: 'hello', Arguments: '', Context: 'myContext'
2021-09-29T15:38:51+02:00 ✘ error="unable to run workflow: 500 Internal Server Error"
Error: an error occurred invoking 'workflow run'
with variables: {WorkflowName:hello Arguments: ContextName:myContext}
caused by: unable to run workflow: 500 Internal Server Error
suggestion: check that your workflow and context are defined for this project

[jaamarks]

@beukueb, did you ever figure out what was causing this error? I too saw this error when trying to run the "read" workflow.

[wleepang]

Can you try running with a -v flag and provide the output here?

[jaamarks]

I tried to run it a couple different ways to provide more context.

$ cd ~/agc/examples/demo-wdl-project/workflows/read/
$ agc context status
INSTANCE        spotCtx         STARTED true


$ agc workflow  run  read --context spotCtx -v -a read.inputs.json

2021-10-07T18:11:31Z �  Running workflow. Workflow name: 'read', Arguments: 'read.inputs.json', Context: 'spotCtx'
2021-10-07T18:11:31Z ✘   error="unable to run workflow: couldn't read file read.inputs.json: open read.inputs.json: no such file or directory"
Error: an error occurred invoking 'workflow run'
with variables: {WorkflowName:read Arguments:read.inputs.json ContextName:spotCtx}
caused by: unable to run workflow: couldn't read file read.inputs.json: open read.inputs.json: no such file or directory




$ agc workflow  run  read --context spotCtx -v -a workflows/read/read.inputs.json

2021-10-07T18:15:36Z �  Running workflow. Workflow name: 'read', Arguments: 'workflows/read/read.inputs.json', Context: 'spotCtx'
2021-10-07T18:15:36Z ↓  Skipping upload for '/home/ec2-user/agc/examples/demo-wdl-project/workflows/read/hello.txt', files already exist at 's3://agc-404545384114-us-east-1/project/Demo/userid/jmarkstf7cw/data/ReadFile.input_file/hello.txt'
2021-10-07T18:15:36Z ↓  EstablishWesConnection(https://6bqbik694a.execute-api.us-east-1.amazonaws.com/prod/ga4gh/wes/v1)
2021-10-07T18:15:36Z ↓  Ping WES endpoint until we get an answer:
2021-10-07T18:15:36Z ↓  Attempt 1 of 3
2021-10-07T18:15:36Z ↓  Connected to WES endpoint
2021-10-07T18:15:37Z ✘   error="unable to run workflow: 500 Internal Server Error"
Error: an error occurred invoking 'workflow run'
with variables: {WorkflowName:read Arguments:workflows/read/read.inputs.json ContextName:spotCtx}
caused by: unable to run workflow: 500 Internal Server Error

[wleepang]

@jaamarks - The error you're getting is likely an issue with the read.inputs.json file. You can diagnose this by getting the adapter logs:

agc logs adapter -c spotCtx

If this was a while ago, you might need provide a longer look-back window - e.g. -l 12h

[jaamarks]

Thank you, that command was helpful. The issue was not having either a MANIFEST.json file or a main.wdl—as detailed here https://aws.github.io/amazon-genomics-cli/docs/concepts/workflows/

I added a MINIFEST.json and it ran successfully.

{
    "mainWorkflowURL": "read.wdl",
    "inputFileURLs": [
        "read.inputs.json"
    ]
}

[markjschreiber]

Although using a MANIFEST is one way to make this work, I do think the command should have worked without it. Labeling this as a bug and investigating.

[markjschreiber]

Error occurs in the WES Adapter. As the workflow is packaged into a zip the adapter expects the workflow to be named main.wdl if there is no manifest.

The specific name of the WDL is not passed through.

[Mon, 11 Oct 2021 12:42:30 -0400	[2021-10-11 16:42:30,106] [RemoteWESAdapter] [DEBUG] RUN_WORKFLOW :: wes service is healthy]
[Mon, 11 Oct 2021 12:42:30 -0400	[2021-10-11 16:42:30,107] [RemoteWESAdapter] [DEBUG] RUN_WORKFLOW :: tmpdir=/tmp/tmphojbf6jx]
[Mon, 11 Oct 2021 12:42:30 -0400	[2021-10-11 16:42:30,107] [RemoteWESAdapter] [DEBUG] RUN_WORKFLOW :: urlparse(workflow_url)=ParseResult(scheme='s3', netloc='agc-581254785006-us-east-1', path='/project/Demo/userid/mrschre4GqyMA/context/spotCtx/workflow/read/workflow.zip', params='', query='', fragment='')]
[Mon, 11 Oct 2021 12:42:30 -0400	[2021-10-11 16:42:30,107] [RemoteWESAdapter] [DEBUG] RUN_WORKFLOW :: retrieving 's3://agc-581254785006-us-east-1/project/Demo/userid/mrschre4GqyMA/context/spotCtx/workflow/read/workflow.zip' => /tmp/tmphojbf6jx]
[Mon, 11 Oct 2021 12:42:30 -0400	[2021-10-11 16:42:30,260] [wes] [ERROR] Exception on /ga4gh/wes/v1/runs [POST]]
[Mon, 11 Oct 2021 12:42:30 -0400	Traceback (most recent call last):]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/rosalind/wes/adapters/RemoteWESAdapter.py", line 430, in get_workflow_from_s3]
[Mon, 11 Oct 2021 12:42:30 -0400	    props = parse_workflow_zip_file(file, workflow_type)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/rosalind/wes/adapters/RemoteWESAdapter.py", line 494, in parse_workflow_zip_file]
[Mon, 11 Oct 2021 12:42:30 -0400	    raise RuntimeError(f"'main.{workflow_type}' file not found")]
[Mon, 11 Oct 2021 12:42:30 -0400	RuntimeError: 'main.WDL' file not found]
[Mon, 11 Oct 2021 12:42:30 -0400	During handling of the above exception, another exception occurred:]
[Mon, 11 Oct 2021 12:42:30 -0400	Traceback (most recent call last):]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/rosalind/wes/adapters/RemoteWESAdapter.py", line 223, in run_workflow]
[Mon, 11 Oct 2021 12:42:30 -0400	    props = get_workflow_from_s3(workflow_url, tmpdir, workflow_type)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/rosalind/wes/adapters/RemoteWESAdapter.py", line 432, in get_workflow_from_s3]
[Mon, 11 Oct 2021 12:42:30 -0400	    raise RuntimeError(f"{s3_uri} is not a valid workflow.zip file: {e}")]
[Mon, 11 Oct 2021 12:42:30 -0400	RuntimeError: s3://agc-581254785006-us-east-1/project/Demo/userid/mrschre4GqyMA/context/spotCtx/workflow/read/workflow.zip is not a valid workflow.zip file: 'main.WDL' file not found]
[Mon, 11 Oct 2021 12:42:30 -0400	During handling of the above exception, another exception occurred:]
[Mon, 11 Oct 2021 12:42:30 -0400	Traceback (most recent call last):]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request]
[Mon, 11 Oct 2021 12:42:30 -0400	    rv = self.dispatch_request()]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request]
[Mon, 11 Oct 2021 12:42:30 -0400	    return self.view_functions[rule.endpoint](**req.view_args)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/decorators/decorator.py", line 68, in wrapper]
[Mon, 11 Oct 2021 12:42:30 -0400	    response = function(request)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper]
[Mon, 11 Oct 2021 12:42:30 -0400	    response = function(request)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/decorators/validation.py", line 193, in wrapper]
[Mon, 11 Oct 2021 12:42:30 -0400	    response = function(request)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/decorators/parameter.py", line 115, in wrapper]
[Mon, 11 Oct 2021 12:42:30 -0400	    return function(**kwargs)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/wes/controllers/workflow_execution_service_controller.py", line 145, in run_workflow]
[Mon, 11 Oct 2021 12:42:30 -0400	    return adapter.run_workflow(**args)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/rosalind/wes/adapters/RosalindWESAdapter.py", line 187, in run_workflow]
[Mon, 11 Oct 2021 12:42:30 -0400	    return self.adapter.run_workflow(**args)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/rosalind/wes/adapters/RemoteWESAdapter.py", line 225, in run_workflow]
[Mon, 11 Oct 2021 12:42:30 -0400	    raise InvalidRequestError(e)]
[Mon, 11 Oct 2021 12:42:30 -0400	wes.exception.Exceptions.InvalidRequestError: 400 Bad Request: s3://agc-581254785006-us-east-1/project/Demo/userid/mrschre4GqyMA/context/spotCtx/workflow/read/workflow.zip is not a valid workflow.zip file: 'main.WDL' file not found]
[Mon, 11 Oct 2021 12:42:30 -0400	During handling of the above exception, another exception occurred:]
[Mon, 11 Oct 2021 12:42:30 -0400	Traceback (most recent call last):]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app]
[Mon, 11 Oct 2021 12:42:30 -0400	    response = self.full_dispatch_request()]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request]
[Mon, 11 Oct 2021 12:42:30 -0400	    rv = self.handle_user_exception(e)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/flask/app.py", line 1816, in handle_user_exception]
[Mon, 11 Oct 2021 12:42:30 -0400	    return self.handle_http_exception(e)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/flask/app.py", line 1744, in handle_http_exception]
[Mon, 11 Oct 2021 12:42:30 -0400	    return handler(e)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/apps/flask_app.py", line 62, in common_error_handler]
[Mon, 11 Oct 2021 12:42:30 -0400	    return FlaskApi.get_response(response)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/apis/flask_api.py", line 149, in get_response]
[Mon, 11 Oct 2021 12:42:30 -0400	    return cls._get_response(response, mimetype=mimetype, extra_context={"url": flask.request.url})]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/apis/abstract.py", line 288, in _get_response]
[Mon, 11 Oct 2021 12:42:30 -0400	    framework_response = cls._connexion_to_framework_response(response, mimetype, extra_context)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/apis/flask_api.py", line 170, in _connexion_to_framework_response]
[Mon, 11 Oct 2021 12:42:30 -0400	    flask_response = cls._build_response(]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/apis/flask_api.py", line 186, in _build_response]
[Mon, 11 Oct 2021 12:42:30 -0400	    data, status_code, serialized_mimetype = cls._prepare_body_and_status_code(data=data, mimetype=mimetype, status_code=status_code, extra_context=extra_context)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/apis/abstract.py", line 417, in _prepare_body_and_status_code]
[Mon, 11 Oct 2021 12:42:30 -0400	    body, mimetype = cls._serialize_data(data, mimetype)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/apis/flask_api.py", line 203, in _serialize_data]
[Mon, 11 Oct 2021 12:42:30 -0400	    body = cls.jsonifier.dumps(data)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/jsonifier.py", line 57, in dumps]
[Mon, 11 Oct 2021 12:42:30 -0400	    return self.json.dumps(data, **kwargs) + '\n']
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/flask/json/__init__.py", line 211, in dumps]
[Mon, 11 Oct 2021 12:42:30 -0400	    rv = _json.dumps(obj, **kwargs)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/usr/lib64/python3.8/json/__init__.py", line 234, in dumps]
[Mon, 11 Oct 2021 12:42:30 -0400	    return cls(]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/usr/lib64/python3.8/json/encoder.py", line 201, in encode]
[Mon, 11 Oct 2021 12:42:30 -0400	    chunks = list(chunks)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/usr/lib64/python3.8/json/encoder.py", line 431, in _iterencode]
[Mon, 11 Oct 2021 12:42:30 -0400	    yield from _iterencode_dict(o, _current_indent_level)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/usr/lib64/python3.8/json/encoder.py", line 405, in _iterencode_dict]
[Mon, 11 Oct 2021 12:42:30 -0400	    yield from chunks]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/usr/lib64/python3.8/json/encoder.py", line 438, in _iterencode]
[Mon, 11 Oct 2021 12:42:30 -0400	    o = _default(o)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/wes/encoder.py", line 20, in default]
[Mon, 11 Oct 2021 12:42:30 -0400	    return FlaskJSONEncoder.default(self, o)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/connexion/apps/flask_app.py", line 146, in default]
[Mon, 11 Oct 2021 12:42:30 -0400	    return json.JSONEncoder.default(self, o)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/venv/lib64/python3.8/site-packages/flask/json/__init__.py", line 100, in default]
[Mon, 11 Oct 2021 12:42:30 -0400	    return _json.JSONEncoder.default(self, o)]
[Mon, 11 Oct 2021 12:42:30 -0400	  File "/usr/lib64/python3.8/json/encoder.py", line 179, in default]
[Mon, 11 Oct 2021 12:42:30 -0400	    raise TypeError(f'Object of type {o.__class__.__name__} ']

The name of the WDL is not passed through because the declaration in the agc-project.yaml is:

  read:
    type:
      language: wdl
      version: 1.0
    sourceURL: workflows/read

Potential fixes, update the name of the workflow to main.wdl or specify the workflow name rather than folder in the project spec.

[markjschreiber]

With the following change to the agc-project.yaml the workflow will run with agc workflow run read -c spotCtx -a workflows/read/read.inputs.json

Due to a current limitation with path resolution of -a arguments being relative to the directory of the agc-project.yaml. This is being tracked in #83

  read:
    type:
      language: wdl
      version: 1.0
    sourceURL: workflows/read/read.wdl

[markjschreiber]

Closing as the primary issue is resolved by PR #86