Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(xmlupload): improve error handling of Timeout Error (DEV-2299) #411

Merged
merged 6 commits into from Jun 20, 2023

Conversation

jnussbaum
Copy link
Collaborator

resolves DEV-

@jnussbaum jnussbaum self-assigned this Jun 16, 2023
@jnussbaum jnussbaum changed the title chore: improve error handling (DEV-xxxx) chore(xmlupload): improve error handling of Timeout Error (DEV-2299) Jun 19, 2023
@linear
Copy link

linear bot commented Jun 19, 2023

DEV-2299 DSP-TOOLS: xmlupload: improve error handling of Timeout Error

Old behaviour in case of a timeout:

dsp-tools xmlupload testdata/xml-data/test-data-systematic.xml
...
2023-06-19T17:26:59.714574: Try reconnecting to DSP server, next attempt in 1 seconds...
2023-06-19T17:27:00.896490: Try reconnecting to DSP server, next attempt in 2 seconds...
2023-06-19T17:27:03.032554: Try reconnecting to DSP server, next attempt in 4 seconds...
2023-06-19T17:27:07.158967: Try reconnecting to DSP server, next attempt in 8 seconds...
2023-06-19T17:27:15.297954: Try reconnecting to DSP server, next attempt in 16 seconds...
2023-06-19T17:27:31.423494: Try reconnecting to DSP server, next attempt in 32 seconds...
2023-06-19T17:28:03.569195: Try reconnecting to DSP server, next attempt in 64 seconds...
WARNING: Unable to upload file 'testdata/bitstreams/test.mp4' of resource 'Moviething' (video_thing_1): Permanently unable to execute the network action. See logs for more details....
...
WARNING: Unable to create resource 'First Testthing' (test_thing_1): Target of link is not a IRI: "video_thing_1"
...
Upload the stashed resptrs...
  Upload resptrs of resource "test_thing_0"...
    WARNING: Unable to upload the resptr prop of 'testonto:hasTestThing' of resource 'test_thing_0'. Original error message: Invalid IRI: test_thing_1

==========================================
xmlupload must be aborted because of an error
The mapping of internal IDs to IRIs was written to ...
There are stashed text properties that could not be reapplied to the resources they were stripped from. They were saved to ...
There are stashed resptr properties that could not be reapplied to the resources they were stripped from. They were saved to ...
Independently of this error, there were some resources that could not be uploaded: ...
==========================================

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/Users/nussbaum/.vscode/extensions/ms-python.python-2023.10.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/Users/nussbaum/.vscode/extensions/ms-python.python-2023.10.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/Users/nussbaum/.vscode/extensions/ms-python.python-2023.10.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 317, in run_module
    run_module_as_main(options.target, alter_argv=True)
  File "/Users/nussbaum/.vscode/extensions/ms-python.python-2023.10.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 238, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nussbaum/.vscode/extensions/ms-python.python-2023.10.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/Users/nussbaum/Desktop/dsp-tools/src/dsp_tools/dsp_tools.py", line 470, in <module>
    main()
  File "/Users/nussbaum/Desktop/dsp-tools/src/dsp_tools/dsp_tools.py", line 459, in main
    success = call_requested_action(user_args=sys.argv[1:], parser=parser)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nussbaum/Desktop/dsp-tools/src/dsp_tools/dsp_tools.py", line 362, in call_requested_action
    success = xml_upload(
              ^^^^^^^^^^^
  File "/Users/nussbaum/Desktop/dsp-tools/src/dsp_tools/utils/xml_upload.py", line 605, in xml_upload
    _handle_upload_error(
  File "/Users/nussbaum/Desktop/dsp-tools/src/dsp_tools/utils/xml_upload.py", line 1115, in _handle_upload_error
    raise err
  File "/Users/nussbaum/Desktop/dsp-tools/src/dsp_tools/utils/xml_upload.py", line 600, in xml_upload
    raise BaseError(msg)
dsp_tools.models.exceptions.BaseError: Some stashed resptrs or XML texts could not be reapplied to their resources on the DSP server.

New behaviour in case of a timeout:

xmlupload testdata/xml-data/test-data-systematic.xml
...
2023-06-20T10:41:27.924901: Timeout Error: Try reconnecting to DSP server, next attempt in 1 seconds...
2023-06-20T10:41:29.258539: Timeout Error: Try reconnecting to DSP server, next attempt in 2 seconds...
2023-06-20T10:41:31.577019: Timeout Error: Try reconnecting to DSP server, next attempt in 4 seconds...
2023-06-20T10:41:35.909626: Timeout Error: Try reconnecting to DSP server, next attempt in 8 seconds...
2023-06-20T10:41:44.252310: Timeout Error: Try reconnecting to DSP server, next attempt in 16 seconds...
2023-06-20T10:42:00.574545: Timeout Error: Try reconnecting to DSP server, next attempt in 32 seconds...
2023-06-20T10:42:32.899971: Timeout Error: Try reconnecting to DSP server, next attempt in 64 seconds...
WARNING: Unable to upload file 'testdata/bitstreams/test.mp4' of resource 'Moviething' (video_thing_1): Permanently unable to execute the network action. See logs for more details.
...
WARNING: Unable to create resource 'First Testthing' (test_thing_1): Target of link is not a IRI: "video_thing_1"
...
Upload the stashed resptrs...
  Upload resptrs of resource "test_thing_0"...
    WARNING: Unable to upload the resptr prop of 'testonto:hasTestThing' of resource 'test_thing_0'. Original error message: Invalid IRI: test_thing_1

==========================================
xmlupload must be aborted because of an error.
Error message: 'Some stashed resptrs or XML texts could not be reapplied to their resources on the DSP server.'

The mapping of internal IDs to IRIs was written to ...

There are stashed text properties that could not be reapplied to the resources they were stripped from. They were saved to ...

There are stashed resptr properties that could not be reapplied to the resources they were stripped from. They were saved to ...

Independently of this error, there were some resources that could not be uploaded: ...

Copy link
Collaborator

@BalduinLandolt BalduinLandolt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

src/dsp_tools/utils/shared.py Outdated Show resolved Hide resolved
src/dsp_tools/utils/shared.py Outdated Show resolved Hide resolved
@jnussbaum jnussbaum merged commit b9cf8eb into main Jun 20, 2023
4 checks passed
@jnussbaum jnussbaum deleted the wip/streamline-err-handling branch June 20, 2023 09:28
@daschbot daschbot mentioned this pull request Jun 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants