Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

some workflow seems to fail #1147

Closed
goern opened this issue Sep 26, 2022 · 2 comments · Fixed by thoth-station/workflow-helpers#424
Closed

some workflow seems to fail #1147

goern opened this issue Sep 26, 2022 · 2 comments · Fixed by thoth-station/workflow-helpers#424
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.

Comments

@goern
Copy link
Member

goern commented Sep 26, 2022

Bug description

2022-09-26 08:15:31,014 1 WARNING thoth.common:340: Logging to a Sentry instance is turned off
2022-09-26 08:15:31,423 1 INFO thoth.run_kebechet_administrator:52: Thoth workflow-helpers task: run_kebechet_administrator v0.9.13+thamos.1.27.11.common.0.36.4.python.0.16.10analyzer.0.1.8.storages.0.73.3
2022-09-26 08:15:32,122 1 INFO thoth.run_kebechet_administrator:201: Kebechet administrator triggered by: thoth.solver.solved-package
2022-09-26 08:15:33,128 1 INFO thoth.run_kebechet_administrator:205: Number of messages to be sent: 0
2022-09-26 08:15:33,128 1 ERROR thoth.workflow_helpers.common:81: [Errno 2] No such file or directory: '/mnt/workdir/messages_to_be_sent.json'
Traceback (most recent call last):
File "/opt/app-root/src/thoth/workflow_helpers/common.py", line 74, in store_messages
with open(MSG_OUT_FILE, "r") as json_file:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/workdir/messages_to_be_sent.json'

Steps to Reproduce

Steps to reproduce the behavior:

  1. Go to https://console-openshift-console.apps.ocp4.prod.psi.redhat.com/k8s/ns/thoth-backend-stage/pods
  2. check the log of the main container of one of the kebechet-admin pods
  3. See error

Actual behavior

n/a

Expected behavior

n/a

Additional context

n/a

/kind bug
/priority critical-urgent
/assign @KPostOffice
/assign @VannTen

@goern goern added the kind/bug Categorizes issue or PR as related to a bug. label Sep 26, 2022
@sesheta sesheta added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Sep 26, 2022
@VannTen
Copy link
Member

VannTen commented Sep 27, 2022

From the log of the wait containers in those pods and the annotations it looks like this messages_to_be_sent.json is the output of that pod.

Together with the fact that the number of messages to be sent is 0 maybe the process tried to reread the file it writes, and fail when it did not write it because there was no messages ?

I'll take a look.

@VannTen
Copy link
Member

VannTen commented Sep 27, 2022

Hum, it's not as bad as it looks.
We just log it wrong :

    # Store message to file that need to be sent.
    try:
        with open(MSG_OUT_FILE, "r") as json_file:
            if os.stat(MSG_OUT_FILE).st_size != 0:
                all_messages: list = json.load(json_file)
                if type(all_messages) != list:
                    raise TypeError(f"Message file must be a list of messages. Got type {type(all_messages)}")
                all_messages = all_messages + output_messages
    except Exception as e:
        _LOGGER.exception(e)
        all_messages = output_messages

_LOGGER.exception is _LOGGER.error with extra stuff. This should probably be more specific and not and error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants