Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add error messages to report.csv #103

Closed
jeremyfowers opened this issue Feb 8, 2024 · 0 comments · Fixed by #111
Closed

Add error messages to report.csv #103

jeremyfowers opened this issue Feb 8, 2024 · 0 comments · Fixed by #111
Assignees
Labels
bug Something isn't working p2 Low priority
Milestone

Comments

@jeremyfowers
Copy link
Collaborator

Current behavior:

  • Error messages are logged into *_log.txt files that correspond to the Stages and Runtimes that execute.
  • turnkey cache report produces a report.csv file that includes status (e.g., error), but does not provide a convenient way to find out what the error was.
  • Users are responsible for acquiring the logs and correlating them with the report to learn more about each error.

Proposed solution: if a function fails, include the contents of its logfile in the report CSV file

Potential problem: this might make the CSV file absurdly large on some datasets. Not sure if that is a problem or not, since we could handle a 1 GB CSV file if needed. Perhaps this could be mitigated by only storing the last 10 lines of the log into the CSV.

cc @danielholanda @ramkrishna2910

@jeremyfowers jeremyfowers added bug Something isn't working p2 Low priority labels Feb 8, 2024
@jeremyfowers jeremyfowers added this to the Data Quality milestone Feb 8, 2024
@jeremyfowers jeremyfowers linked a pull request Feb 12, 2024 that will close this issue
@jeremyfowers jeremyfowers self-assigned this Feb 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working p2 Low priority
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant