Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle large log output #74

Closed
tchiotludo opened this issue Feb 27, 2020 · 0 comments · Fixed by #97
Closed

Handle large log output #74

tchiotludo opened this issue Feb 27, 2020 · 0 comments · Fixed by #97
Labels
backend Need some backend code changes bug Something isn't working

Comments

@tchiotludo
Copy link
Member

Seems that with Kafka runner, some very long line will kill a consuming worker silently.
No error will be in logs and the consumer is remove for Kafka silently.

Really need to investigate
@tchiotludo tchiotludo added bug Something isn't working backend Need some backend code changes labels Feb 27, 2020
@tchiotludo tchiotludo mentioned this issue Mar 1, 2020
Merged
tchiotludo added a commit that referenced this issue Mar 1, 2020
@tchiotludo
fix(runner-kafka): handle queue emit exception
903520f 
If an emit failed, before the error was silenced and the thread is removed silently.
Now, we try the save mark the WorkerTask has failed and if we can't we just kill the application

relate to #74, #77
tchiotludo added a commit that referenced this issue Mar 20, 2020
@tchiotludo
feat(core): remove log from Execution
c90c4d6 
Large log can break the Execution object.
Now we put the log a new queue and save it to repository.
This also allow log to be realtime.

close #74
close #34
@tchiotludo tchiotludo mentioned this issue Mar 20, 2020
Merged
tchiotludo added a commit that referenced this issue Mar 20, 2020
@tchiotludo
feat(core): remove log from Execution
3cfeca8 
Large log can break the Execution object.
Now we put the log a new queue and save it to repository.
This also allow log to be realtime.

close #74
close #34
tchiotludo added a commit that referenced this issue May 6, 2020
@tchiotludo
feat(core): remove log from Execution
a8ac72f 
Large log can break the Execution object.
Now we put the log a new queue and save it to repository.
This also allow log to be realtime.

close #74
close #34
tchiotludo added a commit that referenced this issue May 27, 2020
@tchiotludo
feat(core): remove log from Execution
6d2addc 
Large log can break the Execution object.
Now we put the log a new queue and save it to repository.
This also allow log to be realtime.

close #74
close #34
tchiotludo added a commit that referenced this issue Jul 6, 2020
@tchiotludo
feat(core): remove log from Execution
fdd40fd 
Large log can break the Execution object.
Now we put the log a new queue and save it to repository.
This also allow log to be realtime.

close #74
close #34
tchiotludo added a commit that referenced this issue Aug 14, 2020
@tchiotludo
feat(core): remove log from Execution
da1ab31 
Large log can break the Execution object.
Now we put the log a new queue and save it to repository.
This also allow log to be realtime.

close #74
close #34
tchiotludo added a commit that referenced this issue Aug 20, 2020
@tchiotludo @eregnier
feat(core): remove log from Execution and use a dedicated Queue / Rep…
86e0b79 
…ository (#97)

Large log can lead to  the Execution object.
Now we put the log line by line on a new queue and save it to dedicated repository.
This also allow log to be real time.

close #74
close #34

Co-authored-by: Eric Régnier <utopman@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
backend Need some backend code changes bug Something isn't working
Projects
No open projects
All issues
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove log from Execution kestra-io/kestra
1 participant
@tchiotludo