Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide string prefix/identifier in connector sync logs to distinguish log messages written by source/destination vs written by Airbyte #5889

Closed
cjwooo opened this issue Sep 7, 2021 · 2 comments
Assignees
Labels
area/platform issues related to the platform type/enhancement New feature or request

Comments

@cjwooo
Copy link
Contributor

cjwooo commented Sep 7, 2021

Tell us about the problem you're trying to solve

Our use case for Airbyte involves surfacing sync logs to our users. We only want to show the logs written by the actual source and destination, e.g.

2021-09-04 01:03:54 INFO  Read 25 records from comments stream
2021-09-04 01:03:54 INFO  Syncing stream: commit_comments 

and not the ones written by Airbyte, e.g.

2021-09-04 01:05:07 INFO  Closed all resources for pod source-github-worker-29-0-tcjov
2021-09-04 01:05:07 INFO  Closed all resources for pod source-github-worker-29-0-tcjov

Describe the solution you’d like

A procedure for distinguish logs written by source or destination vs written by Airbyte in the response given by calling api/v1/jobs/get.

Describe the alternative you’ve considered or used

A clear and concise description of any alternative solutions or features you've considered or are using today.

Additional context

Add any other context or screenshots about the feature request here.

Are you willing to submit a PR?

Remove this with your answer :-)

@cjwooo cjwooo added the type/enhancement New feature or request label Sep 7, 2021
@marcosmarxm
Copy link
Member

@cjwooo could be having the container name as an identifier helps? (source, destination, scheduler, server)

@marcosmarxm
Copy link
Member

@davinchia / @Phlair is it possible to estimate what is needed to change to implement this feature?

@benmoriceau benmoriceau self-assigned this Oct 19, 2021
@cgardens cgardens added this to the Platform 2021-10-27 milestone Oct 20, 2021
benmoriceau added a commit that referenced this issue Oct 22, 2021
This is preparing the tagging of the application that generate a log which is requested in #5889

It is adding helper to add a color to the log and safely manipulate the MDC.

It is also modifying the log4j pattern.

The use of those helper will will be implemented in another PR.
@cgardens cgardens added the area/platform issues related to the platform label Oct 25, 2021
benmoriceau added a commit that referenced this issue Oct 28, 2021
This is adding the application name in the logs produced by the docker containers.

This is related to #5889.
schlattk pushed a commit to schlattk/airbyte that referenced this issue Jan 4, 2022
This is preparing the tagging of the application that generate a log which is requested in airbytehq#5889

It is adding helper to add a color to the log and safely manipulate the MDC.

It is also modifying the log4j pattern.

The use of those helper will will be implemented in another PR.
schlattk pushed a commit to schlattk/airbyte that referenced this issue Jan 4, 2022
)

This is adding the application name in the logs produced by the docker containers.

This is related to airbytehq#5889.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform issues related to the platform type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants