-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error viewing logs in UI #2
Comments
@saritatewari4 can you please share some additional info?
|
Hello @snjypl , Please find details below: Python: 3.9.6 Airflow configuration for logging as below:
Connection hook details:
These line of code I appended in log_config.py inside config folder:
Logs in airflow ui trying to load but never loaded . Getting this error on inspecting the code in ui console: In grafana loki running on localhost getting logs: `
` I am using s3 bucket as backend for loki storage but switching to in-memory storage also I tried and found logs are not loading in airflow ui. These are the airflow logs in terminal:
NOTE: Logs are present in grafana loki running in local but not loading in airflow ui. |
@saritatewari4
restart your websever. can try to retrieve the log from airflow ui. in the websever log you should see the loki query params:
also, verify that the logs are having proper labels in loki. |
Hi @snjypl , I updated the version to latest dev as you mentioned but in airflow still logs are not loading. Logs are having labels in loki: These are the logs in airflow console:
|
@saritatewari4 the webserver worker is exiting due to error
can you check if you are getting this issue with fileloghandler? just to make sure the issue is with the loki handler. |
@snjypl , I am not getting any error when setting remote logging to false. could you please elaborate what I should use in case of fileloghandler. |
@saritatewari4 can you try this gist. it queries the loki for airflow log. wanted to verify that there is no issue retrieving logs from loki. you will have to update the loki credentails in the code. |
@snjypl , This is the output when I ran this:
|
@saritatewari4 Can you please share the complete websever log? you can restart the webserver and access the log from UI, and share the whole log, it will help in debugging. also the airflow webserver config, worker class etc. |
@snjypl , Please check the logs below: |
@saritatewari4 i meant the airflow logs in the terminal. you had share part of it previously. please share the complete logs. |
also, which webserver worker class are you using? is it sync, gevent or eventlet? i will be able to see it in the start of the airflow webserver log. |
Please get the logs below:
|
@snjypl . worker_class is set as sync |
@saritatewari4 thanks, there is a known issue with 'gevent' worker class, but since you are using sync worker class that shouldn't be a problem. locally, i tested it on i have pushed some changes with extra logging to
|
@snjypl , which version of python and airflow you are using? |
Hello @snjypl , To quickly spin up grafana and loki in local I have uploaded the composer file here Please use this command this command :
Script to install airflow as standalone: script Airflow custom logging folder Airflow configuration for logging as below:
Loki connection hook details:
Installed airflow grafana provider as follows: pip install --force git+https://github.com/snjypl/airflow-provider-grafana-loki.git@debug/issue-2-webserver-signal-11-on-log-read --no-dep Please let me know where I am missing the configuration. |
@saritatewari4 can you please share the latest logs? |
thanks @saritatewari4 i will try to replicate your environment, in the meanwhile, can you please share the terminal logs while viewing the logs in UI |
@snjypl , Please check the logs: airflow standalone |
where you seeing this error previously? |
I could see these error lines in previous run also in terminal. @snjypl I am running this in mac machine (JFYI). Could you please share your setup details. |
python 3.8 |
the issue could be because of that error. please stop the run and try to access the task log in airflow ui. it will give you a cleaner error logs in terminal . please share the log. later maybe you can also try a clean installation of airflow and see if you are getting the error. |
the following commands would help you in doing a db reset. backup airflow connections |
@snjypl , when I ran airflow webserver -d , I was able to see the logs in ui. Logs you can check here: error!="JSONParserErr"\n ', 'start': '2022-11-14T16:29:26.099527+00:00', 'end': '2022-11-29T17:29:26.279744+00:00', 'direction': 'forward'} |
@saritatewari4 later when i get a chance, i will try to reproduce your issue. |
the links are broken. can you please check them |
@snjypl , I am trying to run them severalty using docker composer. I will update you. Thanks a lot for your prompt response and support. |
Please use this repo. All the files are updated here. I will update the read me file. |
@saritatewari4 is it a private repo? https://github.com/saritatewari4/airflow-loki |
@saritatewari4 i have opened another issue to debug the airflow standalone mode issue: will close this issue, |
You are welcome !! |
@snjypl Please try to access that repo now. |
Hello @snjypl ,
I used this plugin with airflow 2.4.2 version. It worked fine. I am able to see logs in loki UI. I couldn't see the logs in airflow ui for dags and task when using remote logging as loki.
How I can get those logs from loki to airflow ui.
Please help me to get the solution.
The text was updated successfully, but these errors were encountered: