Airflow UI does not pick up TFX component logs #185
Comments
|
This is likely because the "CsvExampleGen" component you see in the web UI is actually a sub-DAG in Airflow, which includes four components (.check_cache, .exec, .publish and .publish_cache). I am not sure why the documented way of log investigation for sub dag did not work in our case. It may be related to how the sub-dag is created? cc + @ruoyu90 @krazyhaas |
|
Will this happen to all other components as well or is it just the Example Gen? Is there a way to change the config back to a normal DAG? |
|
We're looking into it. It's likely related to all components. |
|
I'll likely have a fix for this out tomorrow. I'd like another day to test the potential change. |
|
Awesome thank you. Happy to test myself, once you have it ready. 👍 |
|
Any update? |
|
Just waiting for approvals on the code review. |
|
Quick update -- we're having issues with an unrelated test that's blocking the checkin. If you want to patch the change into your code, it's literally a two-line change to enable this. |
|
I assume you are talking about #188? If so I see your changes and should be able to replicate them. Thanks :-) |
|
Done! You should now be able to see both the component/subcomponent logs as well as the actual nodes in the DAG viewer. Sorry about the delay -- getting the test updated and passing was actually more work than the code change itself. :) |
|
I can confirm that it works. Both logs appear in the UI. Thank you. |
|
@rummens Do you remember what was causing that error with not being able to open the database file? I'm getting that error now on the taxi example. |
|
Yes the path that I was using was just wrong or not accessible because of privilege problems. Change the metadb root variable and make sure the path is okay. |
|
Dang, that's not my problem. The path seems fine. It just starting doing that out of the blue. I'll keep digging, thanks. |
|
Changed permissions? Keep in mind that the airflow path might differ from your development path (except if you are developing in ~/airflow. |
|
Oh, wait, it looks like they changed it where you need to put the db name on the end of the path and not just pass the path. Hallelujah! Thanks! |
Summary
For some reason TFX writes the log files in a different format/path then expected by the Airflow Web UI. Therefore the logs on the UI don't show the really intressting logs (the root cause) but only that something went wrong (example s. below).
So for example, if the log root of Airflow is:
/usr/local/airflow/logsOfficial Log File
The log that appears on the UI is:
/usr/local/airflow/logs/chicago_taxi_simple/CsvExampleGen/2019-05-29T11:03:26.751866+00:00Which is follwing the offical format:
{dag_id}/{task_id}/{execution_date}/{try_number}.logActual TFX log file
But the relevant logs are here:
/usr/local/airflow/logs/chicago_taxi_simple.CsvExampleGen/chicago_taxi_simple.CsvExampleGen.checkcache/2019-05-29T11:03:26.751866+00:00This seems to follow a different format (dot instead of sub folder):
{dag_id}.{task_id}/{execution_date}/{try_number}.logQuestion
Is there a way to either change the AirFlow UI or TFX, so that the relevant log appears in the UI. I had a look at the LoggerConfig but it does not contain a setting to change anything but the root log folder.
Additional Information
Both a local airflow instance and one running on a K8s cluster have the same problem. The airflow version is v1.10.2. I am running the chicago example pipeline with the settings described in this repo.
Just to make clear what I mean. This is the log on the UI (no information about the actual error):
An this is the log with the real information (saying that the metadata store was unable to open database file):
The text was updated successfully, but these errors were encountered: