[Feature][framework] Users be able to download pipeline logging archives from config-ui

[klesh]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Objectives

1. Users should be able to set the path of a folder for logging file storage, by Environment Variable or .env file, namely LOGGING_DIR
2. All logging files of a specific pipeline should be placed in one folder, i.e. PIPELINE_LOGGING_DIR=$LOGGING_DIR/pipeline-$PIPELINE_ID-$PIPELINE_CREATED_AT
3. All logging files of tasks belongs to the pipeline should be in the pipeline logging dir, i.e. $PIPELINE_LOGGING_DIR/task-$TASK_ID-$ROW-$COL-$PLUGIN.log
4. Provide an API endpoint /pipelines/:pipeline_id/logging.tar.gz for user to download the archived logging folder of the pipeline

@keon94 @e2corporation

Use case

Users can download the logging archive and send it to us for debugging

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[keon94]

@klesh Should the scope of the log files be just be INFO/WARN/ERROR/DEBUG logs that happen inside of RunPluginSubTasks()? Also, do we want the log file to grow indefinitely in size over time, or should we implement any log-file-rotation policy?

[klesh]

e log file to grow indefinitely in size over time, or should we implement any log-file-rotation polic
@keon94
Each pipeline has its own folder and each task has its own log file, because tasks can run in parallel, we have to separate them for debugging's sake. I ideally, we should record every single log for the pipeline including the pipeline log, and task log(task log and subtasks logs).

The logging level should be determined by the setting (from .env or Environment Variable).

So, in this way, we shouldn't have any ever-growing logging files but folders. and due to the fact, that our logging messages are quite brief, log-file-rotation is not a hard requirement at the point. But, you may propose your solution to that end.

[Startrekzky]

Hi @keon94 , when can I test it?

[keon94]

Hi @keon94 , when can I test it?

The functional work is done, so you can test on the branch itself for now if you want. We're just discussing backend internal implementation details at this point.