Use QueueHandler for Pipeline logging
#489
Conversation
| @@ -72,8 +70,8 @@ def generate( | |||
| per input in `inputs`.""" | |||
| pass | |||
|
|
|||
| @cached_property | |||
| def generate_parameters(self) -> List[inspect.Parameter]: | |||
| @property | |||
There was a problem hiding this comment.
Then the from functools import cached_property can be removed too, right?
There was a problem hiding this comment.
yes! BTW I removed it because in some environments pickle was not able to serialize it.
| @@ -45,6 +45,8 @@ class GenerateEmbeddings(Step): | |||
| llm: LLM | |||
|
|
|||
| def load(self) -> None: | |||
| """Loads the `LLM` used to generate the embeddings.""" | |||
| super().load() | |||
There was a problem hiding this comment.
If all the steps inheriting from Step need to actually call the super().load() isn't it better if we just add it as a separate method that runs within the post_init? i.e. model_post_init on _Step?
There was a problem hiding this comment.
Sadly we can't do that :( We need to create the logger after having called setup_logging in each process.
There was a problem hiding this comment.
Can we then just do step.load() and then step.setup_logging? But to manage that within the Pipeline instead of having to always add that as part of the load method :/
Description
This PR updates the logging setup for the local
Pipelineto use aQueueHandlerand aQueueListener(a thread in the main process). With this setup, child processes send log messages using amultiprocessing.Queueand theQueueHandlerto theQueueListener, which will be in charge of handling the log messages.This change was mainly motivated to improve the logging messages in environments like Google Colab and IPython Notebooks, in which not all the logging messages (from the child processes) were being displayed.