You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Lithops configures Python logging itself, which can be handy, but equally can make it hard to configure logging for other (non-Lithops) modules. For example, using logging.basicConfig in your code doesn't work well if Lithops has set up logging.
This issue is to discuss different ways to configure logging, and eventually add to the documentation and reflect in the examples.
1. Using Lithops logging setup
The default is to log at INFO level to the console (stderr). This is fine, but can be quite verbose, particularly for large workloads.
But you can also send Lithops logs to a file. This is a useful option as it keeps log information in case you want to debug a problem after the Lithops job has failed.
For example, set log_filename to lithops.log in your Lithops conifguration. Note that the log file will be appended too (and could therefore grow indefinitely). If you want something more sophisticated (such as rotating log file), then turn off Lithops logging setup, and configure logging yourself, as described below.
2. Configuring logging yourself
Another option is to turn off Lithops logging setup, by passing log_level=None to LithopsDagExecutor (which is forward to Lithops' FunctionExecutor). This tells Lithops not to configure logging, which means that you can configure logging yourself, and still get log messages from Lithops - it gives you full control, although you have to do a bit more work to get what you want.
Option 1: Set logging level globally to INFO with the given format. Note that Lithops will produce quite a lot of output
Lithops configures Python logging itself, which can be handy, but equally can make it hard to configure logging for other (non-Lithops) modules. For example, using
logging.basicConfig
in your code doesn't work well if Lithops has set up logging.This issue is to discuss different ways to configure logging, and eventually add to the documentation and reflect in the examples.
1. Using Lithops logging setup
The default is to log at INFO level to the console (stderr). This is fine, but can be quite verbose, particularly for large workloads.
But you can also send Lithops logs to a file. This is a useful option as it keeps log information in case you want to debug a problem after the Lithops job has failed.
For example, set
log_filename
tolithops.log
in your Lithops conifguration. Note that the log file will be appended too (and could therefore grow indefinitely). If you want something more sophisticated (such as rotating log file), then turn off Lithops logging setup, and configure logging yourself, as described below.2. Configuring logging yourself
Another option is to turn off Lithops logging setup, by passing
log_level=None
toLithopsDagExecutor
(which is forward to Lithops'FunctionExecutor
). This tells Lithops not to configure logging, which means that you can configure logging yourself, and still get log messages from Lithops - it gives you full control, although you have to do a bit more work to get what you want.Option 1: Set logging level globally to INFO with the given format. Note that Lithops will produce quite a lot of output
Optin 2: Set the logging level for Lithops to WARN, but other modules to INFO
Option 3: Send Lithops logs to a file, while sending everything else to the console
The text was updated successfully, but these errors were encountered: