[pyspark] rework the log

[wbo4958]

To fix #10065

It will show the log like below by default

2024-02-29 10:29:05,749 INFO XGBoost-PySpark: _fit Running xgboost-2.1.0-dev on 1 workers with
INFO:XGBoost-PySpark:Leveraging cuda:0 to train with QDM: off
2024-02-29 10:29:09,637 INFO XGBoost-PySpark: _fit Finished xgboost training!
2024-02-29 10:29:11,410 INFO XGBoost-PySpark: predict_udf CUDF or Cupy is unavailable, fallback the inference on the CPUs

users can disable the log by

import logging

logging.getLogger('XGBoost-PySpark').setLevel('WARN') 

[trivialfis]

I think this doesn't resolve the issue since the verbosity is still set by the get logger function?

[wbo4958]

I think this doesn't resolve the issue since the verbosity is still set by the get logger function?

It doesn't matter to just show a few lines of log by default, which can help debug.

[trivialfis]

You link the issue and claim to fix it?

[wbo4958]

You link the issue and claim to fix it?

Yeah, I replied to it in the issue.

[danmar3]

Hi @wbo4958 , thanks for looking into this. Reducing the number of lines printed is great, however I wanted to make an additional suggestion. How about in addition to the current solution, we also update the get_logger as follows:

# make level optional
def get_logger(name: str, level: Optional[str] = None) -> logging.Logger:
    """Gets a logger by name, or creates and configures it for the first time."""
    logger = logging.getLogger(name)
    # <HERE> only update the level if not None
    if level is not None:
        logger.setLevel(level)
    # If the logger is configured, skip the configure
    if not logger.handlers and not logging.getLogger().handlers:
        handler = logging.StreamHandler(sys.stderr)
        formatter = logging.Formatter(
            "%(asctime)s %(levelname)s %(name)s: %(funcName)s %(message)s"
        )
        handler.setFormatter(formatter)
        logger.addHandler(handler)
    return logger

I believe users in general do not expect a logger level to be reset. It would be great if users could set the logger configuration at the beginning of their application/notebook, so the user chooses which logs to see/hide. Users will still be able to see the logs if we would like so by calling logging.getLogger('XGBoost-PySpark').setLevel('INFO') at the beginning of the notebook.

Thanks!

[wbo4958]

@danmar3 Good suggestion, let me refine this PR for it.