pytest with capsys causes random test failures

[xmatthias]

Problem: running tests with output capturing causes random failures
catboost version:1.1
Operating System: linux

---

running tests with output capturing causes random test failures due to sys.stdout (and sys.stderr) hard-binding in catboost

Reproduction

A simplified reproduction can be found below. This is obviously simplified, and could be fixed by changing the imports - but it's to show the problem in a more complex space.

File: process.py

from catboost import CatBoostRegressor


def train_and_predict():
    # Initialize data

    train_data = [[1, 4, 5, 6],
                [4, 5, 6, 7],
                [30, 40, 50, 60]]

    eval_data = [[2, 4, 6, 8],
                [1, 4, 50, 60]]

    train_labels = [10, 20, 30]
    # Initialize CatBoostRegressor
    model = CatBoostRegressor(iterations=2,
                            learning_rate=1,
                            depth=2)
    # Fit model
    model.fit(train_data, train_labels)
    # Get predictions
    preds = model.predict(eval_data)
    return preds

test_model.py

def test_1(capsys):

    import process

    assert True == True


def test_2(capsys):

    import process
    preds = process.train_and_predict()
    assert preds is not None

Running pytest test_model.py::test_2 test_model.py::test_1 works, while pytest test_model.py::test_1 test_model.py::test_2 does not work, and will fail with

$ pytest 
================================================================================== test session starts ==================================================================================
platform linux -- Python 3.9.7, pytest-7.1.3, pluggy-1.0.0
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
rootdir: /home/xmatt/devel/cryptos/catboost_test
plugins: asyncio-0.19.0, cov-4.0.0, mock-3.10.0, typeguard-2.13.3
asyncio: mode=strict
collected 2 items                                                                                                                                                                       

test_model.py .F                                                                                                                                                                  [100%]

======================================================================================= FAILURES ========================================================================================
________________________________________________________________________________________ test_2 _________________________________________________________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f42cf5f6670>

    def test_2(capsys):
    
        import process
>       process.train_and_predict()

test_model.py:11: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
process.py:20: in train_and_predict
    model.fit(train_data, train_labels)
env/lib/python3.9/site-packages/catboost/core.py:5731: in fit
    return self._fit(X, y, cat_features, text_features, embedding_features, None, sample_weight, None, None, None, None, baseline,
env/lib/python3.9/site-packages/catboost/core.py:2368: in _fit
    elif (len(self.get_embedding_feature_indices()) > 0):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <catboost.core.CatBoostRegressor object at 0x7f42cf5f6f10>

    def get_embedding_feature_indices(self):
        if not self.is_fitted():
>           raise CatBoostError("Model is not fitted")
E           _catboost.CatBoostError: Model is not fitted

env/lib/python3.9/site-packages/catboost/core.py:2819: CatBoostError
----------------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------------
ERROR    catboost.core:core.py:128 <method '_train' of '_catboost._CatBoost' objects> returned a result with an error set
Traceback (most recent call last):
  File "_catboost.pyx", line 5949, in _catboost._WriteLog
  File "_catboost.pyx", line 5949, in _catboost._WriteLog
ValueError: I/O operation on closed file.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "env/lib/python3.9/site-packages/catboost/core.py", line 126, in log_fixup
    yield
  File "env/lib/python3.9/site-packages/catboost/core.py", line 2356, in _fit
    self._train(
  File "env/lib/python3.9/site-packages/catboost/core.py", line 1760, in _train
    self._object._train(train_pool, test_pool, params, allow_clear_pool, init_model._object if init_model else None)
SystemError: <method '_train' of '_catboost._CatBoost' objects> returned a result with an error set
================================================================================ short test summary info ================================================================================
FAILED test_model.py::test_2 - _catboost.CatBoostError: Model is not fitted

Now the real failure here is ValueError: I/O operation on closed file..

This is caused by the way catboost is initializing default parameters (for example in fit()).

catboost/catboost/python-package/catboost/core.py

Lines 5624 to 5629 in 479b2ba

	def fit(self, X, y=None, cat_features=None, text_features=None, embedding_features=None,
	sample_weight=None, baseline=None, use_best_model=None,
	eval_set=None, verbose=None, logging_level=None, plot=False, plot_file=None, column_description=None,
	verbose_eval=None, metric_period=None, silent=None, early_stopping_rounds=None,
	save_snapshot=None, snapshot_file=None, snapshot_interval=None, init_model=None, callbacks=None,
	log_cout=sys.stdout, log_cerr=sys.stderr):

By using sys.stdout as default - it's binding this value at initialization. If something happens to that stream and a new instance should be used (for example through pytest catsys fixtures) - then catboost won't realize and will continue using the now closed stream - causing the above failures.

Running them in reverse works as test_1 is not actually calling any method in catboost - but is only loading the dependency.

A proper way in my opinion is to leave the default as None - and check if the content is None - and then get an instance of sys.stdout (and sys.stderr).