'AutoMLState' has no attribute 'sample_weight_all' when split_type='time'

[ChristopherRussell]

I'm struggling to find a minimal example for this to share, but I think I can describe the issue. I think I could find a minimal example if I better knew how to get into this elif clause in the _search method at the end of fitting.

What I think is happening:
The only place I can see the sample_weight_all attribute being set is in AutoML._prepare_data and that only happens when split_type is in SHUFFLE_SPLIT_TYPES = ["uniform", "stratified"]. Then if AutoML._state.retrain_final is true then we call AutoMLState._train_with_config which leads to AutoMLState._prepare_sample_train_data which leads to the exception here because sample_weight_all is not set.

[sonichi]

Thanks @ChristopherRussell . There could be a bug based on what you described. If you know how to fix it, feel free to make a PR. This test

FLAML/test/automl/test_split.py

Line 51 in cdc4201

def test_time():

passes because no "sample_weight" is passed from fit_kwargs. Your example must have "sample_weight" in Automl.fit(), right?

[ChristopherRussell]

I do pass in sample_weight in Automl.fit().

I'd guess that the sample_weight_all attribute needs to be set somewhere, in a similar way to here:

        SHUFFLE_SPLIT_TYPES = ["uniform", "stratified"]
        if self._split_type in SHUFFLE_SPLIT_TYPES:
            if self._sample_weight_full is not None:
                X_train_all, y_train_all, self._state.sample_weight_all = shuffle(
                    X_train_all,
                    y_train_all,
                    self._sample_weight_full,
                    random_state=RANDOM_SEED,
                )

but I'm not confident enough in my understanding of the context to do it correctly.

[sonichi]

Add an "else" clause after this block:

FLAML/flaml/automl/automl.py

Lines 1313 to 1335 in cdc4201

	if self._split_type in SHUFFLE_SPLIT_TYPES:
	if self._sample_weight_full is not None:
	X_train_all, y_train_all, self._state.sample_weight_all = shuffle(
	X_train_all,
	y_train_all,
	self._sample_weight_full,
	random_state=RANDOM_SEED,
	)
	self._state.fit_kwargs[
	"sample_weight"
	] = (
	self._state.sample_weight_all
	) # NOTE: _prepare_data is before kwargs is updated to fit_kwargs_by_estimator
	if isinstance(self._state.sample_weight_all, pd.Series):
	self._state.sample_weight_all.reset_index(drop=True, inplace=True)
	else:
	X_train_all, y_train_all = shuffle(
	X_train_all, y_train_all, random_state=RANDOM_SEED
	)
	if self._df:
	X_train_all.reset_index(drop=True, inplace=True)
	if isinstance(y_train_all, pd.Series):
	y_train_all.reset_index(drop=True, inplace=True)

else:
    self._state.sample_weight_all = self._sample_weight_full

[immu4989]

Picking this up — the fix described by @sonichi above still applies. The affected branch has moved a bit since 2023; the structural problem now lives in flaml/automl/task/generic_task.py around line 883 (the SHUFFLE_SPLIT_TYPES branching inside prepare_data), and the failing access is in flaml/automl/state.py:255 where self.sample_weight_all is read whenever sample_weight is in fit_kwargs, regardless of split type.

Plan:

• Initialize state.sample_weight_all = sample_weight_full once before the shuffle branching, so it's always set when the user passes sample_weight. The SHUFFLE_SPLIT_TYPES branch keeps its existing behavior (re-assigns the shuffled weight after shuffle(...)).
• Add a regression test in test/automl/test_split.py covering split_type="time" + sample_weight + retrain_full=True (the path that triggers the original AttributeError).

Will open a PR shortly.

[immu4989]

While drafting the fix and a regression test, I found that the original AttributeError is hard to reach through the public API after the refactors that moved _prepare_data into flaml/automl/task/generic_task.py:

• With eval_method="cv", state.data_size[0] == full_size, so the buggy else branch in AutoMLState.prepare_sample_train_data (flaml/automl/state.py:249-255) is never entered.
• With eval_method="holdout" + split_type="time" (or "group") + sample_weight, the code now raises a different ValueError earlier in prepare_data — before ever reaching the sample_weight_all access. That's a second bug I've now filed separately as a new issue, with a clean repro.

The defensive state.sample_weight_all = sample_weight_full init you proposed in 2023 is still correct — and once the new bug is fixed, the same test case will exercise the original AttributeError path. I'll bundle both fixes into one PR (closing this issue and the new one) since they share a regression test.