Only a fraction of models show up in modeltime_resamples...

[Steviey]

Ubuntu 16.x LTS, R latest, modeltime.ensemble latest

A submodels_tbl has 15 correctly fitted models.
When I try to use them with modeltime_fit_resamples(), only a fraction of them show up in the result of that function (only 4)
Is there an explanation available?

resamples_tscv <- df_train %>%
	time_series_cv(
		assess   = test_len
		,initial = train_len
		#skip    = "2 years",
		,slice_limit = dplyr::n()
	) 

submodel_predictions <- submodels_tbl %>%
	modeltime_fit_resamples(
	resamples = resamples_tscv,
	control   = control_resamples(verbose = TRUE)
)

debugAnalyse<-1
if(debugAnalyse>0){
	# Visualize the Resample Sets
	myPlot<-resamples_tscv %>%
		tk_time_series_cv_plan() %>%
		plot_time_series_cv_plan(
			date, value,
			.facet_ncol  = 2,
			.interactive = TRUE
		)

	print(myPlot)    

	myPlot<-submodel_predictions %>%
		plot_modeltime_resamples(
			.interactive = TRUE
		)

	print(myPlot)

	#View(submodel_predictions)
	
	predictions_tbl <- modeltime.resample::unnest_modeltime_resamples(submodel_predictions)
	
	View(predictions_tbl)

	predictions_tbl$editDate        <- format(Sys.time(), "%Y-%m-%d %H:%M:%S")
	predictions_tbl$pslMetaLearner  <- metaLearner
	fastInOut('predictions_tbl.Rds',predictions_tbl)

	predictions_by_rowid_tbl <- predictions_tbl %>%
		dplyr::select(.row_id, .model_id, .pred) %>%
		dplyr::mutate(.model_id = stringr::str_c(".model_id_", .model_id)) %>%
		tidyr::pivot_wider(names_from  = .model_id,values_from = .pred)

	View(predictions_by_rowid_tbl)                        
				
}    

Side note: When I use less then the 15 models, the code breaks while fitting a glmnet-metaLearner. It promps:

 x Slice1: preprocessor 1/1, model 1/1: Error: For the glmnet engine, `penalty` 
 must be a single number (or a value of `tune()`).

... where the model ist correctly tagged with 'penalty=tune::tune()'
I noticed the same effect with lasso (mixture=1).

My guess is, it will be forgotten anywhere in modeltime.ensemble-internal code. Currently I'm testing different metaLearners. Xgboost metaLearner seem to work only without xgboost submodels. Others work fine so far.

It would be nice to have a fallback/try-catch option in modeltime.resample. Otherwise code breaks in huge projects, any time something fails at this point.

[Steviey]

Could it be related to different lengths of .resample_results per model/workflow?

I noticed doing this:

predictions_tbl <- modeltime.resample::unnest_modeltime_resamples(m750_training_resamples_fitted)
View(predictions_tbl)

... results in a clickable, ready to drill down view in Rstudio.

... comparing with this, the result seems to be corrupted and is not clickable. The difference lies in the different lengths of .resample_results.

What would be, if we were able to reduce the predictions on min. length of resample.results?

update: There is more... .predictions=NULL in a GLMNET-model

Every failing model will be lost...
Since we have no influence on how models will be treated in modeltime_fit_resamples(), there is no chance to fix it- other then hardcoded.

Error in if (is.numeric(args$mixture) && (args$mixture < 0 | args$mixture > : missing value where TRUE/FALSE needed