You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ValueError: Expected parameter df (Tensor of shape (32, 168, 1)) of distribution Chi2() to satisfy the constraint GreaterThan(lower_bound=0.0), but found invalid values
#469
Closed
1 of 3 tasks
Yuang-Deng opened this issue
Aug 11, 2022
· 3 comments
· Fixed by #471
Your issue may already be reported!
Also, please search on the issue tracker before creating one.
I'm submitting a ...
bug report
feature request
support request => Please do not submit support request here, see note at the top of this template.
Issue Description
When Issue Happens
Steps To Reproduce
1.Auto-PyTorch from git@github.com:automl/Auto-PyTorch.git
1.config space from git@github.com:automl/ConfigSpace.git
1.run examples/APT-TS/APT_task.py with electricity
Expected Behavior
Current Behavior
Possible Solution
Your Code
importnumpyasnpfromautoPyTorch.api.time_series_forecastingimportTimeSeriesForecastingTaskimportpandasaspdsfromdatetimeimportdatetimeimportwarningsimportosimportcopyfrompathlibimportPathimportargparseimportcsvimportshutilfromautoPyTorch.datasets.resampling_strategyimport (
CrossValTypes,
HoldoutValTypes,
)
importdata_loaderfromconstantimportVALUE_COL_NAME, TIME_COL_NAME, SEASONALITY_MAP, FREQUENCY_MAP, DATASETSdefcompute_loss(forecast_horizon, seasonality, final_forecasts, test_series_list, train_series_list):
epsilon=0.1MASE= []
sMAPE= []
msMAPE= []
MAE= []
RMSE= []
sqrt_forecast_horizon=np.sqrt(forecast_horizon)
idx=0forf, y, y_datainzip(final_forecasts, test_series_list, train_series_list):
M=len(y_data)
diff_abs=np.abs(f-y)
ifM==seasonality:
mase_denominator=0else:
mase_denominator_coefficient=forecast_horizon/ (M-seasonality)
mase_denominator=mase_denominator_coefficient* \
np.sum(np.abs(y_data[seasonality:] -y_data[:-seasonality]))
abs_loss=np.sum(diff_abs)
mase=abs_loss/mase_denominatorifmase_denominator==0:
mase_denominator_coefficient=forecast_horizon/ (M-1)
mase_denominator=mase_denominator_coefficient* \
np.sum(np.abs(y_data[1:] -y_data[:-1]))
mase=abs_loss/mase_denominatorifnp.isnan(mase) ornp.isinf(mase):
# see the R filepasselse:
MASE.append(mase)
smape=2*diff_abs/ (np.abs(y) +np.abs(f))
smape[diff_abs==0] =0smape=np.sum(smape) /forecast_horizonsMAPE.append(smape)
msmape=np.sum(2*diff_abs/ (np.maximum(np.abs(y) +np.abs(f) +epsilon, epsilon+0.5))) /forecast_horizonmsMAPE.append(msmape)
mae=abs_loss/forecast_horizonMAE.append(mae)
rmse=np.linalg.norm(f-y) /sqrt_forecast_horizonRMSE.append(rmse)
idx+=1res= {}
res['Mean MASE'] =np.mean(MASE)
res['Median MASE'] =np.median(MASE)
res['Mean sMAPE'] =np.mean(sMAPE)
res['Median sMAPE'] =np.median(sMAPE)
res['Mean mSMAPE'] =np.mean(msMAPE)
res['Median mSMAPE'] =np.median(msMAPE)
res['Mean MAE'] =np.mean(MAE)
res['Median MAE'] =np.median(MAE)
res['Mean RMSE'] =np.mean(RMSE)
res['Median RMSE'] =np.median(RMSE)
returnresdefmain(working_dir="/home/$USER/tmp/tmp",
dataset_name='nn5_daily',
budget_type='dataset_size',
res_dir="/home/ubuntu/autopytorch/Auto-PyTorch/work_dirs/tsf_res",
validation='holdout',
seed=1):
file_name, external_forecast_horizon, integer_conversion=DATASETS[dataset_name]
dataset_path=Path("/home/ubuntu/autopytorch/tsf_data") /dataset_name/file_namedf, frequency, forecast_horizon, contain_missing_values, contain_equal_length= \
data_loader.convert_tsf_to_dataframe(str(dataset_path))
# If the forecast horizon is not given within the .tsf file, then it should be provided as a function inputifforecast_horizonisNone:
ifexternal_forecast_horizonisNone:
raiseException("Please provide the required prediction steps")
else:
forecast_horizon=external_forecast_horizontrain_series_list= []
test_series_list= []
X_train= []
y_train= []
X_test= []
y_test= []
iffrequencyisnotNone:
freq=FREQUENCY_MAP[frequency]
seasonality=SEASONALITY_MAP[frequency]
else:
freq="1Y"seasonality=1shortest_sequence=np.inftrain_start_time_list= []
forindex, rowindf.iterrows():
ifTIME_COL_NAMEindf.columns:
train_start_time=row[TIME_COL_NAME]
else:
train_start_time=datetime.strptime('1900-01-01 00-00-00',
'%Y-%m-%d %H-%M-%S') # Adding a dummy timestamp, if the timestamps are not available in the dataset or consider_time is Falsetrain_start_time_list.append(pds.Timestamp(train_start_time, freq=freq))
series_data=row[VALUE_COL_NAME].to_numpy()
# Creating training and test series. Test series will be only used during evaluationtrain_series_data=series_data[:len(series_data) -forecast_horizon]
test_series_data=series_data[(len(series_data) -forecast_horizon): len(series_data)]
y_test.append(series_data[-forecast_horizon:])
train_series_list.append(train_series_data)
test_series_list.append(test_series_data)
shortest_sequence=min(len(train_series_data), shortest_sequence)
ifvalidation=='cv':
n_splits=3whileshortest_sequence-forecast_horizon-forecast_horizon*n_splits<=0:
n_splits-=1ifn_splits>=2:
resampling_strategy=CrossValTypes.time_series_cross_validationresampling_strategy_args= {'num_splits': n_splits}
else:
warnings.warn('The dataset is not suitable for cross validation, we will try holdout instead')
validation='holdout'elifvalidation=='holdout_ts':
resampling_strategy=CrossValTypes.time_series_ts_cross_validationresampling_strategy_args=Noneifvalidation=='holdout':
resampling_strategy=HoldoutValTypes.time_series_hold_out_validationresampling_strategy_args=NoneX_train=copy.deepcopy(train_series_list)
y_train=copy.deepcopy(train_series_list)
X_test=copy.deepcopy(X_train)
path=Path(working_dir) /'APT_run'path_log=str(path/dataset_name/budget_type/f'{seed}'/"log")
path_pred=str(path/dataset_name/budget_type/f'{seed}'/"output")
# Remove intermediate filestry:
shutil.rmtree(path_log)
shutil.rmtree(path_pred)
exceptOSErrorase:
print("Error: %s - %s."% (e.filename, e.strerror))
smac_source_dir=Path(path_log) /"smac3-output"api=TimeSeriesForecastingTask(
#delete_tmp_folder_after_terminate=False,#delete_output_folder_after_terminate=False,seed=seed,
ensemble_size=20,
resampling_strategy=resampling_strategy,
resampling_strategy_args=resampling_strategy_args,
temporary_directory=path_log,
output_directory=path_pred,
)
api.set_pipeline_config(device="cuda",
torch_num_threads=8,
early_stopping=20)
ifbudget_type=="random_search":
budget_kwargs= {'budget_type': 'random_search',
'max_budget': None,
'min_budget': None}
elifbudget_type!='full_budget':
fromautoPyTorch.constantsimportFORECASTING_BUDGET_TYPEifbudget_typenotinFORECASTING_BUDGET_TYPEandbudget_type!='epochs':
raiseNotImplementedError('Unknown Budget Type!')
budget_kwargs= {'budget_type': budget_type,
'max_budget': 50ifbudget_type=='epochs'else1.0,
'min_budget': 5ifbudget_type=='epochs'else0.1}
else:
budget_kwargs= {'budget_type': 'epochs',
'max_budget': 50,
'min_budget': 50}
api.search(
X_train=None,
y_train=copy.deepcopy(y_train),
optimize_metric='mean_MASE_forecasting',
n_prediction_steps=forecast_horizon,
**budget_kwargs,
freq=freq,
# start_times_train=train_start_time_list,memory_limit=32*1024,
normalize_y=False,
total_walltime_limit=600,
min_num_test_instances=1000,
)
fromautoPyTorch.data.time_series_forecasting_validatorimportTimeSeriesForecastingInputValidatorres_dir=Path(res_dir)
res_dir_task=res_dir/dataset_name/budget_type/f'{seed}'smac_res_path=res_dir_task/'smac3-output'ifnotos.path.exists(str(res_dir_task)):
os.makedirs(str(res_dir_task))
try:
shutil.rmtree(smac_res_path)
exceptOSErrorase:
print("Error: %s - %s."% (e.filename, e.strerror))
try:
shutil.copytree(str(smac_source_dir), smac_res_path)
exceptOSErrorase: # python >2.5print("Error: %s - %s."% (e.filename, e.strerror))
refit_dataset=api.dataset.create_refit_set()
train_pred_seq= []
test_sets=api.dataset.generate_test_seqs()
# try:# api.refit(refit_dataset, 0)# # pred = api.predict(test_sets)# except Exception as e:# print(e)# exit()api.refit(refit_dataset, 0)
pred=api.predict(test_sets)
ifinteger_conversion:
final_forecasts=np.round(pred)
else:
final_forecasts=prediffrequencyisnotNone:
freq=FREQUENCY_MAP[frequency]
seasonality=SEASONALITY_MAP[frequency]
else:
freq="1Y"seasonality=1ifisinstance(seasonality, list):
seasonality=min(seasonality) # Use to calculate MASEseasonality=int(seasonality)
res=compute_loss(forecast_horizon, seasonality, pred, y_test, train_series_data)
print(res)
# write the forecasting results to a fileforecast_file_path=res_dir_task/f"{dataset_name}_{budget_type}_results.txt"withopen(forecast_file_path, "w") asoutput:
writer=csv.writer(output, lineterminator='\n')
writer.writerows(final_forecasts)
# Write training dataset and the actual results into separate files, which are then used for error calculations# We do not use the built-in evaluation method in GluonTS as some of the error measures we use are not implemented in thattemp_dataset_path=res_dir_task/f"{dataset_name}_dataset.txt"temp_results_path=res_dir_task/f"{dataset_name}_ground_truth.txt"# with open(str(temp_dataset_path), "w") as output_dataset:# writer = csv.writer(output_dataset, lineterminator='\n')# writer.writerows(train_series_list)withopen(str(temp_results_path), "w") asoutput_results:
writer=csv.writer(output_results, lineterminator='\n')
writer.writerows(test_series_list)
if__name__=='__main__':
parser=argparse.ArgumentParser(description='APT_Task')
parser.add_argument('--dataset_name', type=str, default="electricity_hourly", help='dataset name')
parser.add_argument("--budget_type", default="epochs", type=str, help='budget type')
parser.add_argument("--working_dir", default="/home/ubuntu/autopytorch/Auto-PyTorch/work_dirs", type=str,
help="directory where datasets and tmp files are stored")
parser.add_argument('--validation', type=str, default="holdout", help='type of validation')
parser.add_argument('--seed', type=int, default="10", help='random seed')
args=parser.parse_args()
dataset_name=args.dataset_namebudget_type=args.budget_typeworking_dir=args.working_dirvalidation=args.validationseed=args.seedmain(working_dir=working_dir, dataset_name=dataset_name, budget_type=budget_type, validation=validation, seed=seed)
Error message
Traceback (most recent call last):
File "/home/ubuntu/.pyenv/versions/3.8.9/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ubuntu/.pyenv/versions/3.8.9/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ubuntu/.vscode-server/extensions/ms-python.python-2022.12.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
cli.main()
File "/home/ubuntu/.vscode-server/extensions/ms-python.python-2022.12.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
run()
File "/home/ubuntu/.vscode-server/extensions/ms-python.python-2022.12.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
runpy.run_path(target, run_name="__main__")
File "/home/ubuntu/.vscode-server/extensions/ms-python.python-2022.12.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
return _run_module_code(code, init_globals, run_name,
File "/home/ubuntu/.vscode-server/extensions/ms-python.python-2022.12.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/ubuntu/.vscode-server/extensions/ms-python.python-2022.12.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "./examples/APT-TS/APT_task.py", line 346, in <module>
main(working_dir=working_dir, dataset_name=dataset_name, budget_type=budget_type, validation=validation, seed=seed)
File "./examples/APT-TS/APT_task.py", line 283, in main
api.refit(refit_dataset, 0)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/autoPyTorch/api/base_task.py", line 1441, in refit
fit_and_suppress_warnings(self._logger, model, X, y=None)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/autoPyTorch/evaluation/abstract_evaluator.py", line 338, in fit_and_suppress_warnings
pipeline.fit(X, y)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/autoPyTorch/pipeline/base_pipeline.py", line 155, in fit
self.fit_estimator(X, y, **fit_params)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/autoPyTorch/pipeline/base_pipeline.py", line 174, in fit_estimator
self._final_estimator.fit(X, y, **fit_params)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/autoPyTorch/pipeline/components/training/trainer/__init__.py", line 211, in fit
self._fit(
File "/home/ubuntu/autopytorch/Auto-PyTorch2/autoPyTorch/pipeline/components/training/trainer/__init__.py", line 310, in _fit
train_loss, train_metrics = self.choice.train_epoch(
File "/home/ubuntu/autopytorch/Auto-PyTorch2/autoPyTorch/pipeline/components/training/trainer/forecasting_trainer/forecasting_base_trainer.py", line 106, in train_epoch
loss, outputs = self.train_step(data, targets)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/autoPyTorch/pipeline/components/training/trainer/forecasting_trainer/forecasting_base_trainer.py", line 206, in train_step
outputs = self.model(past_targets=past_target,
File "/home/ubuntu/autopytorch/Auto-PyTorch2/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/autoPyTorch/pipeline/components/setup/network/forecasting_architecture.py", line 602, in forward
output = self.head(decoder_output)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/autoPyTorch/pipeline/components/setup/network_head/forecasting_network_head/distribution.py", line 95, in forward
return self.dist_cls(*self.domain_map(*params_unbounded))
File "/home/ubuntu/autopytorch/Auto-PyTorch2/.venv/lib/python3.8/site-packages/torch/distributions/studentT.py", line 50, in __init__
self._chi2 = Chi2(self.df)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/.venv/lib/python3.8/site-packages/torch/distributions/chi2.py", line 22, in __init__
super(Chi2, self).__init__(0.5 * df, 0.5, validate_args=validate_args)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/.venv/lib/python3.8/site-packages/torch/distributions/gamma.py", line 52, in __init__
super(Gamma, self).__init__(batch_shape, validate_args=validate_args)
File "/home/ubuntu/autopytorch/Auto-PyTorch2/.venv/lib/python3.8/site-packages/torch/distributions/distribution.py", line 55, in __init__
raise ValueError(
ValueError: Expected parameter df (Tensor of shape (32, 168, 1)) of distribution Chi2() to satisfy the constraint GreaterThan(lower_bound=0.0), but found invalid values:
tensor([[[nan],
[nan],
[nan],
...,
[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan],
...,
[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan],
...,
[nan],
[nan],
[nan]],
...,
[[nan],
[nan],
[nan],
...,
[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan],
...,
[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan],
...,
[nan],
[nan],
[nan]]], grad_fn=<MulBackward0>)
Make sure to add all the information needed to understand the bug so that someone can help.
If the info is missing, we'll add the 'Needs more information' label and close the issue until there is enough information.
The text was updated successfully, but these errors were encountered:
Hi,
this error happens because one hyperparameter configuration passed the HPO process (only trained on the training set) but failed during the refit process (when the model is trained on the training+validation set). Currently, our optimizer does not consider catching the exception within refit. We will fix this ASAP and incorporate it into the next release.
Before that, you could simply remove that line called refit() (though this might slightly weaken the final performance).
NOTE: ISSUES ARE NOT FOR CODE HELP - Ask for Help at https://stackoverflow.com
Your issue may already be reported!
Also, please search on the issue tracker before creating one.
Issue Description
1.Auto-PyTorch from git@github.com:automl/Auto-PyTorch.git
1.config space from git@github.com:automl/ConfigSpace.git
1.run examples/APT-TS/APT_task.py with electricity
Expected Behavior
Current Behavior
Possible Solution
Your Code
Error message
Your Local environment
pip freeze
orconda list
Make sure to add all the information needed to understand the bug so that someone can help.
If the info is missing, we'll add the 'Needs more information' label and close the issue until there is enough information.
The text was updated successfully, but these errors were encountered: