ValueError: assignment destination is read-only

[mlittmanabbvie]

Look into the FAQ of the readme. Can the bug be resolved by one of those solutions?
No
Describe the bug

When using the joblib as the parallel distributor, if the number of processes / size of them gets too big then an error will be thrown
ValueError: assignment destination is read-only
()

This issue is described here

Code to reproduce the behavior
scikit-learn/scikit-learn#5956

Error message from command line
ValueError: assignment destination is read-only

System information:

• OS Platform and Distribution: Windows 10
• Python version: 3.8
• Hyperactive version 3.3.3

Additional context
The issue is not actually with Hyperactive however, the fix for the ValueError issue is to add max_nbytes='50M' to the Parallel instantiation. The issue is that when instantiating Hyperactive, there is no way to pass this argument through to joblib without changing the underlying Hyperactive package.

[SimonBlanke]

Hello @mlittmanabbvie,

thank you for opening this issue and providing detailed information about the problem! :-)

Could you additionally show the code that generated the error with Hyperactive?
Hyperactive might already be equipped to solve this problem, but I would like to try it out beforehand.

[mlittmanabbvie]

Hey Simon,

I tried to come up with code that could replicate the issue (that isnt my code) and I am struggling for some reason. Perhaps the reason for the issue is not what I think it is? Here is the stack trace, maybe you might be able to figure out why?

---

_RemoteTraceback Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/cdsw/.local/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 436, in _process_worker
r = call_item()
File "/home/cdsw/.local/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 288, in call
return self.fn(*self.args, **self.kwargs)
File "/home/cdsw/.local/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 595, in call
return self.func(*args, **kwargs)
File "/home/cdsw/.local/lib/python3.8/site-packages/joblib/parallel.py", line 262, in call
return [func(*args, **kwargs)
File "/home/cdsw/.local/lib/python3.8/site-packages/joblib/parallel.py", line 262, in
return [func(*args, **kwargs)
File "/home/cdsw/.local/lib/python3.8/site-packages/hyperactive/process.py", line 25, in process
optimizer.search(
File "/home/cdsw/.local/lib/python3.8/site-packages/hyperactive/optimizers/gfo_wrapper.py", line 123, in search
self._optimizer.search(
File "/home/cdsw/.local/lib/python3.8/site-packages/gradient_free_optimizers/search.py", line 207, in search
self._initialization(init_pos, nth_iter)
File "/home/cdsw/.local/lib/python3.8/site-packages/gradient_free_optimizers/times_tracker.py", line 27, in wrapper
res = func(self, *args, **kwargs)
File "/home/cdsw/.local/lib/python3.8/site-packages/gradient_free_optimizers/search.py", line 107, in _initialization
score_new = self._score(init_pos)
File "/home/cdsw/.local/lib/python3.8/site-packages/gradient_free_optimizers/times_tracker.py", line 18, in wrapper
res = func(self, *args, **kwargs)
File "/home/cdsw/.local/lib/python3.8/site-packages/gradient_free_optimizers/search.py", line 98, in _score
return self.score(pos)
File "/home/cdsw/.local/lib/python3.8/site-packages/gradient_free_optimizers/results_manager.py", line 31, in _wrapper
results_dict = self._obj_func_results(objective_function, para)
File "/home/cdsw/.local/lib/python3.8/site-packages/gradient_free_optimizers/results_manager.py", line 14, in _obj_func_results
results = objective_function(para)
File "/home/cdsw/.local/lib/python3.8/site-packages/hyperactive/optimizers/objective_function.py", line 47, in _model
results = self.objective_function(self)
File "/home/cdsw/ClusteringAttempthyperlagrangianscoreinonedf.py", line 503, in func_mina
total,p = core_func_min(df,exponent,slope,freq_multiplier,c,p,ascending,discount_type,use_pca,last,patid,events_repeat_often,score_col,fin_score_column)
File "/home/cdsw/ClusteringAttempthyperlagrangianscoreinonedf.py", line 186, in core_func_min
fin2,davies_bouldin,silhouette_avg,calinski_harabasz = score_and_cluster(df=df,exponent=exponent,clust_num=c,freq_multiplier=freq_multiplier,slope=slope,ascending=ascending,discount_type=discount_type,use_pca = use_pca,plot = False, p = p,events_repeat_often = events_repeat_often,score_col = score_col)
File "/home/cdsw/ClusteringAttempthyperlagrangianscoreinonedf.py", line 359, in score_and_cluster
cluster_prepped = prep_for_cluster(df,exponent,clust_num,freq_multiplier,slope,ascending,discount_type,patid = patid,event_column=event_column, event_or_eventtrans = event_or_eventtrans, times_reweight = times_reweight, score_col = score_col)
File "/home/cdsw/ClusteringAttempthyperlagrangianscoreinonedf.py", line 340, in prep_for_cluster
df.loc[:,'score'+ str(times_reweight)] = df1.loc[:,'score'+ str(times_reweight)]
File "/home/cdsw/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 692, in setitem
iloc._setitem_with_indexer(indexer, value, self.name)
File "/home/cdsw/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1635, in _setitem_with_indexer
self._setitem_with_indexer_split_path(indexer, value, name)
File "/home/cdsw/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1676, in _setitem_with_indexer_split_path
self._setitem_single_column(ilocs[0], value, pi)
File "/home/cdsw/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1817, in _setitem_single_column
self.obj._iset_item(loc, ser)
File "/home/cdsw/.local/lib/python3.8/site-packages/pandas/core/frame.py", line 3223, in _iset_item
NDFrame._iset_item(self, loc, value)
File "/home/cdsw/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 3821, in _iset_item
self._mgr.iset(loc, value)
File "/home/cdsw/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1110, in iset
blk.set_inplace(blk_locs, value_getitem(val_locs))
File "/home/cdsw/.local/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 363, in set_inplace
self.values[locs] = values
ValueError: assignment destination is read-only
"""

The above exception was the direct cause of the following exception:

ValueError Traceback (most recent call last)
in
254 score_col = None)
255
--> 256 results, resultscopy = main(**kwargs)
257
258 #results = main(**kwargs)

~/ClusteringAttempthyperlagrangianscoreinonedf.py in main(**kwargs)
728 start = datetime.now()
729 print(f'START CLOCK: {start}\n________________________________\n')
--> 730 df,transition_limit = dockkes_clustering(**kwargs)
731 print(type(df_copy))
732 df_copy = df.copy(deep = True)

~/ClusteringAttempthyperlagrangianscoreinonedf.py in dockkes_clustering(df, )
709 else:
710 tried_everything = False
--> 711 prev = perform_clustering(df,

~/ClusteringAttempthyperlagrangianscoreinonedf.py in perform_clustering(df, )
915 return
916
--> 917 finp = multiprocess_clusters(df,ascending,,

~/ClusteringAttempthyperlagrangianscoreinonedf.py in multiprocess_clusters(df, )
604 print(f'\nBEST NUMBER OF CLUSTERS = {finp.max_cluster} Clusters:\n----------------------------\nOptimizing now...\n')
605
--> 606 return optimize(df)
607
608

~/ClusteringAttempthyperlagrangianscoreinonedf.py in optimize(df, )
527
--> 528 h.run()
529
530 print('THE BEST PARAMETERS:\n',{k:v for k,v in h.best_para(func_mina).items() if k!= 'df'})

~/.local/lib/python3.8/site-packages/hyperactive/hyperactive.py in run(self, max_time, _test_st_backend)
178 progress_board.open_dashboard()
179
--> 180 self.results_list = run_search(
181 self.process_infos, self.distribution, self.n_processes
182 )

~/.local/lib/python3.8/site-packages/hyperactive/run_search.py in run_search(search_processes_infos, distribution, n_processes)
49 (distribution, process_func), dist_paras = _get_distribution(distribution)
50
---> 51 results_list = distribution(
52 process_func, process_infos, n_processes, **dist_paras
53 )

~/.local/lib/python3.8/site-packages/hyperactive/distribution.py in joblib_wrapper(process_func, search_processes_paras, n_processes, **kwargs)
32 def joblib_wrapper(process_func, search_processes_paras, n_processes, **kwargs):
33 jobs = [delayed(process_func)(**info_dict) for info_dict in search_processes_paras]
---> 34 results = Parallel(n_jobs=n_processes, **kwargs)(jobs)
35
36 return results

~/.local/lib/python3.8/site-packages/joblib/parallel.py in call(self, iterable)
1054
1055 with self._backend.retrieval_context():
-> 1056 self.retrieve()
1057 # Make sure that we get a last message telling us we are done
1058 elapsed_time = time.time() - self._start_time

~/.local/lib/python3.8/site-packages/joblib/parallel.py in retrieve(self)
933 try:
934 if getattr(self._backend, 'supports_timeout', False):
--> 935 self._output.extend(job.get(timeout=self.timeout))
936 else:
937 self._output.extend(job.get())

~/.local/lib/python3.8/site-packages/joblib/_parallel_backends.py in wrap_future_result(future, timeout)
540 AsyncResults.get from multiprocessing."""
541 try:
--> 542 return future.result(timeout=timeout)
543 except CfTimeoutError as e:
544 raise TimeoutError from e

/usr/local/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
442 raise CancelledError()
443 elif self._state == FINISHED:
--> 444 return self.__get_result()
445 else:
446 raise TimeoutError()

/usr/local/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
387 if self._exception:
388 try:
--> 389 raise self._exception
390 finally:
391 # Break a reference cycle with the exception in self._exception

ValueError: assignment destination is read-only

[mlittmanabbvie]

Doesn't seem to be an issue with Hyperactive. Apologies!