-
Notifications
You must be signed in to change notification settings - Fork 118
chunksize is ignored in TICA #846
Comments
With .get_output() it's the same problem. The ckunksize parameter of the transformer and it's source are not respected. |
the self.chunk_size property is delegated through the pipeline, so it should be taken from there. Your approach is the correct fix, but do not take the property from the iterable but self instead. I can not see why the chunksize in Iterable.get_output should not be respected, since it is passed correctly: |
And you are right, the handling in param_stage is wrong, since the set chunksize of the reader/datasource is overridden with 0. |
Only set it in case the user has overriden it, otherwise take default chunksize from Iterable. Fixes markovmodel#846
@fabian-paul can you please confirm that the behaviour correct with the suggested PR? |
@marscher yes, this solved the problems. |
Thank you for reporting and testing!
|
@clonker with these changes the test_fragmented_reader_xtc fails with wrong coordinates. Do you have any clue what could be broken now? |
I'd guess that the chunked mode is somehow broken, since the tested behaviour used always chunksize=0 |
I've found the culprit: in patches.iterload there was no proper setter for chunksize... cc @clonker |
Apparently setting the chunk size parameter doesn't have any effect in TICA anymore.
I have a long (microsecond) trajectory that I want to use to evaluate a larger set of features. Setting the chunk_size doesn't have any effect on the feature reader as can be seen from the output below.
I added the following print statement here:
This is the output I get.
I'm not sure how this should be fixed. As a quick fix, I changed https://github.com/markovmodel/PyEMMA/blob/devel/pyemma/coordinates/transform/tica.py#L264
to
However this works only, if I instantiate the TICA object directly. When I use the tica() API function, the chunksize is set back to zero.
IMHO this looks very broken to me. There are some lines such as https://github.com/markovmodel/PyEMMA/blob/devel/pyemma/coordinates/api.py#L837 that don't make any sense to me. _param_stage only gets called with its chunk_size parameter left at its default value = 0, so this can only work if the chunk size parameters of the source is also zero.
What's the recommanded way to pass the chunk_size parameter between different transformers? If an iterator is created in _estimate() where should the chunk_size parameter be taken from?
The text was updated successfully, but these errors were encountered: