You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe what you want to do
During debugging, I was changing the list of files I wanted to process, as well as associated metadata. I then noticed that the list of files updated fine, while metadata did not. Is this pattern intended?
The metadata arriving in the processor does not update. @nsmith- found a solution to this, metadata_cache={} can be used in processor.Runner.
I came across this in a notebook, where I kept changing fileset and re-running the steps below the fileset definition to process data.
Explain what documentation is missing
Assuming that this is intended behavior, it would be useful to describe this somewhere prominently, perhaps in the processor section. It is unintuitive to me that the list of files changes (not demonstrated above, but to do so, can duplicate the file in the second fileset), but the metadata coming from the same dict does not.
The text was updated successfully, but these errors were encountered:
The original purpose of the metadata cache was to not have to re-scan the file list to determine the number of events and cluster boundaries of the files, under the assumption that input files are immutable. It should not have been saving also the user-supplied metadata. So I'm inclined to call this a bug. It looks like we could change
Describe what you want to do
During debugging, I was changing the list of files I wanted to process, as well as associated metadata. I then noticed that the list of files updated fine, while metadata did not. Is this pattern intended?
This script is doing the exact same thing twice, with only the
metadata
in thefileset
changing. Output:The metadata arriving in the processor does not update. @nsmith- found a solution to this,
metadata_cache={}
can be used inprocessor.Runner
.I came across this in a notebook, where I kept changing
fileset
and re-running the steps below thefileset
definition to process data.Explain what documentation is missing
Assuming that this is intended behavior, it would be useful to describe this somewhere prominently, perhaps in the processor section. It is unintuitive to me that the list of files changes (not demonstrated above, but to do so, can duplicate the file in the second
fileset
), but themetadata
coming from the same dict does not.The text was updated successfully, but these errors were encountered: