Improve preprocessing logging

[brandenchan]

Reduce duplicated message when preprocessing

[tholor]

I really appreciate the goal, but I am concerned about the approach as it would delay the logging messages to the end of processing. That could be confusing during debugging where you might want to understand what parts of the code got executed etc.
We probably want to have a similar design in other parts of FARM / Haystack - so it's probably worth making it neat.

My proposal would be to switch self.log to a Set, when we want to log a new msg we check if it's already in self.log. If yes: skip logging. If no: do logging and add msg to self.log
With that, we have the same effect and no time delay. What do you think?

[brandenchan]

Yup I'm all for this idea!

I really appreciate the goal, but I am concerned about the approach as it would delay the logging messages to the end of processing. That could be confusing during debugging where you might want to understand what parts of the code got executed etc.
We probably want to have a similar design in other parts of FARM / Haystack - so it's probably worth making it neat.

My proposal would be to switch self.log to a Set, when we want to log a new msg we check if it's already in self.log. If yes: skip logging. If no: do logging and add msg to self.log
With that, we have the same effect and no time delay. What do you think?

Yup I'm all for this idea!

[tholor]

LGTM