Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConsoleRenderer and TimeStamper can't be serialized #126

Closed
bonheml opened this issue Jun 2, 2017 · 12 comments

Comments

@bonheml
Copy link

@bonheml bonheml commented Jun 2, 2017

These two object should be able to be pickled.
For now, they can't be used with multiprocessing which is common use case.

Here is a script to recreate the bug:

import structlog
import pickle

if __name__=="__main__":
    try:
        pickle.dumps(structlog.dev.ConsoleRenderer(colors=False))
    except Exception as e:
        print(e)
    try:
        pickle.dumps(structlog.processors.TimeStamper(fmt="%Y-%m-%d %H:%M:%S"))
    except Exception as e:
        print(e)

It leads to the following stackTrace

/usr/bin/python3.5 /home/lisa/Dev/log_sanbox/main_logger_structlog.py
Can't pickle local object 'ConsoleRenderer.__init__.<locals>._repr'
Can't pickle <class 'structlog.processors.TimeStamper'>: it's not the same object as structlog.processors.TimeStamper
@bonheml bonheml changed the title ConsoleRenderer and TimeStamper can't be pickled ConsoleRenderer and TimeStamper can't be serialized Jun 2, 2017
@hynek

This comment has been minimized.

Copy link
Owner

@hynek hynek commented Jun 4, 2017

Hmmm, the problem with those two processors is that that they write JIT optimal versions of themselves for each possible configuration.

Do you happen to know more about pickling whether we could work around that somehow?

@bonheml

This comment has been minimized.

Copy link
Author

@bonheml bonheml commented Jun 5, 2017

Thanks for your answer. I'm not sure to understand correctly your question about pickling but I'll try to detail more my use case.

I have a program which use python multiprocessing and during spawn of processes, my logger can't be pickled leading to an error.

I work with python 3.5 on Fedora

@MarcBoissonneault

This comment has been minimized.

Copy link

@MarcBoissonneault MarcBoissonneault commented Sep 6, 2018

Any new on this issue?
The ConsoleRenderer still seems to be not serializable:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.7/copy.py:180: in deepcopy
    y = _reconstruct(x, memo, *rv)
/usr/lib/python3.7/copy.py:280: in _reconstruct
    state = deepcopy(state, memo)
/usr/lib/python3.7/copy.py:150: in deepcopy
    y = copier(x, memo)
/usr/lib/python3.7/copy.py:240: in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
/usr/lib/python3.7/copy.py:161: in deepcopy
    y = copier(memo)
/usr/local/lib/python3.7/dist-packages/structlog/_base.py:176: in _proxy_to_logger
    args, kw = self._process_event(method_name, event, event_kw)
/usr/local/lib/python3.7/dist-packages/structlog/_base.py:136: in _process_event
    event_dict = proc(self._logger, method_name, event_dict)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <structlog.dev.ConsoleRenderer object at 0x7f99a35ae208>
_ = <PrintLogger(file=<_pytest.capture.EncodedFile object at 0x7f9a31086748>)>
__ = '__deepcopy__', event_dict = {}

    def __call__(self, _, __, event_dict):
        sio = StringIO()
    
        ts = event_dict.pop("timestamp", None)
        if ts is not None:
            sio.write(
                # can be a number if timestamp is UNIXy
                self._styles.timestamp + str(ts) + self._styles.reset + " "
            )
        level = event_dict.pop("level", None)
        if level is not None:
            sio.write(
                "[" + self._level_to_color[level] +
                _pad(level, self._longest_level) +
                self._styles.reset + "] "
            )
    
        event = event_dict.pop("event")
        if event_dict:
            event = _pad(event, self._pad_event) + self._styles.reset + " "
        else:
>           event += self._styles.reset
E           TypeError: unsupported operand type(s) for +=: 'dict' and 'str'

/usr/local/lib/python3.7/dist-packages/structlog/dev.py:191: TypeError
@creslinux

This comment has been minimized.

Copy link

@creslinux creslinux commented May 3, 2019

Ive ran into this. Was there any resolution ?
I seem tied to a single core with structlog in my application, this isn't workable as i need to bring 10's of cores online to crunch a dataframe many ways.

@berlinguyinca

This comment has been minimized.

Copy link

@berlinguyinca berlinguyinca commented May 3, 2019

could this be fixed at this stage? It's been a couple of years now

@hynek

This comment has been minimized.

Copy link
Owner

@hynek hynek commented May 8, 2019

I would love to fix it but it's not as simple without a notable performance regression so it's not on the top of my todo list. :(

@berlinguyinca

This comment has been minimized.

Copy link

@berlinguyinca berlinguyinca commented May 8, 2019

@hynek

This comment has been minimized.

Copy link
Owner

@hynek hynek commented May 8, 2019

If that's all you want, you can write your own. If you implement only what you need, it becomes three lines of code or so:

def stamper(self, _, __, event_dict):
event_dict[key] = now_method().strftime(fmt)
return event_dict

If someone want to help out and fix ConsoleRenderer: I do not care about its performance so it can be implemented however.

hynek added a commit that referenced this issue Sep 21, 2019
hynek added a commit that referenced this issue Sep 21, 2019
hynek added a commit that referenced this issue Sep 21, 2019
@hynek

This comment has been minimized.

Copy link
Owner

@hynek hynek commented Sep 21, 2019

I've just made TimeStamper, ConsoleRenderer, the config proxy you get from get_logger, PrintLogger and the generic BoundLogger pickleable.

Is anything missing or can I release soon? It would be great if you could check your use cases against master. The default config should definitely work.

@berlinguyinca

This comment has been minimized.

Copy link

@berlinguyinca berlinguyinca commented Sep 21, 2019

@berlinguyinca

This comment has been minimized.

Copy link

@berlinguyinca berlinguyinca commented Sep 23, 2019

@hynek

This comment has been minimized.

Copy link
Owner

@hynek hynek commented Sep 24, 2019

OK I’m closing this then. Please report if y’all have problems with any other class.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.