Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catalog.creation_date and Catalog.revision_date from babel.message.Catalog are unpickleable when running in multiprocessing #12060

Open
hoangduytran opened this issue Mar 11, 2024 · 1 comment
Labels

Comments

@hoangduytran
Copy link

hoangduytran commented Mar 11, 2024

Describe the bug

I was attempt to make use multiple PO files as a group of dictionaries, each with different functions (one for terminologies, one for uppercase allowed sentences, one to check for starting sentence allowed terms, etc..) and loading time for each file was taking too long, so in order to speed them up, I sought to make use of multiprocessing, so each process is loading one file. During the process of attempting to do this, I found the pickle error for Catalog and the mentioned fields are causing the Catalog class to be unpickleable.

How to Reproduce

Run this python script:

import multiprocessing
from sphinx_intl import catalog as c
def loadOneDict(path: str):
    cat = c.load_po(path)
    return (path, cat)

def mapCallBack(result):
   print(f'mapCallBack: {result}')

def errorCallBack(result):
   print(f'errorCallBack: {result}')

pool = multiprocessing.Pool()
dict_path = <path to a po file>
pool.apply_async(loadOneDict, args=[dict_path, ], callback=mapCallBack, error_callback=errorCallBack)
pool.close()
pool.join()

and use this PO file for example:

https://github.com/hoangduytran/blender_ui/blob/main/4.0/vi.po

Environment Information

Install the followings:
sphinx==7.2.6

# Sphinx dependencies that are important
Jinja2==3.1.2
Pygments==2.16.1
docutils==0.18.1
snowballstemmer==2.2.0
babel==2.13.0
requests==2.31.0

# Only needed for building translations.
sphinx-intl==2.1.0

# Only needed to match the theme used for the official documentation.
# Without this theme, the default theme will be used.
sphinx_rtd_theme==2.0.0rc4

# Only for convenience, allows live updating while editing RST files.
# Access by running:
#   make livehtml
sphinx-autobuild==2021.3.14

# Required for spell-checking
pyenchant

Sphinx extensions

No response

Additional context

In order to remove the error, temporary do this in Catalog class of the babel.message

class Catalog:
        def __getstate__(self):
        if isinstance(self.creation_date, (datetime.datetime, datetime.time, int, float)):
            self.creation_date = format_datetime(self.creation_date, 'yyyy-MM-dd HH:mmZ',
                        locale='en')
            print(f'converted self.creation_date to string:{self.creation_date}')

        if isinstance(self.revision_date, (datetime.datetime, datetime.time, int, float)):
            self.revision_date = format_datetime(self.revision_date,
                                            'yyyy-MM-dd HH:mmZ', locale='en')
            print(f'converted self.revision_date to string:{self.revision_date}')
        state = self.__dict__.copy()
        return state

    def __setstate__(self, state):
        self.__dict__.update(state)

        if isinstance(self.creation_date, str):
            self.creation_date = _parse_datetime_header(self.creation_date)
            print(f'_parse_datetime_header self.creation_date from string:{self.creation_date}')

        if isinstance(self.revision_date, str) and (not 'YEAR' in self.revision_date):
            self.revision_date = _parse_datetime_header(self.revision_date)
            print(f'_parse_datetime_header self.revision_date from string:{self.revision_date}')

then run the above script again, and you should see no error

@hoangduytran
Copy link
Author

Changed the solution by convert to string, then parse again back to DateTime data type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant