Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JobTask.dump raises TypeError when replacing bytes to string #3284

Open
marinelay opened this issue Apr 8, 2024 · 0 comments
Open

JobTask.dump raises TypeError when replacing bytes to string #3284

marinelay opened this issue Apr 8, 2024 · 0 comments

Comments

@marinelay
Copy link

marinelay commented Apr 8, 2024

In the file luigi.contrib.hadoop.py, the JobTask.dump method shows a weird behavior.

def dump(self, directory=''):
"""
Dump instance to file.
"""
with self.no_unpicklable_properties():
file_name = os.path.join(directory, 'job-instance.pickle')
if self.__module__ == '__main__':
d = pickle.dumps(self)
module_name = os.path.basename(sys.argv[0]).rsplit('.', 1)[0]
d = d.replace(b'(c__main__', "(c" + module_name)
open(file_name, "wb").write(d)
else:
pickle.dump(self, open(file_name, "wb"))

I believe the variable d is bytes type by pickle.dumps (https://docs.python.org/3/library/pickle.html#pickle.dumps),
and the variable module_name should be string type because sys.argv[0] is string type (https://docs.python.org/3/library/sys.html#sys.argv).

Then, d.replace(b'(c__main__', "(c" + module_name) always return TypeError: a bytes-like object is required, not 'str' because its replacement should be any bytes-like object (https://docs.python.org/3/library/stdtypes.html#bytes.replace).
It is simplified code of this situation:

from luigi.contrib.hadoop import JobTask

class My(JobTask):
    pass

a = My()
a.dump('my')

Output :

Traceback (most recent call last):
  File "/home/wonseok/current/luigi/my_test.py", line 8, in <module>
    a.dump('my')
  File "/home/wonseok/current/luigi/luigi/contrib/hadoop.py", line 974, in dump
    d = d.replace(b'(c__main__', "(c" + module_name)
TypeError: a bytes-like object is required, not 'str'

It is related issue #2402
If I'm mistaken, I'd appreciate it if you could let me know.
Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant