Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CsvItemExporter fails on py3 #2553

Closed
elacuesta opened this issue Feb 10, 2017 · 5 comments
Closed

CsvItemExporter fails on py3 #2553

elacuesta opened this issue Feb 10, 2017 · 5 comments

Comments

@elacuesta
Copy link
Member

@elacuesta elacuesta commented Feb 10, 2017

from scrapy.exporters import CsvItemExporter
with open('temp.csv', 'w') as f:
    exporter = CsvItemExporter(f)
    exporter.start_exporting()
    exporter.export_item({'a': 'b'})
    exporter.finish_exporting()

The previous snippet works fine on python 2, however the following error appears when running on python 3:

Traceback (most recent call last):
  File "exporter.py", line 5, in <module>
    exporter.export_item({'a': 'b'})
  File "/Users/eugenio/.../venv/lib/python3.5/site-packages/scrapy/exporters.py", line 206, in export_item
    self._write_headers_and_set_fields_to_export(item)
  File "/Users/eugenio/.../venv/lib/python3.5/site-packages/scrapy/exporters.py", line 230, in _write_headers_and_set_fields_to_export
    self.csv_writer.writerow(row)
TypeError: write() argument must be str, not bytes

Am I missing something? Thanks!

@TDominiqueWilliams
Copy link

@TDominiqueWilliams TDominiqueWilliams commented Feb 13, 2017

It appears to be a problem with your "with open('temp.csv', 'w') as f:" statement. The mode you are using does not properly write to the file in the way you need it. When I tested the line myself, I added 'b' to the mode for the open statement. Try the following:

om scrapy.exporters import CsvItemExporter
with open('temp.csv', 'wb') as f:
exporter = CsvItemExporter(f)
exporter.start_exporting()
exporter.export_item({'a': 'b'})
exporter.finish_exporting()

I have pasted a helpful link below:
http://stackoverflow.com/questions/23051062/open-files-in-rt-and-wt-modes

@kmike
Copy link
Member

@kmike kmike commented Feb 14, 2017

I think this is working as intended; as @TDominiqueWilliams said, exporters expect files opened in binary mode.

@kmike
Copy link
Member

@kmike kmike commented Feb 14, 2017

Example at https://doc.scrapy.org/en/latest/topics/exporters.html shows that the file should be opened in binary mode. But we don't document explicitly that file arguments should be opened in binary mode; this is what can be improved.

@elacuesta
Copy link
Member Author

@elacuesta elacuesta commented Feb 14, 2017

Thanks, makes sense now 😄
I'll make a PR to mention this in the docs, if that's OK.

@kmike
Copy link
Member

@kmike kmike commented Feb 14, 2017

That'd be awesome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.