Add a command-line option for overwriting exported file

[kmike]

What do you think about adding an option to overwrite/recreate exported file?
Something like

$ scrapy crawl myspider -o data.jl --overwrite

or

$ scrapy crawl myspider -O data.jl

This is useful during development where old data is not needed. I usually run

$ rm data.jl; scrapy crawl myspider -o data.jl

multiple times. This is not DRY: if I want to change file name for the next iteration and preserve existing file then I must be careful and update both names (of course, the command comes from shell autocompletion).

[dangra]

I've been bitten by this too, I prefer the single option to the modifier.

+1 for -O as short option name and --overwrite-output (or --truncate-output or --new-output) as long option name

[rmax]

Why not overwrite by default and provide an option to append?

On Mon, Jan 20, 2014 at 9:16 AM, Daniel Graña notifications@github.comwrote:

I've been bitten by this too, I prefer the single option to the modifier.

+1 for -O as short option name and --overwrite-output (or
--truncate-output or --new-output) as long option name

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/547#issuecomment-32758317
.

[dangra]

@darkrho : it's an option, but it changes current behavior, does it really deserve a backward incompatible change?

[nramirezuy]

@darkrho and is nicer to have -o and -O.

[kmike]

If we have -o and -O then how can you tell which one is default? Both must be written explicitly.

But @darkrho raised an interesting question because -o doesn't play well with csv, xml and pickle currently: csv adds an extra header, xml starts a new xml instead of adding to previous, and pickle does the same.

As a side note: scrapy crawl myspider -o combined.jl -O last.jl can also be useful, and users will try writing this.

[nyov]

How about emulating wget here, of all things. By default create new files if one exists, as filename.out.1 with incrementing numbers. Then have --append and --overwrite/--replace as seperate options.
Feels like the safest to me, since some exporters apparently don't play nice with appending right now anyway.

We have a PR, could anyone take a look (the question was asked why unittests are failing for it).

[kmike]

-o option creating filename.out.1 filename.out.2 filename.out.3 files by default looks good to me.

[kmike]

By the way, does anyone need appending option? What is it useful for?

[umrashrf]

+1

I realized it was appending the other day when my app broke updating JSON file with scrapy crawl. I don't want it to append.

Not sure if current design allows it, how about sending "only items" to stdout with -o and use it with >/>> ?

[tanmaysahay94]

+1
-o option for creating serialized file names makes sense
Should I go ahead with it?

[alfg]

+1

Any progress on this?

[AaronTao1990]

+1 for -o option

[nyov]

The current PR looks inadequate to me. Sorry.
As an intermediate step, it should just fix the current issue, which is: appending doesn't work. This would mean producing filename.out.1 filename.out.2 filename.out.3 filenames, instead.
(This could be the easy task)

Changing the behaviour to append/overwrite, looks more daunting. Doing it right would mean modifying the Exporter's codebase to handle append or overwrite options like a File interface (IMHO); implementation depending on the actual Exporter backend (local disk, s3, datastore, ...).
E.g. the FTP exporter would need to say "APPE" instead of "STOR" or even ftp-server dependend alternatives.

Also @kmike's proposal

scrapy crawl myspider -o combined.jl -O last.jl can also be useful, and users will try writing this.

would require more thought. It sounds great, but if --append/--overwrite is not just a switch to -o, but an argument taking a destination, Exporters would have to be able to handle that (opening multiple "filepointers" as given). That doesn't warrant an easy tag, I think.

[jschilling1]

I've been using a custom CsvItemExporter to overwrite existing exports and use a custom CSV_DELIMITER in settings. Now i need to set the FEED_URI in the spider. The problem with that is that my custom CsvItemExporter no longer clears existing exports which it only does on init.

[dannykopping]

You can simply output to stdout and redirect that output to a file:
scrapy runspider spider.py -t json --nolog -o - > out.json

[lucaspottersky]

You can simply output to stdout and redirect that output to a file:

@dannykopping this is not an option when scheduling through scrapyd.

[robkorv]

In the meantime one could subclass FileFeedStorage. I just typed an answer over at http://stackoverflow.com/a/40823149/2546958 before I spotted this issue.