--resume-extract #327

gperciva · 2018-09-15T22:10:05Z

No description provided.

gperciva · 2018-09-15T22:11:12Z

Example, with a weird example of an archive containing two copies of the same file:

$ ./tarsnap -tf random-100k-random-100k
random-100k
random-100k
$ rm random-100k
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k
Connection ended with in / out:	204029	1185
$ rm random-100k
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k --resume-extract
Connection ended with in / out:	103500	1039

cperciva · 2018-09-15T23:12:35Z

Do we want to support resume-extract as a configuration file setting? It seems like it could be dangerous as a "set and forget" option -- unless there's a compelling reason it seems like it might be better to leave it as command-line-only until someone comes forward saying that they need to put it into a configuration file.

And if we're not supporting it as a configuration file option, we don't need --no-resume-extract.

gperciva · 2018-09-15T23:20:47Z

I wondered about that, but then I asked myself the flipped version: if I think it's a risky thing to have as a "set and forget" option, then clearly I don't have confidence in it doing the right thing.

I'm happy to have it as command-line-only if you prefer that, though.

cperciva · 2018-09-21T23:03:24Z

While I agree that --resume-extract should normally be safe, I can imagine situations where it could cause problems; for example, if someone archives (using @- functionality) synthetic tarballs created by tools (e.g., database dumps) which don't fill in timestamps. If someone deliberately uses --resume-extract when extracting such broken archives they deserve what they get; but I could imagine someone setting that in a configuration file and forgetting about it.

So for the sake of minimizing user error I think let's keep it as command-line-only... at least until someone steps forward and says "I really wish I could put this into my configuration file because ".

gperciva · 2018-09-22T20:44:54Z

Ok, removed resume-extract from a config file option. Rebased history.

cperciva · 2018-09-22T21:55:48Z

tar/bsdtar.c

@@ -1779,6 +1782,11 @@ dooption(struct bsdtar *bsdtar, const char * conf_opt,

 		bsdtar->option_quiet = 1;
 		bsdtar->option_quiet_set = 1;
+	} else if (strcmp(conf_opt, "resume-extract") == 0) {


This code here is for processing configuration file options. Why do we need it?

cperciva · 2018-09-22T21:56:30Z

tar/tarsnap.1-man.in

@@ -624,6 +624,11 @@ Ignore any
 \fBquiet\fP
 option specified in a configuration file.
 .TP
+\fB\--no-resume-extract\fP


Not needed any more?

cperciva · 2018-09-22T21:56:41Z

tar/tarsnap.1-mdoc.in

@@ -561,6 +561,10 @@ option specified in a configuration file.
 Ignore any
 .Cm quiet
 option specified in a configuration file.
+.It Fl -no-resume-extract


Not needed any more?

cperciva · 2018-09-22T21:57:40Z

Looks like there's still some --no-resume-extract scattered around.

gperciva · 2018-09-23T18:50:01Z

Sorry, was distracted by a viola gig.

git grep no-resume now returns nothing
I added (x mode only) to the man-page
fixed a minor typo in NEWS ("an --dump-config" -> "a --dump-config")

cperciva · 2018-09-26T20:11:06Z

tar/read.c

+			/* Don't extract if the file already matches it. */
+			if (bsdtar->option_resume_extract) {
+				if ((r = check_skip_file(
+				    archive_entry_pathname(entry), st)) == -1) {


How carefully did you check that archive_entry_pathname(entry) is the correct path to the file we would create? I think this is correct but with options like -C and -s and the handling of absolute paths and paths containing .. I want to be sure that we have this right and aren't accidentally comparing metadata against the wrong place.

woah, -s looks wacky! Didn't occur to me to check that one, I'll do it soon.

gperciva · 2018-11-13T20:02:57Z

I think this is working. I made sure that check_skip_file() came after edit_pathname(). See below for the tests; I think I covered all the weird cases.

Basic usage:

$ ./tarsnap -tf random-100k-random-100k
random-100k
random-100k
$ rm -f random-100k
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k
Connection ended with in / out:	204029	1185
$ rm -f random-100k
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k --resume-extract
Connection ended with in / out:	103500	1039
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k --resume-extract
Connection ended with in / out:	2483	747
$

-C out/

$ mkdir out/
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k --resume-extract -C out/
Connection ended with in / out:	103500	1039
$

-s

$ rm -f foo-100k
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k --resume-extract -s "/random/foo/g"
Connection ended with in / out:	103500	1039
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k --resume-extract -s "/random/foo/g"
Connection ended with in / out:	2483	747
$

-C out/ -s

$ mkdir out/
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k --resume-extract -s "/random/foo/g" -C out
Connection ended with in / out:	103500	1039
$ ls out/
foo-100k
$

absolute paths combined with -C

$ ./tarsnap -tf random-100k-random-100k-absolute
/home/td/src/tarsnap/build/random-100k
/home/td/src/tarsnap/build/random-100k
$ rm -f random-100k
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k-absolute --resume-extract -P -C out
Connection ended with in / out:	103598	1039
$ ./tarsnap --debug-network-stats -xf random-100k-random-100k-absolute --resume-extract -P -C out
Connection ended with in / out:	2581	747
td@mac: ~/src/tarsnap/build (resume-extract)
$ ls out/
home
td@mac: ~/src/tarsnap/build (resume-extract)
$

relative dirs

$ cd out/
$ ls
$ ../tarsnap --debug-network-stats -x -f random-100k-random-100k-updir --resume-extract -P
Connection ended with in / out:	103524	1039
$ ../tarsnap --debug-network-stats -x -f random-100k-random-100k-updir --resume-extract -P
Connection ended with in / out:	2507	747
$ ls
$

cperciva · 2018-12-29T04:52:32Z

tar/tarsnap.1-man.in

+(x mode only)
+Don't extract files whose filesize and mtime matches existing files on the
+disk.  Primarily used to resume an archive extraction which was interrupted.
+.TP


Probably worth adding something along the lines of "This differs from -k in that --resume-extract will overwrite a file if the size or modification time do not match, as can happen if tarsnap is killed partway through extracting a file."

cperciva · 2018-12-29T04:52:57Z

NEWS.md

@@ -7,6 +7,8 @@
  non-blank lines read from config files.
 - tarsnap now gives an error if there are unused command-line arguments.
  (i.e. "tarsnap -d -f a1 a2", where "a2" is unused.)
+- tarsnap now accepts a --resume-extract option to skip extracting files whose


This needs to have a merge conflict fixed.

cperciva · 2018-12-29T04:56:47Z

tar/read.c

+		goto err0;
+	}
+
+	/* Compare file size and mtime. */


The comment here should point out that we're comparing the seconds of the mtime, not the complete timespec; some filesystems don't have sub-second timestamp precision, so comparing timespecs would produce a lot of false negatives.

cperciva · 2018-12-29T05:04:04Z

Ok, looks good. I've confirmed by inspection of the libarchive source code that archive_entry_pathname is definitely the path to where we're going to write on disk.

gperciva · 2019-01-01T21:00:48Z

Rebased to fix merge conflicts, but I left the two recent fixes to your comments are separate commits for easy examination.

Fun fact: I misread the POSIX and linux man-pages about stat(2). My brain just couldn't process that it was actually st_mtim instead of st_mtime. The latter is #defined as st_mtim.tv_sec for backwards compatibility on my desktop, so I didn't realize the discrepancy.

gperciva · 2019-05-29T18:22:01Z

Updated to current master. I checked that all the tests in #327 (comment) work.

gperciva force-pushed the resume-extract branch from 17f1dd6 to c22b387 Compare September 22, 2018 20:44

cperciva reviewed Sep 22, 2018

View reviewed changes

gperciva force-pushed the resume-extract branch from c22b387 to 13f7e7a Compare September 23, 2018 18:48

cperciva reviewed Sep 26, 2018

View reviewed changes

gperciva force-pushed the resume-extract branch from 13f7e7a to 926d63c Compare November 13, 2018 20:00

cperciva reviewed Dec 29, 2018

View reviewed changes

gperciva force-pushed the resume-extract branch from 926d63c to 1251971 Compare January 1, 2019 20:58

Add --resume-extract

62af767

gperciva force-pushed the resume-extract branch from 1251971 to 31e4232 Compare May 29, 2019 18:00

gperciva added 3 commits May 29, 2019 11:02

man: --resume-extract

c40119e

NEWS.md: --resume-extract

1e99db5

bash-completion: --resume-extract

3b4153c

gperciva force-pushed the resume-extract branch from 31e4232 to 3b4153c Compare May 29, 2019 18:02

cperciva merged commit 8e8e444 into master May 29, 2019

gperciva deleted the resume-extract branch May 29, 2019 23:06

gperciva mentioned this pull request May 1, 2023

Add --resume-extract option #116

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

--resume-extract #327

--resume-extract #327

gperciva commented Sep 15, 2018

gperciva commented Sep 15, 2018

cperciva commented Sep 15, 2018

gperciva commented Sep 15, 2018

cperciva commented Sep 21, 2018

gperciva commented Sep 22, 2018

cperciva Sep 22, 2018

cperciva Sep 22, 2018

cperciva Sep 22, 2018

cperciva commented Sep 22, 2018

gperciva commented Sep 23, 2018

cperciva Sep 26, 2018

gperciva Sep 26, 2018

gperciva commented Nov 13, 2018 •

edited

cperciva Dec 29, 2018

cperciva Dec 29, 2018

cperciva Dec 29, 2018

cperciva commented Dec 29, 2018

gperciva commented Jan 1, 2019

gperciva commented May 29, 2019

--resume-extract #327

--resume-extract #327

Conversation

gperciva commented Sep 15, 2018

gperciva commented Sep 15, 2018

cperciva commented Sep 15, 2018

gperciva commented Sep 15, 2018

cperciva commented Sep 21, 2018

gperciva commented Sep 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cperciva commented Sep 22, 2018

gperciva commented Sep 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gperciva commented Nov 13, 2018 • edited

Basic usage:

-C out/

-s

-C out/ -s

absolute paths combined with -C

relative dirs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cperciva commented Dec 29, 2018

gperciva commented Jan 1, 2019

gperciva commented May 29, 2019

gperciva commented Nov 13, 2018 •

edited