Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore code produces inconsistent timestamps/permissions #1212

Closed
lloeki opened this issue Sep 5, 2017 · 6 comments · Fixed by #2906
Closed

Restore code produces inconsistent timestamps/permissions #1212

lloeki opened this issue Sep 5, 2017 · 6 comments · Fixed by #2906
Labels
category: restore state: need implementing cause/request established, need work/solution type: bug

Comments

@lloeki
Copy link
Contributor

lloeki commented Sep 5, 2017

Follow-up to #982 and #1044, relevant info excerpted below.

Output of restic version

> restic version
restic 0.6.0-rc.1 (v0.6.0-rc.1-0-g17ff41af)
compiled with go1.8.3 on darwin/amd64

How did you start restic exactly? (Include the complete command line)

See repro steps below.

What backend/server/service did you use?

Local + S3.

Expected behavior

  • timestamps should be restored consistently for intermediate directories
  • rights should be restored consistently for intermediate directories

Actual behavior

  • timestamps and rights are not restored correctly depending on the pattern and the fact that the path matches or not.

Steps to reproduce the behavior

mkdir test/{foo,bar}
chmod 700 test/foo
touch test/{foo,bar}/{what,wut}
restic -r repo init
restic -r repo backup test
restic -r repo restore f608e300 --include 'wut' --target restored
rm -rf restored
ls -ld restored/test      # timestamp is at restore date, rights are 0700, should be 0755
ls -ld restored/test/bar  # timestamp is at restore date, rights are 0700, should be 0755
ls -ld restored/test/foo  # timestamp is at restore date, rights are 0700
restic -r repo restore f608e300 --include 'foo' --target restored
ls -ld restored/test      # timestamp is at restore date, rights are 0700, should be 0755
ls -ld restored/test/foo  # timestamp is at backup date, rights are 0700
rm -rf restored
restic -r repo restore f608e300 --include 'test' --target restored
ls -ld restored/test      # timestamp is at restore date, rights are 0755
ls -ld restored/test/bar  # timestamp is at backup date, rights are 0755
ls -ld restored/test/foo  # timestamp is at backup date, rights are 0700
restic -r repo restore f608e300 --include '/test/**' --target restored
ls -ld restored/test      # timestamp is at restore date, rights are 0700, should be 0755
ls -ld restored/test/bar  # timestamp is at backup date, rights are 0755
ls -ld restored/test/foo  # timestamp is at backup date, rights are 0700
rm -rf restored

Do you have any idea what may have caused this?

Yes.

--- a/src/restic/restorer.go
+++ b/src/restic/restorer.go 
+			// FIXME: timestamp is restored only for final directory matching the pattern, e.g only for baz in foo/bar/baz, if baz is a dir, and in no other case
+			// instead, restoreTo could return a `hasRestorable` flag (along with the existing err) set to true if something was indeed found to be restored during recursion
 			if selectedForRestore {
 				// Restore directory timestamp at the end. If we would do it earlier, restoring files within
 				// the directory would overwrite the timestamp of the directory they are in.
@@ -96,6 +102,9 @@ func (res *Restorer) restoreNodeTo(node *Node, dir string, dst string, idx *Hard
 	if err != nil && os.IsNotExist(errors.Cause(err)) {
 		debug.Log("create intermediate paths")
 
+		// FIXME: not mkdir str, restore node for each path component (which requires passing the current node stack), else attrs are incorrect
+		// also, one should take care of not doing restore ops twice or more if the node path component is already there.
 		// Create parent directories and retry
 		err = fs.MkdirAll(filepath.Dir(dstPath), 0700)
 		if err == nil || os.IsExist(errors.Cause(err)) { 
@lloeki
Copy link
Contributor Author

lloeki commented Sep 5, 2017

I'll be working on this as time permits.

@fd0 fd0 added the type: bug label Sep 5, 2017
@fd0 fd0 added the state: need implementing cause/request established, need work/solution label May 26, 2018
@CapacitorSet
Copy link

This issue is still partially present in v0.9.4. I was able to reproduce the timestamp mismatch, but not the permissions one.

@deviantintegral
Copy link

I am able to reproduce both permissions and timestamp bugs on restic 0.9.6 compiled with go1.13.4 on darwin/amd64.

I took the above and turned it into a reproducible script. Note the stat calls are BSD style and would need tweaking for running on Linux.

#!/bin/bash

check_equal_permission()
{
  echo "Comparing permissions on $1 to $2..."
  (test $(stat -f %p $1) -eq $(stat -f %p $2) && echo "Passed!") || (echo "Comparison failed." && ls -ld $1 && ls -ld $2)
  echo ""
}

check_equal_modified()
{
  echo "Comparing modified dates on $1 to $2..."
  (test $(stat -f %m $1) -eq $(stat -f %m $2) && echo "Passed!") || (echo "Comparison failed." && ls -ld $1 && ls -ld $2)
  echo ""
}

check_file()
{
  check_equal_permission $1 $2
  check_equal_modified $1 $2
}

mkdir -p test/{foo,bar}
chmod 700 test/foo
touch -t 201901010000 test/{foo,bar}/{what,wut} test/{foo,bar} test
restic --password-command='echo pass' -r repo init
restic --password-command='echo pass' -r repo backup test
restic --password-command='echo pass' -r repo restore latest --include 'wut' --target restored
check_file test restored/test
check_file test/foo restored/test/foo
check_file test/bar restored/test/bar

rm -rf restored

restic --password-command='echo pass' -r repo restore latest --include 'foo' --target restored
check_file test restored/test
check_file test/foo restored/test/foo
rm -rf restored

restic --password-command='echo pass' -r repo restore latest --include 'test' --target restored
check_file test restored/test
check_file test/foo restored/test/foo
check_file test/bar restored/test/bar

restic --password-command='echo pass' -r repo restore latest --include '/test/**' --target restored
check_file test restored/test
check_file test/foo restored/test/foo
check_file test/bar restored/test/bar

rm -rf test repo restored

I get the following output:

created restic repository 3962af18b5 at repo

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
created new cache in /Users/andrew/Library/Caches/restic

Files:           4 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 358 B

processed 4 files, 0 B in 0:00
snapshot af5381ad saved
restoring <Snapshot af5381ad of [/Users/andrew/Desktop/test] at 2020-08-10 15:48:38.45648 -0400 EDT by andrew@acutus.lan> to restored
Comparing permissions on test to restored/test...
Comparison failed.
drwxr-xr-x  4 andrew  staff  128  1 Jan  2019 test
drwx------  4 andrew  staff  128 10 Aug 15:48 restored/test

Comparing modified dates on test to restored/test...
Comparison failed.
drwxr-xr-x  4 andrew  staff  128  1 Jan  2019 test
drwx------  4 andrew  staff  128 10 Aug 15:48 restored/test

Comparing permissions on test/foo to restored/test/foo...
Passed!

Comparing modified dates on test/foo to restored/test/foo...
Comparison failed.
drwx------  4 andrew  staff  128  1 Jan  2019 test/foo
drwx------  3 andrew  staff  96 10 Aug 15:48 restored/test/foo

Comparing permissions on test/bar to restored/test/bar...
Comparison failed.
drwxr-xr-x  4 andrew  staff  128  1 Jan  2019 test/bar
drwx------  3 andrew  staff  96 10 Aug 15:48 restored/test/bar

Comparing modified dates on test/bar to restored/test/bar...
Comparison failed.
drwxr-xr-x  4 andrew  staff  128  1 Jan  2019 test/bar
drwx------  3 andrew  staff  96 10 Aug 15:48 restored/test/bar

restoring <Snapshot af5381ad of [/Users/andrew/Desktop/test] at 2020-08-10 15:48:38.45648 -0400 EDT by andrew@acutus.lan> to restored
Comparing permissions on test to restored/test...
Comparison failed.
drwxr-xr-x  4 andrew  staff  128  1 Jan  2019 test
drwx------  3 andrew  staff  96 10 Aug 15:48 restored/test

Comparing modified dates on test to restored/test...
Comparison failed.
drwxr-xr-x  4 andrew  staff  128  1 Jan  2019 test
drwx------  3 andrew  staff  96 10 Aug 15:48 restored/test

Comparing permissions on test/foo to restored/test/foo...
Passed!

Comparing modified dates on test/foo to restored/test/foo...
Passed!

restoring <Snapshot af5381ad of [/Users/andrew/Desktop/test] at 2020-08-10 15:48:38.45648 -0400 EDT by andrew@acutus.lan> to restored
Comparing permissions on test to restored/test...
Passed!

Comparing modified dates on test to restored/test...
Passed!

Comparing permissions on test/foo to restored/test/foo...
Passed!

Comparing modified dates on test/foo to restored/test/foo...
Passed!

Comparing permissions on test/bar to restored/test/bar...
Passed!

Comparing modified dates on test/bar to restored/test/bar...
Passed!

restoring <Snapshot af5381ad of [/Users/andrew/Desktop/test] at 2020-08-10 15:48:38.45648 -0400 EDT by andrew@acutus.lan> to restored
Comparing permissions on test to restored/test...
Passed!

Comparing modified dates on test to restored/test...
Passed!

Comparing permissions on test/foo to restored/test/foo...
Passed!

Comparing modified dates on test/foo to restored/test/foo...
Passed!

Comparing permissions on test/bar to restored/test/bar...
Passed!

Comparing modified dates on test/bar to restored/test/bar...
Passed!

kitone added a commit to kitone/restic that referenced this issue Aug 28, 2020
kitone added a commit to kitone/restic that referenced this issue Aug 28, 2020
…dForRestore to determine if we should visit leaveDir
@kitone
Copy link
Contributor

kitone commented Aug 28, 2020

Hello there,
I have work a little on this issue, can you check my branch give me some feedback ?
https://github.com/kitone/restic/tree/fix-inconsistent-timestamps-permissions

About the change :

  • The first commit add test case
  • The second and third do some refactoring and add log information, i was pretty hard to debug treeVisitor proc functions and understand what happenning.
  • The last one fix the issue

I guess there is still something wrong, using the original reproductible script give me something new.
The fix introduce a new case, trying to restore metadata on a file not restored (I don't investigate yet)
ignoring error for /test/bar: UtimesNano: no such file or directory

More information :

Using the simplified version script

#!/bin/bash

export DEBUG_LOG=restic-debug.log

check_equal_permission()
{
  echo "Comparing permissions on $1 to $2..."
  (test $(stat -f %p $1) -eq $(stat -f %p $2) && echo "Passed!") || (echo "#### Comparison failed." && ls -ld $1 && ls -ld $2)
  echo ""
}

check_equal_modified()
{
  echo "Comparing modified dates on $1 to $2..."
  (test $(stat -f %m $1) -eq $(stat -f %m $2) && echo "Passed!") || (echo "#### Comparison failed." && ls -ld $1 && ls -ld $2)
  echo ""
}

check_file()
{
  check_equal_permission $1 $2
  check_equal_modified $1 $2
}

rm restic-debug.log
mkdir -p test/{dir1,dir2}
mkdir -p test/dir1/dir3/
chmod 700 test/dir1
chmod 700 test/dir1/dir3
touch -t 201901010000 test/{dir1,dir2}/{file1,file2} test/{dir1,dir2} test test/dir1/dir3/file3
restic --password-command='echo pass' -r repo init
restic --password-command='echo pass' -r repo backup test

rm restic-debug.log

restic --password-command='echo pass' -r repo restore latest --include 'file1' --target restored
check_file test restored/test

rm -rf test repo restored

./script.sh && cat restic-debug.log | grep 'pass\|SelectFilter'
Show that leaveDir on the second pass is never executed, this is why the metadata of the root directory isn't restored like promised in :

	err := res.traverseTree(ctx, dst, string(filepath.Separator), *res.sn.Tree, treeVisitor{
		enterDir: func(node *restic.Node, target, location string) error {
			debug.Log("first pass: enterDir mkdir %q, leaveDir should restore metadata", location)
			// create dir with default permissions
			// #leaveDir restores dir metadata after visiting all children
			return fs.MkdirAll(target, 0700)
		},

		visitNode: func(node *restic.Node, target, location string) error {
			debug.Log("first pass: visitNode mkdir %q, second pass (leaveDir) should restore metadata", location)
			// create parent dir with default permissions
			// second pass #leaveDir restores dir metadata after visiting/restoring all children
			err := fs.MkdirAll(filepath.Dir(target), 0700)
restic/main.go:83	main.main	1	main []string{"../restic", "--password-command=echo pass", "-r", "repo", "restore", "latest", "--include", "file1", "--target", "restored"}
restorer/restorer.go:221	restorer.(*Restorer).firstTreePass	1     	first pass: for "/Users/kitone/restic/issues_1212/restored"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false true for "/test"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false true for "/test/dir1"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false true for "/test/dir1/dir3"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false false for "/test/dir1/dir3/file3"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned true false for "/test/dir1/file1"
restorer/restorer.go:232	restorer.(*Restorer).firstTreePass.func2	1	first pass: visitNode mkdir "/test/dir1/file1", second pass (leaveDir) should restore metadata
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false false for "/test/dir1/file2"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false true for "/test/dir2"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned true false for "/test/dir2/file1"
restorer/restorer.go:232	restorer.(*Restorer).firstTreePass.func2	1	first pass: visitNode mkdir "/test/dir2/file1", second pass (leaveDir) should restore metadata
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false false for "/test/dir2/file2"
restorer/restorer.go:278	restorer.(*Restorer).secondTreePass	1	      second pass: for "/Users/kitone/restic/issues_1212/restored"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false true for "/test"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false true for "/test/dir1"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false true for "/test/dir1/dir3"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false false for "/test/dir1/dir3/file3"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned true false for "/test/dir1/file1"
restorer/restorer.go:287	restorer.(*Restorer).secondTreePass.func2	1	second pass: visitNode "/test/dir1/file1"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false false for "/test/dir1/file2"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false true for "/test/dir2"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned true false for "/test/dir2/file1"
restorer/restorer.go:287	restorer.(*Restorer).secondTreePass.func2	1	second pass: visitNode "/test/dir2/file1"
restorer/restorer.go:93	  restorer.(*Restorer).traverseTree	1         SelectFilter returned false false for "/test/dir2/file2"

after fixing we can see that the leaveDir is executed on the second tree pass :

restic/main.go:83	main.main	1	main []string{"../restic", "--password-command=echo pass", "-r", "repo", "restore", "latest", "--include", "file1", "--target", "restored"}
restorer/restorer.go:221	restorer.(*Restorer).firstTreePass	1         first pass: for "/Users/kitone/restic/issues_1212/restored"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false true for "/test"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false true for "/test/dir1"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false true for "/test/dir1/dir3"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false false for "/test/dir1/dir3/file3"
restorer/restorer.go:260	restorer.(*Restorer).firstTreePass.func3	1 	first pass: leaveDir "/test/dir1/dir3", noop
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned true false for "/test/dir1/file1"
restorer/restorer.go:232	restorer.(*Restorer).firstTreePass.func2	1	  first pass: visitNode mkdir "/test/dir1/file1", second pass (leaveDir) should restore metadata
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false false for "/test/dir1/file2"
restorer/restorer.go:260	restorer.(*Restorer).firstTreePass.func3	1	  first pass: leaveDir "/test/dir1", noop
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false true for "/test/dir2"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned true false for "/test/dir2/file1"
restorer/restorer.go:232	restorer.(*Restorer).firstTreePass.func2	1	  first pass: visitNode mkdir "/test/dir2/file1", second pass (leaveDir) should restore metadata
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false false for "/test/dir2/file2"
restorer/restorer.go:260	restorer.(*Restorer).firstTreePass.func3	1	  first pass: leaveDir "/test/dir2", noop
restorer/restorer.go:260	restorer.(*Restorer).firstTreePass.func3	1	  first pass: leaveDir "/test", noop
restorer/restorer.go:278	restorer.(*Restorer).secondTreePass	1	        second pass: for "/Users/kitone/restic/issues_1212/restored"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false true for "/test"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false true for "/test/dir1"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false true for "/test/dir1/dir3"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false false for "/test/dir1/dir3/file3"
restorer/restorer.go:308	restorer.(*Restorer).secondTreePass.func3	1	  second pass: leaveDir "/test/dir1/dir3", restore metadata
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned true false for "/test/dir1/file1"
restorer/restorer.go:287	restorer.(*Restorer).secondTreePass.func2	1	  second pass: visitNode "/test/dir1/file1"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false false for "/test/dir1/file2"
restorer/restorer.go:308	restorer.(*Restorer).secondTreePass.func3	1	  second pass: leaveDir "/test/dir1", restore metadata
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false true for "/test/dir2"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned true false for "/test/dir2/file1"
restorer/restorer.go:287	restorer.(*Restorer).secondTreePass.func2	1	  second pass: visitNode "/test/dir2/file1"
restorer/restorer.go:93	restorer.(*Restorer).traverseTree	1             SelectFilter returned false false for "/test/dir2/file2"
restorer/restorer.go:308	restorer.(*Restorer).secondTreePass.func3	1	  second pass: leaveDir "/test/dir2", restore metadata
restorer/restorer.go:308	restorer.(*Restorer).secondTreePass.func3	1	  second pass: leaveDir "/test", restore metadata

The fix make pass all the reproductible original script.

By the way, the fix broke one test

--- FAIL: TestRestorerTraverseTree (0.01s)
    --- FAIL: TestRestorerTraverseTree/#04 (0.00s)
        testing.go:30: using low-security KDF parameters for test
        config.go:65: disabling check of the chunker polynomial
        restorer_test.go:506: step 1, leaveDir(/dir): expected no more than 1 function calls

@MichaelEischer
Copy link
Member

@kitone Please open a PR, that makes it way easier to discuss the implementation.

kitone added a commit to kitone/restic that referenced this issue Aug 30, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
@kitone
Copy link
Contributor

kitone commented Aug 30, 2020

Done, PR fix the reproductible script (minus one broken test case)

kitone added a commit to kitone/restic that referenced this issue Sep 1, 2020
kitone added a commit to kitone/restic that referenced this issue Sep 1, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
kitone added a commit to kitone/restic that referenced this issue Sep 1, 2020
kitone added a commit to kitone/restic that referenced this issue Sep 1, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
kitone added a commit to kitone/restic that referenced this issue Sep 5, 2020
kitone added a commit to kitone/restic that referenced this issue Sep 5, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
kitone added a commit to kitone/restic that referenced this issue Sep 5, 2020
kitone added a commit to kitone/restic that referenced this issue Sep 5, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
kitone added a commit to kitone/restic that referenced this issue Sep 5, 2020
kitone added a commit to kitone/restic that referenced this issue Sep 5, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
@MichaelEischer MichaelEischer linked a pull request Oct 4, 2020 that will close this issue
7 tasks
kitone added a commit to kitone/restic that referenced this issue Oct 9, 2020
kitone added a commit to kitone/restic that referenced this issue Oct 9, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
kitone added a commit to kitone/restic that referenced this issue Oct 9, 2020
kitone added a commit to kitone/restic that referenced this issue Oct 9, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
kitone added a commit to kitone/restic that referenced this issue Oct 9, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
kitone added a commit to kitone/restic that referenced this issue Oct 10, 2020
kitone added a commit to kitone/restic that referenced this issue Oct 10, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
MichaelEischer pushed a commit to kitone/restic that referenced this issue Oct 10, 2020
MichaelEischer pushed a commit to kitone/restic that referenced this issue Oct 10, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
SanchithHegde pushed a commit to SanchithHegde/restic that referenced this issue Oct 31, 2020
SanchithHegde pushed a commit to SanchithHegde/restic that referenced this issue Oct 31, 2020
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
mfrischknecht pushed a commit to mfrischknecht/restic that referenced this issue Jun 14, 2022
mfrischknecht pushed a commit to mfrischknecht/restic that referenced this issue Jun 14, 2022
…ions.

Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: restore state: need implementing cause/request established, need work/solution type: bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants