Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

utils: Fix broken Windows filelock #14426

Merged
merged 1 commit into from Oct 20, 2018
Merged

utils: Fix broken Windows filelock #14426

merged 1 commit into from Oct 20, 2018

Conversation

@ken2812221
Copy link
Contributor

@ken2812221 ken2812221 commented Oct 7, 2018

Fix broken filelock on Windows, also add a test for this. It's a regression introduced by #13862.

@ken2812221
Copy link
Contributor Author

@ken2812221 ken2812221 commented Oct 7, 2018

@ch4ot1c
Copy link
Contributor

@ch4ot1c ch4ot1c commented Oct 7, 2018

Lgtm, nice test

@donaloconnor
Copy link
Contributor

@donaloconnor donaloconnor commented Oct 8, 2018

utACK 6d336e3

src/fs.cpp Outdated
@@ -89,7 +89,7 @@ bool FileLock::TryLock()
return false;
}
_OVERLAPPED overlapped = {0};
if (!LockFileEx(hFile, LOCKFILE_EXCLUSIVE_LOCK | LOCKFILE_FAIL_IMMEDIATELY, 0, 0, 0, &overlapped)) {
if (!LockFileEx(hFile, LOCKFILE_EXCLUSIVE_LOCK | LOCKFILE_FAIL_IMMEDIATELY, 0, -1, -1, &overlapped)) {
Copy link
Member

@Empact Empact Oct 9, 2018

nit: std::numeric_limits<DWORD>::max() or explicitly casting to unsigned might be more clear.

@ken2812221 ken2812221 force-pushed the filelock-test branch 3 times, most recently from ad5f211 to 474d022 Oct 9, 2018
@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Oct 9, 2018

Reviewers, this pull request conflicts with the following ones:
  • #14320 ([bugfix] wallet: Fix duplicate fileid detection by ken2812221)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@donaloconnor
Copy link
Contributor

@donaloconnor donaloconnor commented Oct 9, 2018

utACK 474d022

self.setup_clean_chain = True
self.num_nodes = 2

def get_node_output(self, *, index, ret_code_expected):
Copy link
Member

@promag promag Oct 10, 2018

Move to node class as get_output? If that's controversial then extract to a module function for now? Then below:

_, output = get_node_output(node=self.nodes[1], ret_code_expected=1)

Copy link
Contributor Author

@ken2812221 ken2812221 Oct 11, 2018

Done

@ken2812221 ken2812221 force-pushed the filelock-test branch 2 times, most recently from 0129895 to 851df19 Oct 11, 2018

def run_test(self):
self.nodes[1].stop()
self.log.info("Using datadir " + self.nodes[0].datadir)
Copy link
Contributor

@practicalswift practicalswift Oct 14, 2018

Specify string format arguments as logging function parameters to allow for lazy evaluation.

Copy link
Contributor Author

@ken2812221 ken2812221 Oct 14, 2018

Done

@jnewbery
Copy link
Member

@jnewbery jnewbery commented Oct 18, 2018

This seems like a reasonable change (although I know nothing about the windows API). LockFileEx documentation is here: https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-lockfileex.

The test case is good, but it has the wrong file permissions and replicates functionality that is already available in the test framework. I've reimplemented it here: https://github.com/jnewbery/bitcoin/tree/pr14426.1 which uses the assert_start_raises_init_error() method. It's more concise, runs faster, also covers trying to open a wallet being used by a different bitcoind process, and I think is clearer. Feel free to take that commit in this PR.

@ken2812221
Copy link
Contributor Author

@ken2812221 ken2812221 commented Oct 18, 2018

Thanks, I didn't know test framework that much, just trying to do what I want. Your commit looks good to me, I'll take it. The only thing I change is to skip wallet test if it's not compiled.

Copy link
Contributor

@ryanofsky ryanofsky left a comment

utACK c7296a98e56a3a38730575c7aa7f524f10e2a0a9, which seems to be updated with John's test.

Although the LockFileEx documentation just says:

Locking a region that goes beyond the current end-of-file position is not an error

The LockFile documentation is more specific and says:

You can lock bytes that are beyond the end of the current file. This is useful to coordinate adding records to the end of a file

which I think explains why the fix works.

@jnewbery
Copy link
Member

@jnewbery jnewbery commented Oct 18, 2018

utACK c7296a98e56a3a38730575c7aa7f524f10e2a0a9.

I think the feature_notifications.py failure in appveyer is unrelated. The feature_filelock.py failure is due to a path slash mismatch: C:/Users\appveyor instead of C:\Users\appveyor.

@donaloconnor
Copy link
Contributor

@donaloconnor donaloconnor commented Oct 18, 2018

@ken2812221 should we explicitly call UnlockFile before closing the handle. The OS does unlock it but it's not instant potentially. Maybe safer to unlock before closing

@ken2812221
Copy link
Contributor Author

@ken2812221 ken2812221 commented Oct 18, 2018

@donaloconnor I am not sure what condition would be unsafe if we don't unlock the file. Also, the boost implementation here only close file in deconstructor.

@ken2812221 ken2812221 force-pushed the filelock-test branch 3 times, most recently from e5837e4 to d49f5ad Oct 18, 2018
@donaloconnor
Copy link
Contributor

@donaloconnor donaloconnor commented Oct 18, 2018

@ken2812221 I did a test there across 2 processes to see if there's any weird race conditions with not unlocking before close and there's not. I thought being explicit might be better. Anyway no big deal.

tested ACK.

@jnewbery
Copy link
Member

@jnewbery jnewbery commented Oct 19, 2018

utACK 369244f

Copy link
Contributor

@ryanofsky ryanofsky left a comment

utACK 369244f, only change is setting ErrorMatch.PARTIAL_REGEX flag in test.

@sipa
Copy link
Member

@sipa sipa commented Oct 20, 2018

utACK 369244f

@sipa sipa merged commit 369244f into bitcoin:master Oct 20, 2018
2 checks passed
sipa added a commit that referenced this issue Oct 20, 2018
369244f utils: Fix broken Windows filelock (Chun Kuan Lee)

Pull request description:

  Fix broken filelock on Windows, also add a test for this. It's a regression introduced by #13862.

Tree-SHA512: 15665b1930cf39ec71f3ab07def8e2897659f6fd4d2de749d63a5a8ec920e4a04282f12bc262f242b1b3d14d2dd9fa191ddbcf16a46fb927b5b2b14d9f6b5d01
@ken2812221 ken2812221 deleted the filelock-test branch Oct 23, 2018
laanwj added a commit that referenced this issue Apr 11, 2019
0b3a654 Avoid redefine warning (Peter Bushnell)

Pull request description:

  Wrap preprocessor definition of NOMINMAX in ifndef conditional to suppress warning when cross compiling Windows.

  `fs.cpp:6:0: warning: "NOMINMAX" redefined`
  `/usr/lib/gcc/x86_64-w64-mingw32/7.3-posix/include/c++/x86_64-w64-mingw32/bits/os_defines.h:45:0: note: this is the location of the previous definition
   #define NOMINMAX 1`

  #define NOMINMAX was introduced in the following merge.

  #14426

ACKs for commit 0b3a65:
  practicalswift:
    utACK 0b3a654
  promag:
    utACK 0b3a654.

Tree-SHA512: 0175195b88e63d3d44ffac2b8cc87ae7b285a45ed4e49605bca0cc82db073006c22024ef9c2f287980d357dac1099f798f1eeaa0bd75bb7a625919dc1632366c
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 3, 2020
Summary:
369244f654 utils: Fix broken Windows filelock (Chun Kuan Lee)

Pull request description:

  Fix broken filelock on Windows, also add a test for this. It's a regression introduced by #13862.

Tree-SHA512: 15665b1930cf39ec71f3ab07def8e2897659f6fd4d2de749d63a5a8ec920e4a04282f12bc262f242b1b3d14d2dd9fa191ddbcf16a46fb927b5b2b14d9f6b5d01

Backport Core of [[bitcoin/bitcoin#14426 | PR14426]]

Test Plan: `ninja check check-functional`

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D6334
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 27, 2021
369244f utils: Fix broken Windows filelock (Chun Kuan Lee)

Pull request description:

  Fix broken filelock on Windows, also add a test for this. It's a regression introduced by bitcoin#13862.

Tree-SHA512: 15665b1930cf39ec71f3ab07def8e2897659f6fd4d2de749d63a5a8ec920e4a04282f12bc262f242b1b3d14d2dd9fa191ddbcf16a46fb927b5b2b14d9f6b5d01
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 27, 2021
0b3a654 Avoid redefine warning (Peter Bushnell)

Pull request description:

  Wrap preprocessor definition of NOMINMAX in ifndef conditional to suppress warning when cross compiling Windows.

  `fs.cpp:6:0: warning: "NOMINMAX" redefined`
  `/usr/lib/gcc/x86_64-w64-mingw32/7.3-posix/include/c++/x86_64-w64-mingw32/bits/os_defines.h:45:0: note: this is the location of the previous definition
   #define NOMINMAX 1`

  #define NOMINMAX was introduced in the following merge.

  bitcoin#14426

ACKs for commit 0b3a65:
  practicalswift:
    utACK 0b3a654
  promag:
    utACK 0b3a654.

Tree-SHA512: 0175195b88e63d3d44ffac2b8cc87ae7b285a45ed4e49605bca0cc82db073006c22024ef9c2f287980d357dac1099f798f1eeaa0bd75bb7a625919dc1632366c
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 28, 2021
369244f utils: Fix broken Windows filelock (Chun Kuan Lee)

Pull request description:

  Fix broken filelock on Windows, also add a test for this. It's a regression introduced by bitcoin#13862.

Tree-SHA512: 15665b1930cf39ec71f3ab07def8e2897659f6fd4d2de749d63a5a8ec920e4a04282f12bc262f242b1b3d14d2dd9fa191ddbcf16a46fb927b5b2b14d9f6b5d01
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 28, 2021
0b3a654 Avoid redefine warning (Peter Bushnell)

Pull request description:

  Wrap preprocessor definition of NOMINMAX in ifndef conditional to suppress warning when cross compiling Windows.

  `fs.cpp:6:0: warning: "NOMINMAX" redefined`
  `/usr/lib/gcc/x86_64-w64-mingw32/7.3-posix/include/c++/x86_64-w64-mingw32/bits/os_defines.h:45:0: note: this is the location of the previous definition
   #define NOMINMAX 1`

  #define NOMINMAX was introduced in the following merge.

  bitcoin#14426

ACKs for commit 0b3a65:
  practicalswift:
    utACK 0b3a654
  promag:
    utACK 0b3a654.

Tree-SHA512: 0175195b88e63d3d44ffac2b8cc87ae7b285a45ed4e49605bca0cc82db073006c22024ef9c2f287980d357dac1099f798f1eeaa0bd75bb7a625919dc1632366c
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 29, 2021
369244f utils: Fix broken Windows filelock (Chun Kuan Lee)

Pull request description:

  Fix broken filelock on Windows, also add a test for this. It's a regression introduced by bitcoin#13862.

Tree-SHA512: 15665b1930cf39ec71f3ab07def8e2897659f6fd4d2de749d63a5a8ec920e4a04282f12bc262f242b1b3d14d2dd9fa191ddbcf16a46fb927b5b2b14d9f6b5d01
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 29, 2021
0b3a654 Avoid redefine warning (Peter Bushnell)

Pull request description:

  Wrap preprocessor definition of NOMINMAX in ifndef conditional to suppress warning when cross compiling Windows.

  `fs.cpp:6:0: warning: "NOMINMAX" redefined`
  `/usr/lib/gcc/x86_64-w64-mingw32/7.3-posix/include/c++/x86_64-w64-mingw32/bits/os_defines.h:45:0: note: this is the location of the previous definition
   #define NOMINMAX 1`

  #define NOMINMAX was introduced in the following merge.

  bitcoin#14426

ACKs for commit 0b3a65:
  practicalswift:
    utACK 0b3a654
  promag:
    utACK 0b3a654.

Tree-SHA512: 0175195b88e63d3d44ffac2b8cc87ae7b285a45ed4e49605bca0cc82db073006c22024ef9c2f287980d357dac1099f798f1eeaa0bd75bb7a625919dc1632366c
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jul 1, 2021
369244f utils: Fix broken Windows filelock (Chun Kuan Lee)

Pull request description:

  Fix broken filelock on Windows, also add a test for this. It's a regression introduced by bitcoin#13862.

Tree-SHA512: 15665b1930cf39ec71f3ab07def8e2897659f6fd4d2de749d63a5a8ec920e4a04282f12bc262f242b1b3d14d2dd9fa191ddbcf16a46fb927b5b2b14d9f6b5d01
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jul 1, 2021
0b3a654 Avoid redefine warning (Peter Bushnell)

Pull request description:

  Wrap preprocessor definition of NOMINMAX in ifndef conditional to suppress warning when cross compiling Windows.

  `fs.cpp:6:0: warning: "NOMINMAX" redefined`
  `/usr/lib/gcc/x86_64-w64-mingw32/7.3-posix/include/c++/x86_64-w64-mingw32/bits/os_defines.h:45:0: note: this is the location of the previous definition
   #define NOMINMAX 1`

  #define NOMINMAX was introduced in the following merge.

  bitcoin#14426

ACKs for commit 0b3a65:
  practicalswift:
    utACK 0b3a654
  promag:
    utACK 0b3a654.

Tree-SHA512: 0175195b88e63d3d44ffac2b8cc87ae7b285a45ed4e49605bca0cc82db073006c22024ef9c2f287980d357dac1099f798f1eeaa0bd75bb7a625919dc1632366c
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jul 1, 2021
369244f utils: Fix broken Windows filelock (Chun Kuan Lee)

Pull request description:

  Fix broken filelock on Windows, also add a test for this. It's a regression introduced by bitcoin#13862.

Tree-SHA512: 15665b1930cf39ec71f3ab07def8e2897659f6fd4d2de749d63a5a8ec920e4a04282f12bc262f242b1b3d14d2dd9fa191ddbcf16a46fb927b5b2b14d9f6b5d01
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jul 1, 2021
0b3a654 Avoid redefine warning (Peter Bushnell)

Pull request description:

  Wrap preprocessor definition of NOMINMAX in ifndef conditional to suppress warning when cross compiling Windows.

  `fs.cpp:6:0: warning: "NOMINMAX" redefined`
  `/usr/lib/gcc/x86_64-w64-mingw32/7.3-posix/include/c++/x86_64-w64-mingw32/bits/os_defines.h:45:0: note: this is the location of the previous definition
   #define NOMINMAX 1`

  #define NOMINMAX was introduced in the following merge.

  bitcoin#14426

ACKs for commit 0b3a65:
  practicalswift:
    utACK 0b3a654
  promag:
    utACK 0b3a654.

Tree-SHA512: 0175195b88e63d3d44ffac2b8cc87ae7b285a45ed4e49605bca0cc82db073006c22024ef9c2f287980d357dac1099f798f1eeaa0bd75bb7a625919dc1632366c
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jul 1, 2021
369244f utils: Fix broken Windows filelock (Chun Kuan Lee)

Pull request description:

  Fix broken filelock on Windows, also add a test for this. It's a regression introduced by bitcoin#13862.

Tree-SHA512: 15665b1930cf39ec71f3ab07def8e2897659f6fd4d2de749d63a5a8ec920e4a04282f12bc262f242b1b3d14d2dd9fa191ddbcf16a46fb927b5b2b14d9f6b5d01
random-zebra added a commit to PIVX-Project/PIVX that referenced this issue Aug 5, 2021
63e0be6 [Remove] By-pass logprint-scanner restriction. (furszy)
280ced3 utils: Fix broken Windows filelock (Chun Kuan Lee)
be89860 utils: Convert Windows args to utf-8 string (Chun Kuan Lee)
e8cfa6e Call unicode API on Windows (Chun Kuan Lee)
1a02a8a tests: Add test case for std::ios_base::ate (Chun Kuan Lee)
2e57cd4 Move boost/std fstream to fsbridge (furszy)
9d8bcd4 utils: Add fsbridge fstream function wrapper (Chun Kuan Lee)
d59d48d utils: Convert fs error messages from multibyte to utf-8 (ken2812221)
9ef58cc Logging: use "fmterr" variable name for errors instead of general "e" that can be used by any other function. (furszy)
dd94241 utils: Use _wfopen and _wreopen on Windows (Chun Kuan Lee)
3993641 add unicode compatible file_lock for Windows (Chun Kuan Lee)
48349f8 Provide relevant error message if datadir is not writable. (murrayn)

Pull request description:

  As the software is currently using the ANSI encoding on Windows, the user's language settings could affect the proper functioning of the node/wallet, to the point of not be able to open some non-ASCII name files and directories.

  This solves the Windows encoding issues, completing the entire bitcoin#13869 work path (and some other required backports). Enabling for example users that use non-english characters in directories and file names to be accepted.

  Backported PRs:
  * bitcoin#12422.
  * bitcoin#12630.
  * bitcoin#13862.
  * bitcoin#13866.
  * bitcoin#13877.
  * bitcoin#13878.
  * bitcoin#13883.
  * bitcoin#13884.
  * bitcoin#13886.
  * bitcoin#13888.
  * bitcoin#14192.
  * bitcoin#13734.
  * bitcoin#14426.

  This is built on top of other two PRs that i have open #2423 and #2369.
  Solves old issues #940 and #2163.

  TODO:
  * Backport `assert_start_raises_init_error` and `ErrorMatch` in TestNode` (bitcoin#12718)

ACKs for top commit:
  Fuzzbawls:
    ACK 63e0be6
  random-zebra:
    ACK 63e0be6 and merging...

Tree-SHA512: cb1f7c23abb5b7b3af50bba18652cc2cad93fd7c2fca9c16ffd3fee34c4c152a3b666dfa87fe6b44c430064dcdee4367144dcb4a41203c91b0173b805bdb3d7d
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet