Make backtraces work on Windows GNU targets again. #39234

segevfiner · 2017-01-21T23:01:57Z

This is done by adding a function that can return a filename
to pass to backtrace_create_state. The filename is obtained in
a safe way by first getting the filename, locking the file so it can't
be moved, and then getting the filename again and making sure it's the same.

See: #37359 (comment)
Issue: #33985

Note though that this isn't that pretty...

I had to implement a WideCharToMultiByte wrapper function to convert to the ANSI code page. This will work better than only allowing ASCII provided that the ANSI code page is set to the user's local language, which is often the case.

Also, please make sure that I didn't break the Unix build.

rust-highfive · 2017-01-21T23:02:10Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @brson (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

petrochenkov · 2017-01-22T08:08:27Z

@segevfiner
Awesome!
I'll review this today.

I had to implement a WideCharToMultiByte wrapper function to convert to the ANSI code page.

👍
It's better to check if the string survives the WideCharToMultiByte -> MultiByteToWideChar cycle rather than work with ASCII only as #37359 (comment) describes.

petrochenkov · 2017-01-22T15:12:39Z

src/libstd/sys_common/gnu/libbacktrace.rs

+
+        let mut filename = ptr::null();
+        let mut file = None;
+        if let Ok(value) = ::sys::backtrace::get_executable_filename() {


Nit: Ok(value) -> Ok((filename, file))

Nit 2: you can do let (filename, file) = match ::sys::backtrace::get_executable_filename() { .... }

petrochenkov · 2017-01-22T15:15:30Z

src/libstd/sys_common/gnu/libbacktrace.rs

+        let mut filename = ptr::null();
+        let mut file = None;
+        if let Ok(value) = ::sys::backtrace::get_executable_filename() {
+            FILENAME_BUF = &value.0;


static FILENAME_BUF 1) contains a dangling reference to stack-allocated Vec<*i8> and 2) is not necessary

petrochenkov · 2017-01-22T15:23:51Z

src/libstd/sys_common/gnu/libbacktrace.rs

@@ -135,7 +151,7 @@ pub fn print(w: &mut Write, idx: isize, addr: *mut libc::c_void,

    // backtrace errors are currently swept under the rug, only I/O
    // errors are reported
-    let state = unsafe { init_state() };
+    let (state, _file) = unsafe { init_state() };


_file needs to be unlocked (dropped) only if we got success from backtrace_syminfo.
Otherwise libbacktrace will try to read it again in next backtrace requests in unlocked state.
On failure the file needs to be either leaked or stored to be unlocked later.

petrochenkov · 2017-01-22T15:31:03Z

src/libstd/sys/windows/mod.rs

+
+        buf.set_len(size as usize);
+
+        Ok(buf)


You need to validate that the reverse conversion with MultiByteToWideChar gives the original u16 string, otherwise the conversion was lossy and libbacktrace will try to open some other file.

I used the lpUsedDefaultChar parameter of WideCharToMultiByte, it can save the need to convert twice assuming that it's not broken in some way.

Actually there's still ways in which data loss can occur other than default char replacement. You'll want to pass WC_NO_BEST_FIT_CHARS.

For strings that require validation, such as file, resource, and user names, the application should always use the WC_NO_BEST_FIT_CHARS flag. This flag prevents the function from mapping characters to characters that appear similar but have very different semantics. In some cases, the semantic change can be extreme. For example, the symbol for "∞" (infinity) maps to 8 (eight) in some code pages.

Ideally the set of flags should mirror whatever is used inside of (most probably) CreateFileA which is used inside posix open in CRT which is used in libbacktrace.

The function which is actually used to convert strings inside A functions is RtlAnsiStringToUnicodeString which in turn calls RtlMultiByteToUnicodeN, not MultiByteToWideChar.

Also another point is that we're going in the opposite direction here, so we can't match the flags anyway.

Ok, if we do want to "fix" this, then this will work 😄

let image_name1 = query_full_process_image_name()?; let ansi_image_name1 = WideCharToMultiByte(image_name1, TRY_TO_GUESS_FLAGS); let file = CreateFileA(ansi_image_name1)?; // instead of OpenOptions::new().blah().open(&image_name1) let image_name2 = query_full_process_image_name()?; if image_name1 != image_name2 { // Simultaneously make sure that the file is not moved and that encoding conversion was correct return Err } return ansi_image_name1;

MSDN documents that WC_NO_BEST_FIT_CHARS should actually make the conversion reversible, which is what you wanted (assuming no surprises). So I added it as suggested.

petrochenkov · 2017-01-22T15:41:06Z

src/libstd/sys/windows/backtrace.rs

+
+fn query_full_process_image_name() -> io::Result<PathBuf> {
+    super::fill_utf16_buf(|buf, mut sz| unsafe {
+        c::QueryFullProcessImageNameW(c::GetCurrentProcess(), 0, buf, &mut sz); sz


Could you use OpenProcess(PROCESS_QUERY_INFORMATION, FALSE, GetCurrentProcessId()) instead of c::GetCurrentProcess() like the #37359 does?
QueryFullProcessImageNameW didn't work with pseudo-handle -1 returned by GetCurrentProcess the last time I tried.

It looks like there's no need to convert this Vec<u16> vector into PathBuf and back, you can just use Vec<u16> through all the process.

Probably a bug they fixed in Windows 8/10 or something. Windows... 😛

I converted it to a PathBuf for the call to OpenOptions::open below, in a way it's a function that queries for a path so it makes sense that it returns a Path[Buf].

petrochenkov · 2017-01-22T20:49:21Z

src/libstd/sys_common/gnu/libbacktrace.rs

+            Ok((filename, file)) => {
+                // filename is purposely leaked here since libbacktrace requires
+                // it to stay allocated permanently
+                ((*Box::into_raw(Box::new(filename))).as_ptr(), Some(file))


let filename_ptr = filename.as_ptr(); mem::forget(filename); (filename_ptr, Some(file))

is probably the simplest solution, but this is ok too.

petrochenkov · 2017-01-22T20:51:34Z

Now I think I'm satisfied with this PR.
r? @alexcrichton for the final decision

segevfiner · 2017-01-22T23:00:09Z

r? @alexcrichton

alexcrichton · 2017-01-23T18:05:12Z

src/libstd/sys/windows/backtrace.rs

+                                                        c::FALSE,
+                                                        c::GetCurrentProcessId()));
+        super::fill_utf16_buf(|buf, mut sz| {
+            c::QueryFullProcessImageNameW(process_handle.raw(), 0, buf, &mut sz); sz


I think we should check for errors here, right?

Reading fill_utf16_buf carefully and seeing that QueryFullProcessImageNameW will leave sz as is on error. You are right.

alexcrichton · 2017-01-23T18:22:08Z

src/libstd/sys/windows/backtrace.rs

+pub fn get_executable_filename() -> io::Result<(Vec<i8>, File)> {
+    let (executable, file) = lock_and_get_executable_filename()?;
+    let u16_executable = super::to_u16s(executable.into_os_string())?;
+    Ok((super::wide_char_to_multi_byte(c::CP_ACP, c::WC_NO_BEST_FIT_CHARS,


Out of curiosity, how much does this buy us over just attempting utf-8 and returning an error if not?

We don't use these standard functions anywhere else in the standard library, so I'm curious if we'd get much benefit from trying to do this here now.

Because your UTF-8 bytes might be valid in some other encoding, but have different meaning, which would result in accessing the wrong file, which could potentially leave a vector open for attack.

Well...

UTF-8 will just break when passed to open/backtrace_create_state on any non ASCII character while the WideCharToMultiByte(CP_ACP, ...) will work on any character that is supported in the current ANSI code page.

The ANSI code page is a setting that you can set in Windows which is used by any non Unicode aware (wide char) program, as such it's often set to a code page for the user's local language allowing such programs to work with it. Sadly Windows doesn't use UTF-8 like Linux and keen.

As such using the ANSI code page is very likely to keep this working despite non ASCII characters in the path.

We don't use these standard functions anywhere else in the standard library

WeThey use them in libbacktrace 🙂
And we have to do the same.

alexcrichton · 2017-01-23T18:23:19Z

src/libstd/sys_common/gnu/libbacktrace.rs

                                       ptr::null_mut());
-        STATE
+        (STATE, file)


How come file is returned here? We're going to unconditionally reuse STATE so long as it's non-null, so it seems like so long as we do that we should keep the file open?

We only need to keep the file open/locked during the first call to a libbacktrace function that operates on the state unless that function fails and as such it will try the initialization again on the next call.

What is currently done is to optionally return the file on the first creation of the state. And if the first call to libbacktrace_syminfo fails we leak it so it stays open, otherwise it will be dropped on exit from print.

The options I can think of:

Refactor the code so it's clearer, I'm open to any suggestions here.

Leak it always. Just mem::forget immediately.

Return it and keep it in a static mut (Annoying since it's a drop type so we need to play around with Box and raw pointers), and drop it explicitly once backtrace_syminfo succeeds the first time.

The same as 3 but place it in a static mut immediately in init_state.

Ah ok, thanks for the explanation! I think it's probably easiest/best to just leak it here rather than special casing the result later on (depending on internal impl details of libbacktrace). We can probably use .into_raw_handle() here as well instead of mem::forget

Actually this is sys_common. Which means I can't use .into_raw_handle() since it only exists on Windows (It's .into_raw_fd() in Unix/Linux) and cfg!/#[cfg] is not allowed in sys_common (Enforced by tidy). 😕

alexcrichton · 2017-01-24T20:44:07Z

Looks good to me, thanks @segevfiner! Want to squash the commits and I'll r+?

This is done by adding a function that can return a filename to pass to backtrace_create_state. The filename is obtained in a safe way by first getting the filename, locking the file so it can't be moved, and then getting the filename again and making sure it's the same. See: rust-lang#37359 (comment) Issue: rust-lang#33985

segevfiner · 2017-01-24T21:13:31Z

Squashed! Thanks for looking at this too! Now I guess it's time to cross fingers for CI. 😛

alexcrichton · 2017-01-25T18:01:47Z

@bors: r+

bors · 2017-01-25T18:01:48Z

📌 Commit 4186037 has been approved by alexcrichton

bors · 2017-01-27T04:50:10Z

⌛ Testing commit 4186037 with merge c588767...

bors · 2017-01-27T04:57:09Z

💔 Test failed - status-travis

petrochenkov · 2017-01-27T09:06:20Z

💔 Test failed - status-travis

get_executable_filename is unused on some platforms

segevfiner · 2017-01-27T09:08:34Z

@petrochenkov Yeah, I noticed. Working on it.

segevfiner · 2017-01-27T09:33:47Z

I tried to eliminate the dead code lints: (New commit so you can see what I did, instead of immediately squashing):

Moved get_executable_filename into sys::backgrace::gnu so I can cfg it out when it's not needed same as done for sys_common::gnu.
I had to split the functions I added in sys/windows/backtrace.rs to sys/windows/backtrace_gnu.rs, which I hope I did correctly. (Windows has the MSVC configuration which doesn't use libbbacktrace).
I left multi_byte_to_wide_char as #[allow(dead_code] since it would be weird to cfg it out on target_env="gnu".
I also had to add a gnu mod to c.rs in sys/windows so that the functions used only by sys::backgrace::gnu won't trigger dead code lints.

I tested x86_64-pc-windows-gnu and x86_64-pc-windows-msvc. My computer is slow and lacking disk space to test Linux and some non-GNU Unix platform right now. Hopefully the fix there is trivial enough that I got it right this time 😞

alexcrichton · 2017-01-27T17:18:17Z

@bors: r+

Thanks!

segevfiner · 2017-01-27T23:07:52Z

Added a commit which fixes this. I can squash to one commit if you want.

P.S. C also allows for even further evil. It requires sizeof(char) == 1 but CHAR_BIT which is the number of bits in char doesn't have to be 8... Good luck getting anything to work on that 😝

alexcrichton · 2017-01-28T00:22:13Z

@bors: r+

bors · 2017-01-28T00:22:14Z

📌 Commit 1b4a6c8 has been approved by alexcrichton

bors · 2017-01-28T00:41:25Z

⌛ Testing commit 1b4a6c8 with merge d7c78aa...

alexcrichton · 2017-01-28T00:43:33Z

@bors: retry

prioritizing rollup

alexcrichton · 2017-01-28T02:50:25Z

@bors: r-

I think this may have been the cause of these failures in the rollup

segevfiner · 2017-01-28T07:34:36Z

@alexcrichton It does look like backtraces are screwed up somehow in i686-pc-windows-gnu... 😖

The symbols are wrong or the stack walk isn't working, it might be a bug elsewhere that wasn't detected until now since backtraces were broken for quite some time and no one tried gdb in this target.

I will try to compile and check...

(So much platform specific fun with this pull request 😢 )

segevfiner · 2017-01-28T08:33:41Z

And the results are...

Build failure: #39357

Obviously I can't try to debug when I can't even build...

segevfiner · 2017-01-28T13:31:10Z

I managed to build using the gcc used in the AppVeyor build and even something as simple as this:

fn f() {
    panic!("Boom!");
}

fn main() {
    f();
}

Doesn't display a proper stack trace in this build. The tests show weird frames with only the function drop while this minimal example seems to punt once it tries to climb one frame back.

petrochenkov · 2017-01-28T13:40:19Z

@segevfiner
When backtraces worked on Windows/GNU they still were relatively buggy and could show garbage (What I see in https://ci.appveyor.com/project/rust-lang/rust/build/1.0.1734/job/h8b3h52gg89wp40o looks very similar to that garbage).
Do run-pass/backtrace.rs and run-pass/backtrace-debuginfo.rs pass on x86_64-pc-windows-gnu?
If yes, then this PR is an improvement and can be merged with these tests disabled on i686.

petrochenkov · 2017-01-28T13:41:09Z

drop::h82dbe416e9337a7a instead of <unknown> means that this PR does its job.

segevfiner · 2017-01-28T14:28:41Z

@petrochenkov I should have looked first... #28218 😆

I think I managed to fix this by simply adding base.eliminate_frame_pointer = false; to "src\librustc_back\target\i686_pc_windows_gnu.rs".

But this completely disables this optimization. MSVC disables frame pointer optimizations by default nowadays, "/Oy-" is added by some arcane msbuild stuff. So I wonder if this is not something that is acceptable to disable so that backtraces work?

So I guess we need a decision:

Disable the tests and hope for someone to fix this in some way in the future. Perhaps in a different way.
Disable frame pointer optimizations, pay the performance price (I don't know how big it is if any) and have backtraces working in i686-pc-windows-gnu.

petrochenkov · 2017-01-28T14:45:15Z

@segevfiner
I think this is a separate issue that should be discussed and fixed in a separate PR and shouldn't block this PR.

petrochenkov · 2017-01-28T14:51:28Z

(I did some measurements with enabled/disabled frame pointers on couple of projects with MSVC+PDB toolchain, but it was years ago. There were small performance differences, but I still never disabled frame pointers even in release builds because it resulted in absolutely horrible debugging experience.
Things could change. Or GNU+DWARF may behave differently. I don't know.)

retep998 · 2017-01-28T15:53:49Z

I think we should disable frame pointer optimization on i686-pc-windows-gnu specifically. The problem with i686 on Windows is that, unlike all other architectures which use table based exceptions, it use stack based exception chaining, which means it kinda depends heavily on the frame pointer to be able to walk the stack.

petrochenkov · 2017-01-28T19:58:21Z

@bors r+

bors · 2017-01-28T19:58:22Z

📌 Commit ab21314 has been approved by petrochenkov

segevfiner · 2017-01-28T20:02:28Z

The follow up commit which disables FPO and restores the test is at segevfiner@befe380 which I will rebase and open a PR for, once the CI will stop finding surprises, so that this can be discussed and tested properly.

bors · 2017-01-28T20:33:00Z

⌛ Testing commit ab21314 with merge 1491e04...

…trochenkov Make backtraces work on Windows GNU targets again. This is done by adding a function that can return a filename to pass to backtrace_create_state. The filename is obtained in a safe way by first getting the filename, locking the file so it can't be moved, and then getting the filename again and making sure it's the same. See: #37359 (comment) Issue: #33985 Note though that this isn't that pretty... I had to implement a `WideCharToMultiByte` wrapper function to convert to the ANSI code page. This will work better than only allowing ASCII provided that the ANSI code page is set to the user's local language, which is often the case. Also, please make sure that I didn't break the Unix build.

bors · 2017-01-28T23:14:18Z

☀️ Test successful - status-appveyor, status-travis
Approved by: petrochenkov
Pushing 1491e04 to master...

…nu, r=alexcrichton Fix backtraces on i686-pc-windows-gnu by disabling FPO This might have performance implications. But do note that MSVC disables FPO by default nowadays and it's use is limited in exception heavy languages like C++. See discussion in: #39234 Closes: #28218

rust-highfive assigned brson Jan 21, 2017

segevfiner force-pushed the fix-backtraces-on-windows-gnu branch 3 times, most recently from ba38be8 to b5dfe50 Compare January 21, 2017 23:37

petrochenkov requested changes Jan 22, 2017

View reviewed changes

petrochenkov reviewed Jan 22, 2017

View reviewed changes

petrochenkov approved these changes Jan 22, 2017

View reviewed changes

rust-highfive assigned alexcrichton and unassigned brson Jan 22, 2017

alexcrichton reviewed Jan 23, 2017

View reviewed changes

segevfiner force-pushed the fix-backtraces-on-windows-gnu branch from ef95321 to 4186037 Compare January 24, 2017 21:10

Attempt at fixing dead code lints

450554e

Disable backtrace tests on i686-pc-windows-gnu since it's broken by FPO

ab21314

bors merged commit ab21314 into rust-lang:master Jan 28, 2017

bors mentioned this pull request Jan 28, 2017

Improve backtrace formating while panicking. #38165

Merged

petrochenkov mentioned this pull request Jan 28, 2017

Rust stack backtrace filled with unknown on Windows 8.1 #33985

Closed

segevfiner mentioned this pull request Jan 29, 2017

Fix backtraces on i686-pc-windows-gnu by disabling FPO #39379

Merged

segevfiner deleted the fix-backtraces-on-windows-gnu branch January 29, 2017 18:06

petrochenkov mentioned this pull request Oct 6, 2020

cleanup: Revert all hacks added in #39234 to avoid feeding untrusted data to libbacktrace #77622

Closed

Make backtraces work on Windows GNU targets again. #39234

Make backtraces work on Windows GNU targets again. #39234

Conversation

segevfiner commented Jan 21, 2017

rust-highfive commented Jan 21, 2017

petrochenkov commented Jan 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

retep998 Jan 22, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrochenkov Jan 22, 2017 • edited

Choose a reason for hiding this comment

segevfiner Jan 22, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrochenkov commented Jan 22, 2017

segevfiner commented Jan 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Jan 24, 2017

segevfiner commented Jan 24, 2017

alexcrichton commented Jan 25, 2017

bors commented Jan 25, 2017

bors commented Jan 27, 2017

bors commented Jan 27, 2017

petrochenkov commented Jan 27, 2017

segevfiner commented Jan 27, 2017

segevfiner commented Jan 27, 2017 • edited

alexcrichton commented Jan 27, 2017

segevfiner commented Jan 27, 2017

alexcrichton commented Jan 28, 2017

bors commented Jan 28, 2017

bors commented Jan 28, 2017

alexcrichton commented Jan 28, 2017

alexcrichton commented Jan 28, 2017

segevfiner commented Jan 28, 2017 • edited

segevfiner commented Jan 28, 2017 • edited

segevfiner commented Jan 28, 2017

petrochenkov commented Jan 28, 2017

petrochenkov commented Jan 28, 2017

segevfiner commented Jan 28, 2017 • edited

petrochenkov commented Jan 28, 2017

petrochenkov commented Jan 28, 2017 • edited

retep998 commented Jan 28, 2017

petrochenkov commented Jan 28, 2017

bors commented Jan 28, 2017

segevfiner commented Jan 28, 2017 • edited

bors commented Jan 28, 2017

bors commented Jan 28, 2017

retep998 Jan 22, 2017 •

edited

petrochenkov Jan 22, 2017 •

edited

segevfiner Jan 22, 2017 •

edited

segevfiner commented Jan 27, 2017 •

edited

segevfiner commented Jan 28, 2017 •

edited

segevfiner commented Jan 28, 2017 •

edited

segevfiner commented Jan 28, 2017 •

edited

petrochenkov commented Jan 28, 2017 •

edited

segevfiner commented Jan 28, 2017 •

edited