Skip to content
This repository has been archived by the owner on Jul 1, 2023. It is now read-only.

Error reading bolt data input file: line 1, column 43: expected single char for mispred bit #28

Closed
J-cztery opened this issue Jul 6, 2018 · 13 comments

Comments

@J-cztery
Copy link

J-cztery commented Jul 6, 2018

Intel compiler + gcc 4.8.5.
Now that #23 has been fixed, i see this:

PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: Spawning perf-script job to read branch events
PERF2BOLT: Spawning perf-script job to read mem events
PERF2BOLT: Spawning perf-script job to read tasks
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: binary build-id is:     270c5a1e626f1b4f158599ab044340721e063450
PERF2BOLT: matched build-id and file name
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0x800000, offset 0x400000
BOLT-INFO: enabling relocation mode
BOLT-WARNING: sizes differ for function __intel_memcpy. FDE : 6055; symbol table : 6064. Using max size.
BOLT-WARNING: sizes differ for function __intel_new_memcpy. FDE : 6055; symbol table : 6064. Using max size.
BOLT-INFO: forcing -jump-tables=move as PIC jump table was detected in function _ZL28read_encoded_value_with_basehmPKhPm/eh_personality.o/1(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x6c8 (address 0x51d0b8) in function __intel_avx_rep_memcpy
BOLT-WARNING: unable to disassemble instruction at offset 0x165 (address 0x51e315) in function __intel_new_memcpy(*2)
PERF2BOLT: Waiting for perf tasks collection to finish...
PERF2BOLT: Parsing perf-script tasks output
PERF2BOLT: Input binary is associated with 1 PID(s)
PERF2BOLT: Waiting for perf events collection to finish...
PERF2BOLT: Aggregating branch events...
Error reading bolt data input file: line 1, column 43: expected single char for mispred bit
Found: 0x7f2c9cf100fe
PERF2BOLT: Failed to parse samples
PERF2BOLT: Wrote 0 objects and 0 memory objects to perf.fdata

Is my perf.data broken?

@maksfb
Copy link
Contributor

maksfb commented Jul 6, 2018

It could be the new format of perf output. What is the perf version that your are running and on what hardware? Ideally I'd like to see the output of perf script -F pid,brstack ..., or at least at a part of it that was causing the trouble.

Also, our diagnostics was broken. Could you please pull the latest changes and re-run the command?

@J-cztery
Copy link
Author

J-cztery commented Jul 6, 2018

./perf --version
perf version 3.10.0-693.5.2.el7.x86_64.debug
The hardware is Intel Xeon Phi 7250. The hardware i am running perf2bolt, though is Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz, same kernel and OS version, Centos 7.

Will respond your other questions later.
Thanks.

@J-cztery
Copy link
Author

J-cztery commented Jul 6, 2018

With the fixed diagnostics it looks like this:

Error reading bolt data input file: line 1, column 43: expected single char for mispred bit
Found: -
PERF2BOLT: Failed to parse samples

@J-cztery
Copy link
Author

J-cztery commented Jul 7, 2018

sudo perf script -f -F pid,brstack
First few lines of output:

 7466  0xffffffff816ac32c/0x7f2c9cf100fe/-/-/-/0 
 7466  0xffffffff816ac32c/0x7f2c9cf100fe/-/-/-/0  0xffffffff816ac32c/0x7f2c9cf100fe/-/-/-/0 
 7466  0xffffffff816ac32c/0x7f2c9cf100fe/-/-/-/0  0xffffffff816ac32c/0x7f2c9cf100fe/-/-/-/0  0xffffffff816ac32c/0x7f2c9cf100fe/-/-/-/0 
 7466  0xffffffff816ac32c/0x7f2c9cf100fe/-/-/-/0  0xffffffff816ac32c/0x7f2c9cf100fe/-/-/-/0  0xffffffff816ac32c/0x7f2c9cf100fe/-/-/-/0  0xffffffff816ac32c/0x7f2c9cf100fe/-/-/-/0

@maksfb
Copy link
Contributor

maksfb commented Jul 9, 2018

For the perf issue it's an easy fix, assuming that otherwise the output format is the same. If you can upgrade to perf version 4.5 or later it might be easier to guarantee that. The more serious issue I see is with __intel_avx_rep_memcpy and __intel_new_memcpy functions referenced in the log. We'll need to add a support for jump tables / data in code, otherwise we cannot process them in relocation mode. The workaround is to use -relocs=0, and it is not good for performance.

@J-cztery
Copy link
Author

J-cztery commented Jul 10, 2018

Grabbed perf from kernel 4.5:
by git checkout v4.5 and compiled it.
$ perf --version
perf version 4.5.gb562e4

Collected traces :
./perf record -e cycles:u -j any,u -o perf.data -- (...)

Then on a different machine after copying the perf.data:
sudo perf script -F pid,brstack -f

172359  0xffffffff816ac32c/0x7fc9b763c0fe/-/-/-/0 
172359  0xffffffff816ac32c/0x7fc9b763c0fe/-/-/-/0  0xffffffff816ac32c/0x7fc9b763c0fe/-/-/-/0 
172359  0xffffffff816ac32c/0x7fc9b763c0fe/-/-/-/0  0xffffffff816ac32c/0x7fc9b763c0fe/-/-/-/0  0xffffffff816ac32c/0x7fc9b763c0fe/-/-/-/0 

I have also tried perf version 4.14.rc5.g7c584a with the same results.

@J-cztery
Copy link
Author

I guess the problem here is not in perf version but possibly perf on KNL not supporting the correct hw event.
From:
https://gitlab.imag.fr/kaunetem/linux-kaunetem/commit/dc323ce8e72d6d1beb9af9bbd29c4d55ce3d7fb0
M/P/-: M=branch target mispredicted or branch direction was mispredicted, P=target predicted or direction predicted, -=not supported

I will have a look how perf gets it.

@J-cztery
Copy link
Author

Yep this one is a KNL issue. On other CPU i got:
0xffffffff816ac32c/0x7f7b322365a0/P/-/-/0

@maksfb
Copy link
Contributor

maksfb commented Jul 10, 2018

Interesting. We only use misprediction bit for ICP optimization, so it should be safe to ignore it. I'll add a patch. Thank you for the investigation.

facebook-github-bot pushed a commit that referenced this issue Jul 10, 2018
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

Reviewed By: rafaelauler

Differential Revision: D8786827

fbshipit-source-id: 0a4ad83
@J-cztery
Copy link
Author

Great, thanks! So how about the intel memcpy (and others) replacements?
Is there anything i can do to help? Should a new issue be raised?

@maksfb
Copy link
Contributor

maksfb commented Jul 12, 2018

Is there any way to force intel compiler to use libc functions? It justifies the new issue in any case.

@J-cztery
Copy link
Author

Looks good, closing:

PERF2BOLT: Waiting for perf tasks collection to finish...
PERF2BOLT: Parsing perf-script tasks output
PERF2BOLT: Input binary is associated with 1 PID(s)
PERF2BOLT: Waiting for perf events collection to finish...
PERF2BOLT: Aggregating branch events...
PERF2BOLT-WARNING: misprediction bit is missing in profile

@J-cztery
Copy link
Author

Is there any way to force intel compiler to use libc functions?

-ffreestanding seems to cut it.
https://software.intel.com/en-us/forums/intel-c-compiler/topic/306063

rafaelauler pushed a commit that referenced this issue May 10, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue May 11, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
aaupov pushed a commit that referenced this issue May 12, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue May 25, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue May 25, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue May 27, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Jun 4, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Jun 18, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
aaupov pushed a commit that referenced this issue Jun 25, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Jul 6, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Jul 16, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Jul 21, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Jul 23, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Aug 5, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Aug 6, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Aug 17, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Aug 25, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Sep 2, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Sep 9, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from commit 0a4ad831e49388ccc49ae15ecdb51c58b6eaafff)
rafaelauler pushed a commit that referenced this issue Dec 8, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from FBD8786827)
rafaelauler pushed a commit that referenced this issue Dec 10, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from FBD8786827)
rafaelauler pushed a commit that referenced this issue Dec 13, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from FBD8786827)
aaupov pushed a commit that referenced this issue Dec 24, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from FBD8786827)
aaupov pushed a commit that referenced this issue Dec 24, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from FBD8786827)
aaupov pushed a commit that referenced this issue Dec 30, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from FBD8786827)
aaupov pushed a commit that referenced this issue Dec 31, 2021
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from FBD8786827)
maksfb added a commit that referenced this issue Jan 10, 2022
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from FBD8786827)
maksfb added a commit that referenced this issue Jan 11, 2022
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from FBD8786827)
maksfb added a commit that referenced this issue Jan 11, 2022
Summary:
As reported in GH-28 `perf` can produce `-` symbol for misprediction bit
if the bit is not supported by the kernel/HW. In this case we can ignore
the bit.

(cherry picked from FBD8786827)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants