Add tools to help measure and compare SBE image size

- Break SBE image size contributions down by module
- Measure code, data and stack usage
- Compare sizes before and after a change to measure change impact
- Compare object section sizes, stack usage, disassembly, map file

Change-Id: Id873ba86cff7b8ecffdd37c028a57091535ea06d
Reviewed-on: http://rchgit01.rchland.ibm.com/gerrit1/69407
Tested-by: Jenkins Server <pfd-jenkins+hostboot@us.ibm.com>
Tested-by: FSP CI Jenkins <fsp-CI-jenkins+hostboot@us.ibm.com>
Reviewed-by: RAJA DAS <rajadas2@in.ibm.com>

src/build/img_defs.mk

@@ -467,6 +467,7 @@ PIPE-CFLAGS = -pipe
 GCC-CFLAGS += -mcpu=ppe42
 GCC-CFLAGS += -ffunction-sections
 GCC-CFLAGS += -fdata-sections
+GCC-CFLAGS += -fstack-usage
 endif
 
 ifeq ($(img), seeprom)

src/tools/utils/sbe-size/README

@@ -0,0 +1,52 @@
+These tools aim to answer two questions:
+  1. How do the individual modules of SBE contribute to code/data size?
+  2. How does my code change affect the size across modules?
+  3. Which symbols ended up increasing/decreasing in size?
+
+They do so by collecting code/data size per object file and allowing
+you to plot them in a stacked bar graph, or to compare two such statistics
+in a similar graph.
+
+The graphs use the object file name on the X axis, sorted alphabetically
+so that each object file is at the same X location every time you generate
+the graph. This should help compare similar graphs.
+
+The tools are currently special-cased to look at a DD2 build only. This can
+easily be changed by modifying collect-section-sizes.
+
+Usage:
+  0. Add the scripts to your PATH:
+     export PATH=$PATH:$PWD/tools/utils/sbe-size
+  1. Fully build your reference commit once
+  2. mkdir ref; cp -av images obj ref
+     Copy the reference build artifacts somewhere safe
+  3. SBEROOT=$PWD/ref collect-section-sizes > ref/sizes.txt
+     This will collect the code/data sizes for all reference object files
+     into a file 'sizes.txt'. It's an ASCII file, happy viewing.
+  4. Hack away at the code, rebuild.
+     If you changed common header files, do a "make clean" before you build
+     since the build system doesn't reliably rebuild all dependent files.
+  5. collect-section-sizes > sizes.txt
+     This will collect the code/data sizes for your changed build.
+  6. view-section-sizes sizes.txt
+     This will plot the sizes from sizes.txt using gnuplot. If your gnuplot
+     supports zooming, you can zoom around the plot to examine it. When
+     you're done, exit gnuplot by hitting Ctrl-D at the 'gnuplot>' prompt.
+  7. compare-section-sizes ref/sizes.txt sizes.txt
+     Compare two sizes files against one another. The plot will show size
+     reductions as bars below the zero line, size increases as bars above.
+  8. compare-section-breakdown images/sbe_seeprom_DD2.map
+     Compare maps for ref/images/sbe_seeprom_DD2.map and
+     images/sbe_seeprom_DD2.map, replacing addresses by a constant value
+     so only the sizes are compared.
+  9. compare-section-breakdown obj/power/fapi2/target.o
+     Compare section sizes for ref/obj/power/fapi2/target.o and
+     obj/power/fapi2/target.o - or any other object file of course.
+     This will filter the C++ mangled names too.
+ 10. compare-section-breakdown obj/power/fapi2/target.su
+     compare-section-breakdown obj/power/fapi2/target.dis
+     compare-section-breakdown obj/power/fapi2/target.diss
+     The same for stack usage, disassembly (with addresses normalized for
+     better diffing) and assembly with source.
+
+If you have questions, ask fenkes@de.ibm.com.

src/tools/utils/sbe-size/absolute.gnuplot

@@ -0,0 +1,6 @@
+set xtics rotate
+set xtics noenhanced
+set style histogram rowstacked
+set style data histogram
+set style fill solid border -1
+plot fn using 2:xtic(1) ti "code" lc rgb "#0000ff", '' u 3 ti "data" lc rgb "#ffff00", '' u 4 ti "stack" lc rgb "#ff0000"

src/tools/utils/sbe-size/collect-section-sizes

@@ -0,0 +1,34 @@
+#!/bin/bash
+# IBM_PROLOG_BEGIN_TAG
+# This is an automatically generated prolog.
+#
+# $Source: src/tools/utils/sbe-size/collect-section-sizes $
+#
+# OpenPOWER sbe Project
+#
+# Contributors Listed Below - COPYRIGHT 2018
+# [+] International Business Machines Corp.
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#
+# IBM_PROLOG_END_TAG
+
+if [ -z "${SBE_INSIDE_WORKON}" ]; then
+    echo "Must be in a workon - exiting."
+    exit 1
+fi
+
+MYDIR=$(dirname $0)
+find $SBEROOT/obj/power -name *.o | xargs python3 $MYDIR/section-sizes.py -m $SBEROOT/images/sbe_seeprom_DD2.map
+python3 $MYDIR/section-sizes.py $SBEROOT/images/sbe_seeprom_DD2.out

src/tools/utils/sbe-size/compare-section-breakdown

@@ -0,0 +1,66 @@
+#!/bin/bash
+# IBM_PROLOG_BEGIN_TAG
+# This is an automatically generated prolog.
+#
+# $Source: src/tools/utils/sbe-size/compare-section-breakdown $
+#
+# OpenPOWER sbe Project
+#
+# Contributors Listed Below - COPYRIGHT 2018
+# [+] International Business Machines Corp.
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#
+# IBM_PROLOG_END_TAG
+
+if [ -z "$1" ]; then
+    echo "Will compare section/symbol sizes between a *.o or *.map file"
+    echo "and its counterpart in the ref/ subdirectory."
+    echo "Usage: $0 current_file"
+    exit 1
+fi
+
+MYDIR=$(dirname $0)
+CROSS_COMPILER_PATH=${CROSS_COMPILER_PATH:-/afs/awd/projects/cte/tools/ppetools/prod}
+OBJDUMP=$CROSS_COMPILER_PATH/powerpc-eabi/bin/objdump
+DIFF=${DIFF:-tkdiff}
+TMPFILEa=$(mktemp)
+TMPFILEb=$(mktemp)
+
+cleanup() {
+    rm -f $TMPFILEa $TMPFILEb
+}
+trap cleanup EXIT
+
+if [[ $1 == *.map ]]; then
+    perl -pe 's/0x[0-9a-f]{16}/0xHIDDENHIDDENHIDD/' ref/$1 > $TMPFILEa
+    perl -pe 's/0x[0-9a-f]{16}/0xHIDDENHIDDENHIDD/' $1 > $TMPFILEb
+elif [[ $1 == *.o ]]; then
+    python3 $MYDIR/section-breakdown.py ref/$1 | c++filt | sort -k2 > $TMPFILEa
+    python3 $MYDIR/section-breakdown.py $1 | c++filt | sort -k2 > $TMPFILEb
+elif [[ $1 == *.su ]]; then
+    cat ref/$1 > $TMPFILEa
+    cat $1 > $TMPFILEb
+elif [[ $1 == *.dis ]]; then
+    $OBJDUMP -dr ref/${1%.dis}.o | perl -pe 's/^\s*[0-9a-fA-F]+:/xxxx:/' > $TMPFILEa
+    $OBJDUMP -dr ${1%.dis}.o | perl -pe 's/^\s*[0-9a-fA-F]+:/xxxx:/' > $TMPFILEb
+elif [[ $1 == *.diss ]]; then
+    $OBJDUMP -Sr ref/${1%.diss}.o > $TMPFILEa
+    $OBJDUMP -Sr ${1%.diss}.o > $TMPFILEb
+else
+    echo "This script can only compare *.map, *.o or *.su files so far."
+    exit 1
+fi
+
+$DIFF $TMPFILEa $TMPFILEb

src/tools/utils/sbe-size/compare-section-sizes

@@ -0,0 +1,35 @@
+#!/bin/bash
+# IBM_PROLOG_BEGIN_TAG
+# This is an automatically generated prolog.
+#
+# $Source: src/tools/utils/sbe-size/compare-section-sizes $
+#
+# OpenPOWER sbe Project
+#
+# Contributors Listed Below - COPYRIGHT 2018
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#
+# IBM_PROLOG_END_TAG
+
+if [ -z "$2" ]; then
+    echo "Usage: $0 reference_sizes current_sizes"
+    exit 1
+fi
+
+MYDIR=$(dirname $0)
+TMPFILE=$(mktemp)
+python3 $MYDIR/section-deltas.py $1 $2 > $TMPFILE
+gnuplot -e "fn='$TMPFILE'" $MYDIR/deltas.gnuplot -
+rm $TMPFILE

src/tools/utils/sbe-size/deltas.gnuplot

@@ -0,0 +1,6 @@
+set xtics rotate
+set xtics noenhanced
+set style histogram clustered
+set style data histogram
+set style fill solid border -1
+plot fn using 2:xtic(1) ti "code" lc rgb "#0000ff", '' u 3 ti "data" lc rgb "#ffff00", '' u 4 ti "stack" lc rgb "#ff0000"

src/tools/utils/sbe-size/section-breakdown.py

@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+# IBM_PROLOG_BEGIN_TAG
+# This is an automatically generated prolog.
+#
+# $Source: src/tools/utils/sbe-size/section-breakdown.py $
+#
+# OpenPOWER sbe Project
+#
+# Contributors Listed Below - COPYRIGHT 2018
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#
+# IBM_PROLOG_END_TAG
+
+from elftools.elf.elffile import ELFFile
+from os import path
+from itertools import groupby
+from sys import argv, stderr
+
+with open(argv[1], "rb") as f:
+    elf = ELFFile(f)
+
+    for section in elf.iter_sections():
+        if not section.name.startswith(".text") or section.header.sh_size == 0:
+            continue
+
+        print("%08x %s" % (section.header.sh_size, section.name.replace(".text.", "")))

src/tools/utils/sbe-size/section-deltas.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+# IBM_PROLOG_BEGIN_TAG
+# This is an automatically generated prolog.
+#
+# $Source: src/tools/utils/sbe-size/section-deltas.py $
+#
+# OpenPOWER sbe Project
+#
+# Contributors Listed Below - COPYRIGHT 2018
+# [+] International Business Machines Corp.
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#
+# IBM_PROLOG_END_TAG
+
+from sys import argv, stderr
+import numpy as np
+
+def read_file(fname):
+    result = dict()
+    with open(fname) as f:
+        for line in f:
+            parts = line.split()
+            result[parts[0]] = np.array(parts[1:], dtype=int)
+    return result
+
+reference = read_file(argv[1])
+current = read_file(argv[2])
+result = dict()
+width = len(next(iter(reference.values())))
+
+keys = set(reference.keys())
+keys.update(current.keys())
+for name in sorted(keys):
+    delta = current.get(name, np.zeros(width, dtype=int)) - reference.get(name, np.zeros(width, dtype=int))
+    if delta.any():
+        print(name, *delta)

src/tools/utils/sbe-size/section-sizes.py

@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+# IBM_PROLOG_BEGIN_TAG
+# This is an automatically generated prolog.
+#
+# $Source: src/tools/utils/sbe-size/section-sizes.py $
+#
+# OpenPOWER sbe Project
+#
+# Contributors Listed Below - COPYRIGHT 2018
+# [+] International Business Machines Corp.
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#
+# IBM_PROLOG_END_TAG
+
+from elftools.elf.elffile import ELFFile
+from os import path
+from itertools import groupby
+from sys import argv, stderr
+from argparse import ArgumentParser
+from collections import defaultdict
+
+def load_mapfile(fname):
+    sections = defaultdict(lambda: set())
+    with open(fname) as f:
+        for line in f:
+            if "memory map" in line:
+                break
+
+        for line in f:
+            if line[0:2] != " .":
+                continue
+
+            parts = line.split()
+            if len(parts) == 1:
+                parts += next(f).split()
+
+            name, addr, size, object = parts
+            if "(" in object:
+                dirname, fname = path.split(object)
+                fname = fname[fname.index("(")+1:-1]
+                object = path.join(dirname, fname)
+
+            sections[path.abspath(object)].add(name)
+
+    return sections
+
+parser = ArgumentParser(description="Collect summary of section sizes")
+parser.add_argument("-m", "--mapfile", help="Count only sections that are used according to MAPFILE")
+parser.add_argument("object", nargs="+", help="List of object files to count")
+args = parser.parse_args()
+
+section_filter = None if not args.mapfile else load_mapfile(args.mapfile)
+
+for fname in sorted(args.object):
+    fname = path.abspath(fname)
+    pname = "/".join(fname.split("/")[-2:])
+    with open(fname, "rb") as f:
+        data = 0
+        code = 0
+        elf = ELFFile(f)
+
+        for section in elf.iter_sections():
+            if section_filter and section.name not in section_filter[fname]:
+                continue
+
+            for start in (".text", ".base"):
+                if section.name.startswith(start):
+                    code += section.header.sh_size
+                    continue
+
+            for start in (".rodata", ".data", ".sdata", ".sbss", ".bss"):
+                if section.name.startswith(start):
+                    data += section.header.sh_size
+                    continue
+        stack = 0
+        if code:
+            try:
+                with open(path.splitext(fname)[0] + ".su") as f:
+                    stack = sum(int(line.split("\t")[1]) for line in f)
+            except IOError:
+                pass
+
+        if code or data or stack:
+            print(pname, code, data, stack)