Can Address Sanitizer be applied to TA?

[SangsubLim]

I would like to apply Address Sanitizer to CA and TA by referring to the link below.

• https://www.osc.edu/resources/getting_started/howto/howto_use_address_sanitizer

Looking at config.mk of optee_os and the link below, it seems that sanitizer can be applied to optee-os(CFG_CORE_SANITIZE_KADDRESS).

• Address sanitizer #1024

As I said before, I plan to apply Sanitizer to CA and TA as well. For this, I used hello world in optee_example on QEMU v8.

First, I tried CA, and modified it as follows.

Makefile

ROOT = <my project dir>
CROSS_COMPILE = $(ROOT)/toolchains/aarch64/bin/aarch64-linux-gnu-
SYSROOT = $(ROOT)/out-br/host/aarch64-buildroot-linux-gnu/sysroot
TEEC_EXPORT = $(SYSROOT)/usr

CC      = $(CROSS_COMPILE)gcc
LD      = $(CROSS_COMPILE)ld
AR      = $(CROSS_COMPILE)ar
NM      = $(CROSS_COMPILE)nm
OBJCOPY = $(CROSS_COMPILE)objcopy
OBJDUMP = $(CROSS_COMPILE)objdump
READELF = $(CROSS_COMPILE)readelf

OBJS = main.o

CFLAGS += -Wall -I../ta/include -I$(TEEC_EXPORT)/include -I./include
LDFLAGS += --sysroot=$(SYSROOT) 

#added for sanitizer
CFLAGS += -fsanitize=address -static-libasan -g
LDFLAGS += -fsanitize=address -static-libasan -g

#Add/link other required libraries here
LDADD += -lteec -L$(TEEC_EXPORT)/lib

BINARY = optee_example_hello_world

.PHONY: all
all: $(BINARY)

$(BINARY): $(OBJS)
	$(CC) $(LDFLAGS) -o $@ $< $(LDADD)

.PHONY: clean
clean:
	rm -f $(OBJS) $(BINARY)

%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

After confirming that it was built normally, I intentionally caused an error by modifying the main.c file as follows.

main.c

int main(void)
{
	TEEC_Result res;
	TEEC_Context ctx;
	TEEC_Session sess;
	TEEC_Operation op;
	TEEC_UUID uuid = TA_HELLO_WORLD_UUID;
	uint32_t err_origin;

    // whoops, forgot c strings are null-terminated
    // and not enough memory was allocated for the copy
    char *s = malloc(12);
    strcpy(s, "Hello world!");
    printf("string is: %s\n", s);
    free(s);

...
}

As a result of the test, it was confirmed that the error occurred as follows.

# ./optee_example_hello_world
=================================================================
==240==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xffff8fc007bc at pc 0x00000041cdbc bp 0xfffffa0d8ba0 sp 0xfffffa0d8bf8
WRITE of size 13 at 0xffff8fc007bc thread T0
...

One curious thing is, optee_example_hello_world binary size has increased significantly. (14088->9785104)
I think it increased as libasan was statically included in the CA binary, is that correct?

Anyway, although it is in the title, the problem I am currently experiencing is that AddressSanitizer cannot be applied on the TA.
First of all, I looked at the link below and tried to add libasan to TA.

• https://optee.readthedocs.io/en/latest/building/trusted_applications.html#build-trusted-applications-submk

sub.mk

global-incdirs-y += include
srcs-y += hello_world_ta.c

# Adds the static library asan to the list of the linker directive -lasan.
libnames += asan

# Adds the directory path to the libraries pathes list. Archive file 
# libasan.a is expected in this directory.
libdirs += <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64

# To remove a certain compiler flag, add a line like this
cflags-template_ta.c-y += -fsanitize=address -static-libasan -g

Makefile

ROOT = <my optee home>
CROSS_COMPILE = $(ROOT)/toolchains/aarch64/bin/aarch64-linux-gnu-
PLATFORM = vexpress-qemu_armv8a
TA_DEV_KIT_DIR = $(ROOT)/optee_os/out/arm/export-ta_arm64

CFLAGS += -fsanitize=address -static-libasan -g
LDFLAGS += -fsanitize=address -static-libasan -g

CFG_TEE_TA_LOG_LEVEL ?= 4

# The UUID for the Trusted Application
BINARY=8aaaf200-2450-11e4-abe2-0002a5d5c51b

-include $(TA_DEV_KIT_DIR)/mk/ta_dev_kit.mk

ifeq ($(wildcard $(TA_DEV_KIT_DIR)/mk/ta_dev_kit.mk), )
clean:
	@echo 'Note: $$(TA_DEV_KIT_DIR)/mk/ta_dev_kit.mk not found, cannot clean TA'
	@echo 'Note: TA_DEV_KIT_DIR=$(TA_DEV_KIT_DIR)'
endif

After modifying as above, the following error occurs when TA is built.

optee_examples/hello_world/ta$ make V=1
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-gcc -fsanitize=address -static-libasan -g -std=gnu99 -fdiagnostics-show-option -Wall -Wcast-align -Werror-implicit-function-declaration -Wextra -Wfloat-equal -Wformat-nonliteral -Wformat-security -Wformat=2 -Winit-self -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wmissing-prototypes -Wnested-externs -Wpointer-arith -Wshadow -Wstrict-prototypes -Wswitch-default -Wwrite-strings -Wno-missing-field-initializers -Wno-format-zero-length -Wno-c2x-extensions -Wredundant-decls -Wold-style-definition -Wstrict-aliasing=2 -Wundef -Os -g3 -fpic -mstrict-align -mno-outline-atomics -MD -MF ./.hello_world_ta.o.d -MT hello_world_ta.o -nostdinc -isystem <my optee home>/toolchains/aarch64/bin/../lib/gcc/aarch64-none-linux-gnu/10.2.1/include -I./include -I./. -DARM64=1 -D__LP64__=1 -DMBEDTLS_SELF_TEST -DTRACE_LEVEL=4 -I. -I/<my optee home>/optee_os/out/arm/export-ta_arm64/include -DCFG_ARM64_ta_arm64=1 -DCFG_TEE_TA_LOG_LEVEL=4 -DCFG_FTRACE_SUPPORT=1 -DCFG_SYSTEM_PTA=1 -DCFG_UNWIND=1 -DCFG_TA_BGET_TEST=1 -DCFG_TA_MBEDTLS=1 -DCFG_TA_MBEDTLS_SELF_TEST=1 -DCFG_TA_MBEDTLS_MPI=1 -DCFG_TA_FLOAT_SUPPORT=1 -D__FILE_ID__=hello_world_ta_c -c hello_world_ta.c -o hello_world_ta.o
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-objcopy --rename-section .rodata=.rodata.hello_world_ta.c --rename-section .rodata.str1.1=.rodata.str1.1.hello_world_ta.c ./hello_world_ta.o
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-gcc -fsanitize=address -static-libasan -g -std=gnu99 -fdiagnostics-show-option -Wall -Wcast-align -Werror-implicit-function-declaration -Wextra -Wfloat-equal -Wformat-nonliteral -Wformat-security -Wformat=2 -Winit-self -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wmissing-prototypes -Wnested-externs -Wpointer-arith -Wshadow -Wstrict-prototypes -Wswitch-default -Wwrite-strings -Wno-missing-field-initializers -Wno-format-zero-length -Wno-c2x-extensions -Wredundant-decls -Wold-style-definition -Wstrict-aliasing=2 -Wundef -Os -g3 -fpic -mstrict-align -mno-outline-atomics -MD -MF ./.user_ta_header.o.d -MT user_ta_header.o -nostdinc -isystem <my optee home>/toolchains/aarch64/bin/../lib/gcc/aarch64-none-linux-gnu/10.2.1/include -I./include -DARM64=1 -D__LP64__=1 -DMBEDTLS_SELF_TEST -DTRACE_LEVEL=4 -I. -I<my optee home>/optee_os/out/arm/export-ta_arm64/include -DCFG_ARM64_ta_arm64=1 -DCFG_TEE_TA_LOG_LEVEL=4 -DCFG_FTRACE_SUPPORT=1 -DCFG_SYSTEM_PTA=1 -DCFG_UNWIND=1 -DCFG_TA_BGET_TEST=1 -DCFG_TA_MBEDTLS=1 -DCFG_TA_MBEDTLS_SELF_TEST=1 -DCFG_TA_MBEDTLS_MPI=1 -DCFG_TA_FLOAT_SUPPORT=1 -D__FILE_ID__=user_ta_header_c -c <my optee home>/optee_os/out/arm/export-ta_arm64/src/user_ta_header.c -o user_ta_header.o
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-objcopy --rename-section .rodata=.rodata.user_ta_header.c --rename-section .rodata.str1.1=.rodata.str1.1.user_ta_header.c ./user_ta_header.o
mkdir -p ./
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-cpp -P -MT ta.lds -MD -MF ./.ta.ld.d -nostdinc -isystem <my optee home>/toolchains/aarch64/bin/../lib/gcc/aarch64-none-linux-gnu/10.2.1/include -I./include -I. -DARM64=1 -D__LP64__=1 -DMBEDTLS_SELF_TEST -DTRACE_LEVEL=4 -I. -I<my optee home>/optee_os/out/arm/export-ta_arm64/include -DCFG_ARM64_ta_arm64=1 -DCFG_TEE_TA_LOG_LEVEL='4' -DCFG_FTRACE_SUPPORT=1 -DCFG_SYSTEM_PTA=1 -DCFG_UNWIND=1 -DCFG_TA_BGET_TEST=1 -DCFG_TA_MBEDTLS=1 -DCFG_TA_MBEDTLS_SELF_TEST=1 -DCFG_TA_MBEDTLS_MPI=1 -DCFG_TA_FLOAT_SUPPORT=1 <my optee home>/optee_os/out/arm/export-ta_arm64/src/ta.ld.S -o ta.lds
mkdir -p ./
echo "{" >dyn_list
echo "__elf_phdr_info;" >>dyn_list
echo "__ftrace_info;" >>dyn_list
echo "trace_ext_prefix;" >>dyn_list
echo "trace_level;" >>dyn_list
echo "};" >>dyn_list
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd -e__ta_entry -pie -T ./ta.lds -Map=./8aaaf200-2450-11e4-abe2-0002a5d5c51b.map --sort-section=alignment -z max-page-size=4096  --as-needed   --dynamic-list ./dyn_list  ./hello_world_ta.o ./user_ta_header.o  -L<my optee home>/optee_os/out/arm/export-ta_arm64/lib -L<my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64 --start-group -lutils -lutee -lmbedtls -ldl -lasan --end-group <my optee home>/toolchains/aarch64/bin/../lib/gcc/aarch64-none-linux-gnu/10.2.1/libgcc.a -lutils -o 8aaaf200-2450-11e4-abe2-0002a5d5c51b.elf
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd: warning: libdl.so.2, needed by <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64/libasan.so, not found (try using -rpath or -rpath-link)
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd: warning: librt.so.1, needed by <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64/libasan.so, not found (try using -rpath or -rpath-link)
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd: warning: libpthread.so.0, needed by <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64/libasan.so, not found (try using -rpath or -rpath-link)
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd: warning: libstdc++.so.6, needed by <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64/libasan.so, not found (try using -rpath or -rpath-link)
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd: warning: libm.so.6, needed by <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64/libasan.so, not found (try using -rpath or -rpath-link)
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd: warning: libc.so.6, needed by <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64/libasan.so, not found (try using -rpath or -rpath-link)
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd: warning: libgcc_s.so.1, needed by <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64/libasan.so, not found (try using -rpath or -rpath-link)
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd: <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64/libasan.so: undefined reference to `__signgam@GLIBC_2.23'
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd: <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64/libasan.so: undefined reference to `signgam@GLIBC_2.17'
<my optee home>/toolchains/aarch64/bin/aarch64-linux-gnu-ld.bfd: <my optee home>/toolchains/aarch64/aarch64-none-linux-gnu/lib64/libasan.so: undefined reference to `close@GLIBC_2.17'

...
make: *** [<my optee home>/optee_os/out/arm/export-ta_arm64/mk/link.mk:120: 8aaaf200-2450-11e4-abe2-0002a5d5c51b.elf] Error 1

#901
I checked the above link, but I don't know what to do.

If anyone knows anything about this, please reply.
thank you

[jenswi-linaro]

Clients using libteec operate in Linux user space so the normal Linux user space based address sanitizer is expected to work without any OP-TEE specific changes.

However, TAs operates in a completely different environment so compiled Linux user space libraries can't be expected to work. In this case, I'm not sure if you even can re-use any of the libasan source code.