How can i backport new zstd kernel module to Linux 4.x?

[urain39]

I am doing the backport of zstd-1.4.10+ for Linux 4.x, but I have encountered some problems.

  aarch64-linux-gnu-ld -EL    -r -o lib/zstd/zstd_decompress.o lib/zstd/zstd_decompress_module.o lib/zstd/common/debug.o lib/zstd/common/entropy_common.o lib/zstd/common/error_private.o lib/zstd/common/fse_decompress.o lib/zstd/common/zstd_common.o lib/zstd/decompress/huf_decompress.o lib/zstd/decompress/zstd_ddict.o lib/zstd/decompress/zstd_decompress.o lib/zstd/decompress/zstd_decompress_block.o 
  aarch64-linux-gnu-ld -EL    -r -o lib/zstd/zstd_compress.o lib/zstd/zstd_compress_module.o lib/zstd/common/debug.o lib/zstd/common/entropy_common.o lib/zstd/common/error_private.o lib/zstd/common/fse_decompress.o lib/zstd/common/zstd_common.o lib/zstd/compress/fse_compress.o lib/zstd/compress/hist.o lib/zstd/compress/huf_compress.o lib/zstd/compress/zstd_compress.o lib/zstd/compress/zstd_compress_literals.o lib/zstd/compress/zstd_compress_sequences.o lib/zstd/compress/zstd_compress_superblock.o lib/zstd/compress/zstd_double_fast.o lib/zstd/compress/zstd_fast.o lib/zstd/compress/zstd_lazy.o lib/zstd/compress/zstd_ldm.o lib/zstd/compress/zstd_opt.o 
   aarch64-linux-gnu-ld -EL    -r -o lib/zstd/built-in.o lib/zstd/zstd_compress.o lib/zstd/zstd_decompress.o 
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_decompress_wksp_bmi2':
/android/lib/zstd/common/fse_decompress.c:375: multiple definition of `FSE_decompress_wksp_bmi2'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/fse_decompress.c:375: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_freeDTable':
/android/lib/zstd/common/fse_decompress.c:69: multiple definition of `FSE_freeDTable'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/fse_decompress.c:69: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_buildDTable_wksp':
/android/lib/zstd/common/fse_decompress.c:177: multiple definition of `FSE_buildDTable_wksp'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/fse_decompress.c:177: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ZSTD_customCalloc':
/android/lib/zstd/common/zstd_common.c:64: multiple definition of `ZSTD_customCalloc'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/zstd_common.c:64: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ZSTD_getErrorString':
/android/lib/zstd/common/zstd_common.c:49: multiple definition of `ZSTD_getErrorString'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/zstd_common.c:49: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ERR_isError':
/android/lib/zstd/common/error_private.h:45: multiple definition of `ZSTD_isError'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/error_private.h:45: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_createDTable':
/android/lib/zstd/common/fse_decompress.c:64: multiple definition of `FSE_createDTable'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/fse_decompress.c:64: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ERR_getErrorCode':
/android/lib/zstd/common/error_private.h:47: multiple definition of `ZSTD_getErrorName'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/error_private.h:47: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ERR_isError':
/android/lib/zstd/common/error_private.h:45: multiple definition of `FSE_isError'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/error_private.h:45: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ERR_getErrorString':
/android/lib/zstd/common/error_private.c:22: multiple definition of `ERR_getErrorString'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/error_private.c:22: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ERR_getErrorCode':
/android/lib/zstd/common/error_private.h:47: multiple definition of `ZSTD_getErrorCode'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/error_private.h:47: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ZSTD_versionNumber':
/android/lib/zstd/common/zstd_common.c:25: multiple definition of `ZSTD_versionNumber'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/zstd_common.c:25: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ERR_getErrorCode':
/android/lib/zstd/common/error_private.h:47: multiple definition of `FSE_getErrorName'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/error_private.h:47: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `HUF_readStats_wksp':
/android/lib/zstd/common/entropy_common.c:349: multiple definition of `HUF_readStats_wksp'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/entropy_common.c:349: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_readNCount_bmi2':
/android/lib/zstd/common/entropy_common.c:226: multiple definition of `FSE_readNCount_bmi2'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/entropy_common.c:226: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ZSTD_customMalloc':
/android/lib/zstd/common/zstd_common.c:57: multiple definition of `ZSTD_customMalloc'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/zstd_common.c:57: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_readNCount':
/android/lib/zstd/common/entropy_common.c:239: multiple definition of `FSE_readNCount'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/entropy_common.c:239: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `HUF_readStats':
/android/lib/zstd/common/entropy_common.c:254: multiple definition of `HUF_readStats'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/entropy_common.c:254: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `HUF_isError':
/android/lib/zstd/common/entropy_common.c:32: multiple definition of `HUF_isError'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/entropy_common.c:32: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `HUF_getErrorName':
/android/lib/zstd/common/entropy_common.c:32: multiple definition of `HUF_getErrorName'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/entropy_common.c:32: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_buildDTable_raw':
/android/lib/zstd/common/fse_decompress.c:211: multiple definition of `FSE_buildDTable_raw'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/fse_decompress.c:211: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_buildDTable_rle':
/android/lib/zstd/common/fse_decompress.c:188: multiple definition of `FSE_buildDTable_rle'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/fse_decompress.c:188: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o:/android/lib/zstd/common/debug.c:24: multiple definition of `g_debuglevel'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/debug.c:24: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_versionNumber':
/android/lib/zstd/common/entropy_common.c:27: multiple definition of `FSE_versionNumber'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/entropy_common.c:27: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_decompress_wksp':
/android/lib/zstd/common/fse_decompress.c:309: multiple definition of `FSE_decompress_wksp'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/fse_decompress.c:309: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ZSTD_versionString':
/android/lib/zstd/common/zstd_common.c:27: multiple definition of `ZSTD_versionString'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/zstd_common.c:27: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `ZSTD_customFree':
/android/lib/zstd/common/zstd_common.c:76: multiple definition of `ZSTD_customFree'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/zstd_common.c:76: first defined here
aarch64-linux-gnu-ld: lib/zstd/zstd_decompress.o: in function `FSE_decompress_usingDTable':
/android/lib/zstd/common/fse_decompress.c:303: multiple definition of `FSE_decompress_usingDTable'; lib/zstd/zstd_compress.o:/android/lib/zstd/common/fse_decompress.c:303: first defined here
make[3]: *** [/android/scripts/Makefile.build:516: lib/zstd/built-in.o] Error 1
make[2]: *** [/android/scripts/Makefile.build:656: lib/zstd] Error 2
make[1]: *** [/android/Makefile:1148: lib] Error 2
make[1]: Leaving directory '/out'
make: *** [Makefile:152: sub-make] Error 2

The output shows that the problem exists only in common/*.

I checked the code for the mainline kernel and found that these issues seem to be caused by build script changes.

(changes built-in.o -> built-in.a)

[terrelln]

If you're porting to a release older than 4.17 I think you'll just need to "undo" this commit:

torvalds/linux@dc35da1

Its a bit different now but basically look for duplicates, and only add them to zstd_compress-y. And then ifneq ($(CONFIG_ZSTD_COMPRESS)$(CONFIG_ZSTD_DECOMPRESS),yy), then also add them to zstd_decompress-y.

[urain39]

Thanks! I'm working on Linux 4.9, which is much older than 4.17.

Before this, I used a dirty trick to avoid this problem, but it doesn't seem to work.

You can see my changes at urain39/android_kernel_meizu_m1721@2561c1e

The main problem is that I get a kernel panic after changing:

dmesg-ramoops-0

 
Oops#1 Part1
<3>[   22.149248] msm_voice_source_tracking_get: Error getting Source Tracking Params, err=-22
<3>[   22.149428] msm_audio_get_copp_idx_from_port_id: Invalid FE, exiting
<3>[   22.149434] msm_audio_sound_focus_get: Could not get copp idx for port_id=16385
<3>[   22.149583] msm_audio_get_copp_idx_from_port_id: Invalid FE, exiting
<3>[   22.149588] msm_audio_source_tracking_get: Could not get copp idx for port_id=16385
<3>[   22.149732] msm_voice_sound_focus_get: Error getting Sound Focus Params, err=-22
<3>[   22.149881] msm_voice_source_tracking_get: Error getting Source Tracking Params, err=-22
<3>[   22.150085] msm_audio_get_copp_idx_from_port_id: Invalid FE, exiting
<3>[   22.150095] msm_audio_sound_focus_get: Could not get copp idx for port_id=4101
<3>[   22.150257] msm_audio_get_copp_idx_from_port_id: Invalid FE, exiting
<3>[   22.150261] msm_audio_source_tracking_get: Could not get copp idx for port_id=4101
<3>[   22.150408] msm_voice_sound_focus_get: Error getting Sound Focus Params, err=-22
<3>[   22.150560] msm_voice_source_tracking_get: Error getting Source Tracking Params, err=-22
<3>[   22.150706] msm_audio_get_copp_idx_from_port_id: Invalid FE, exiting
<3>[   22.150710] msm_audio_sound_focus_get: Could not get copp idx for port_id=4149
<3>[   22.150862] msm_audio_get_copp_idx_from_port_id: Invalid FE, exiting
<3>[   22.150867] msm_audio_source_tracking_get: Could not get copp idx for port_id=4149
<3>[   22.151058] msm_audio_get_copp_idx_from_port_id: Invalid FE, exiting
<3>[   22.151064] msm_audio_sound_focus_get: Could not get copp idx for port_id=36865
<3>[   22.151226] msm_audio_get_copp_idx_from_port_id: Invalid FE, exiting
<3>[   22.151231] msm_audio_source_tracking_get: Could not get copp idx for port_id=36865
<6>[   22.152821] core_get_license_status: cmdrsp_license_result.result = 0x15 for module 0x131ff
<6>[   22.159407] gf: gf_irq: enter
<3>[   22.163231] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.164473] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.165711] msm_pcm_volume_ctl_get substream not found
<3>[   22.166152] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.166935] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.169760] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.171553] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.171766] msm_pcm_volume_ctl_get substream or runtime not found
<3>[   22.174398] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.177962] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.178910] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<6>[   22.180521] gf: gf_irq: enter
<3>[   22.181376] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.184094] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.187142] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.190004] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.192600] msm_pcm_volume_ctl_get substream not found
<3>[   22.193863] msm_pcm_volume_ctl_get substream or runtime not found
<3>[   22.195414] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.198269] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<3>[   22.201536] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<6>[   22.201650] gf: gf_irq: enter
<3>[   22.205328] msm_pcm_volume_ctl_get substream not found
<3>[   22.205898] msm_adsp_stream_callback_get: ASM Stream PP event queue is not initialized.
<6>[   22.222801] gf: gf_irq: enter
<6>[   22.243920] gf: gf_irq: enter
<14>[   22.256986] init: starting service 'vendor.wifi_hal_legacy'...
<14>[   22.261701] init: Control message: Processed ctl.interface_start for 'android.hardware.wifi@1.0::IWifi/default' from pid: 431 (/system/bin/hwservicemanager)
<14>[   22.262882] init: Control message: Processed ctl.interface_start for 'android.hardware.wifi@1.0::IWifi/default' from pid: 431 (/system/bin/hwservicemanager)
<6>[   22.265071] gf: gf_irq: enter
<6>[   22.281395] ------------[ cut here ]------------
<2>[   22.281420] Kernel BUG at strlen+0x60/0x90 [verbose debug info unavailable]
<0>[   22.281425] Internal error: Accessing user space memory outside uaccess.h routines: 96000005 [#1] PREEMPT SMP
<6>[   22.281431] Modules linked in:
<6>[   22.281440] CPU: 4 PID: 578 Comm: touch@1.0-servi Not tainted 4.9.319 #2
<6>[   22.281444] Hardware name: Qualcomm Technologies, Inc. MSM8953 + PMI8950 QRD SKU3 (DT)
<6>[   22.281449] task: 00000000d3ab367e task.stack: 00000000ac71f63d
<6>[   22.281454] PC is at strlen+0x60/0x90
<6>[   22.281461] LR is at disable_mback_write+0xd4/0x120
<6>[   22.281465] pc : [] lr : [] pstate: 80400145
<6>[   22.281468] sp : ffffffc09ad2bd20
<6>[   22.281472] x29: ffffffc09ad2bd20 x28: ffffffc0a68de900 
<6>[   22.281478] x27: ffffff80091f2000 x26: 0000000000000040 
<6>[   22.281484] x25: 00000000000001b3 x24: 0000000000000015 
<6>[   22.281491] x23: 0000000000000000 x22: 0000007fc7e6a091 
<6>[   22.281497] x21: ffffffc09ad2beb0 x20: ffffff800979b1d8 
<6>[   22.281503] x19: 0000000000000002 x18: 00000079bda3c000 
<6>[   22.281509] x17: 00000079bd4aa0b0 x16: ffffff800825e7c0 
<6>[   22.281515] x15: ffffffffffffffff x14: ffffffffffffff97 
<6>[   22.281521] x13: ffffff80ffffffd0 x12: 0000007fc7e69f70 
<6>[   22.281527] x11: 0101010101010101 x10: ffffffc09ad2bd20 
<6>[   22.281532] x9 : 000000000000000a x8 : 0000000000000001 
<6>[   22.281538] x7 : ffffffffffffffff x6 : 0000000000000001 
<6>[   22.281544] x5 : 0000000000000001 x4 : 00000000000000d0 
<6>[   22.281549] x3 : 0000000000000000 x2 : 0000000000000001 
<6>[   22.281555] x1 : 0000007fc7e6a090 x0 : 0000007fc7e6a091 
<6>[   22.281563] 
<6>[   22.281563] PC: strlen+0x20/0x90:
<6>[   22.281567] 8980  b200d86a 8a2800e5 ea2a0126 fa4008a0 54ffff00 cb000020 b4000065 d1002000
<6>[   22.281586] 89a0  aa0503e6 d1002000 dac00cc6 dac010cc 8b4c0c00 d65f03c0 f10020ff cb0703e7
<6>[   22.281603] 89c0  a8c10c22 d37df0e7 92800008 9ac72508 aa080042 aa080064 da9fd042 9a84d063
<6>[   22.281621] 89e0  17ffffe5 00000000 00000000 00000000 b4001002 ca010008 b200c3eb f240091f
<6>[   22.281641] 
<6>[   22.281641] LR: disable_mback_write+0x94/0x120:
<6>[   22.281645] 03f4  91011be0 910103e2 f0003d41 91122021 97db5d97 7100041f 540001c1 b94043e1
<6>[   22.281662] 0414  7100043f 5400016c 7100003f f000b861 1a9f07e2 aa1603e0 392bc022 97db214c
<6>[   22.281680] 0434  93407c13 f90002b3 f9401bf7 17ffffdc 93407c13 f9401bf7 17ffffd9 cb000273
<6>[   22.281696] 0454  aa0003e2 91011be0 52800001 8b130000 97db2067 34fffc77 93407ef3 f9401bf7
<6>[   22.281715] 
<6>[   22.281715] SP: 0xffffffc09ad2bce0:
<6>[   22.281719] bce0  08df0434 ffffff80 9ad2bd20 ffffffc0 084b89c0 ffffff80 80400145 00000000
<6>[   22.281736] bd00  00000140 00000000 9ad2bd68 ffffffc0 ffffffff 0000007f 00000001 00000000
<6>[   22.281753] bd20  9ad2bd70 ffffffc0 082deddc ffffff80 ab6d3200 ffffffc0 fffffffb ffffffff
<6>[   22.281770] bd40  c7e6a091 0000007f 9ad2beb0 ffffffc0 00000000 00000000 bd08fd04 00000079
<0>[   22.281788] Process touch@1.0-servi (pid: 578, stack limit = 0x00000000ac71f63d)
<6>[   22.281791] Call trace:
<6>[   22.281798] Exception stack(0xffffffc09ad2bb30 to 0xffffffc09ad2bc60)
<6>[   22.281803] bb20:                                   0000000000000002 0000007fffffffff
<6>[   22.281808] bb40: 000000004260a000 ffffff80084b89c0 0000000080400145 ffffff800822f144
<6>[   22.281814] bb60: ffffffc09ad2bb70 ffffff8008251e40 ffffffc09ad2bbc0 ffffff80084c76cc
<6>[   22.281819] bb80: ffffffc09ad2bd66 ffffff800979b1d8 ffffffc09ad2bd66 00000079bd08fd04
<6>[   22.281823] bba0: ffffffc09ad2bbe0 0000000a08223ec0 0000000000000001 0000000000040900
<6>[   22.281828] bbc0: ffffffc09ad2bc80 ffffff80084c7abc ffffff800979b1d8 ffffff800979b1d8
<6>[   22.281833] bbe0: ffffffc09ad2beb0 0000000000040900 0000007fc7e6a091 0000007fc7e6a090
<6>[   22.281838] bc00: 0000000000000001 0000000000000000 00000000000000d0 0000000000000001
<6>[   22.281842] bc20: 0000000000000001 ffffffffffffffff 0000000000000001 000000000000000a
<6>[   22.281847] bc40: ffffffc09ad2bd20 0101010101010101 0000007fc7e69f70 ffffff80ffffffd0
<6>[   22.281852] [<000000002366826e>] strlen+0x60/0x90
<6>[   22.281860] [<000000006759e18d>] proc_reg_write+0x5c/0x90
<6>[   22.281867] [<00000000a9649d56>] __vfs_write+0x30/0x130
<6>[   22.281871] [<000000009818d941>] vfs_write+0xa4/0x1b0
<6>[   22.281876] [<00000000d92ef78b>] SyS_write+0x64/0xe0
<6>[   22.281883] [<000000002a779d31>] el0_svc_naked+0x34/0x38
<0>[   22.281889] Code: 8b4c0c00 d65f03c0 f10020ff cb0703e7 (a8c10c22) 
<4>[   22.281959] ---[ end trace 81b56de572cf8b43 ]---
  

Since this is a custom Linux kernel, I can't identify the immediate cause, I'll test with mainline 4.9 when I have time (it may take a bit long time).

[urain39]

Sorry, this is my mistake. This issue (kernel panic) is caused by CONFIG_ARM64_SW_TTBR0_PAN=y because of upstream modification of custom kernel.