un_serialize增加开关以避免读取特定文件末尾512字节时会导致内存暴涨的问题 #65

fzls · 2021-09-05T16:32:32Z

问题说明

目前在下载完文件时，如果这个文件大于512字节，会尝试对其末尾512字节使用pickle.loads函数进行解析，如果未曾编码过，在特定文件内容时，这个函数有可能会造成大量调用内存，比如下面这个例子中会占用34G的内存。

示例

示例文件网盘链接：https://fzls.lanzoui.com/iwnVktkr9je
若链接失效，可直接下载附件文件自行上传后得到新的链接

示例代码

lzy = LanZouCloud()
lzy.down_file_by_url("https://fzls.lanzoui.com/iwnVktkr9je")

现象

修复办法

api/utils.py中增加开关，比如直接复用self._limit_mode，从而在确定不会使用额外编码来绕开官方限制的情况下（不调用ignore_limits的情况下）不会尝试使用pickle解析文件末尾512字节

原来代码

def un_serialize(data: bytes):
    """反序列化文件信息数据"""
    try:
        ret = pickle.loads(data)
        if not isinstance(ret, dict):
            return None
        return ret
    except Exception:  # 这里可能会丢奇怪的异常
        return None

调整后代码

def un_serialize(data: bytes, _limit_mode: bool):
    """反序列化文件信息数据"""
    try:
        if _limit_mode:
            # 不尝试从文件末尾解析额外编码进去的信息
            return None

        ret = pickle.loads(data)
        if not isinstance(ret, dict):
            return None
        return ret
    except Exception:  # 这里可能会丢奇怪的异常
        return None

fzls · 2021-09-05T16:33:46Z

DNF蚊子腿小助手_增量更新文件_v13.11.0_to_v13.13.0.zip

记得改名为 DNF蚊子腿小助手_增量更新文件_v13.11.0_to_v13.13.0.7z
github不让上传7z后缀的文件-。-

zaxtyson · 2021-09-07T12:37:52Z

在 Windows10 和 Linux 上面测试了多次都没有发现内存泄漏的问题。

pickle.loads(data) 处理的数据只有文件尾部 512 字节，并不会读取整个文件，应该也不存在内存泄漏的可能emm

fzls · 2021-09-08T01:46:54Z

在 Windows10 和 Linux 上面测试了多次都没有发现内存泄漏的问题。

pickle.loads(data) 处理的数据只有文件尾部 512 字节，并不会读取整个文件，应该也不存在内存泄漏的可能emm

我这边就目前这一个文件遇到过=、=之前用我那个小工具的好多朋友反馈内存会占用很大，我还以为是他们电脑问题，后来试了只有这一个文件，会出现这种问题= =

我前面测试d的时候发现换一个文件，或者关掉上面提到的那个反序列化的地方就不会有这种情况了= =

fzls · 2021-09-08T01:48:27Z

在 Windows10 和 Linux 上面测试了多次都没有发现内存泄漏的问题。

pickle.loads(data) 处理的数据只有文件尾部 512 字节，并不会读取整个文件，应该也不存在内存泄漏的可能emm

我感觉loads里面可能会根据字节码来进行各种操作，比如上面这个情况可能是误解析为一个很大的数组之类的了？

fzls · 2021-09-08T01:49:53Z

pickle官方文档看到这么一段话，感觉可能就是这个原因-。-

Warning The pickle module is not secure. Only unpickle data you trust.
It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.

Consider signing data with hmac if you need to ensure that it has not been tampered with.

Safer serialization formats such as json may be more appropriate if you are processing untrusted data. See Comparison with json.

fzls · 2021-09-08T02:03:50Z

在 Windows10 和 Linux 上面测试了多次都没有发现内存泄漏的问题。

pickle.loads(data) 处理的数据只有文件尾部 512 字节，并不会读取整个文件，应该也不存在内存泄漏的可能emm

我刚刚测试这个文件好像也没了，有点奇怪，难道跟文件本身以外的东西还有关= =

fzls · 2021-09-08T02:12:08Z

前天是在家里的电脑重现的-。-刚刚在公司电脑上确实没重现出来，我回家再看看能否重现，如果可以重现就把最后512字节打印出来，看看是啥问题

fzls · 2021-09-08T14:55:03Z

重新试了下，在家里的电脑上直接运行、wsl中运行、docker中运行均会出现这种情况，在公司的电脑中，直接运行没事，在wsl中运行会出现问题。怀疑是需要电脑内存超过34G（家中内存为64G，公司为32G），否则在申请内存时会直接提前被killed

新的精简后的测试代码

import multiprocessing
import pickle
import pickletools
import platform
import time

import psutil


def print_memory_info():
    info = psutil.virtual_memory()
    print(f"memory info: svmem(total={info.total}, available={info.available}, percent={info.percent}, used={info.used}, free={info.free})")


def print_memory_usage_every_seconds(total_time=10):
    for i in range(total_time):
        print_memory_info()
        time.sleep(1)


if __name__ == '__main__':
    multiprocessing.freeze_support()

    print(f"test on {platform.uname()}")
    print_memory_info()

    multiprocessing.Process(target=print_memory_usage_every_seconds, daemon=True).start()

    last_512_bytes = b']r(\xad\xad\x85\x91\x05\xe5s2T\xcb/\xf8\xbd\xf2\x03\xc2\x10\x88\xb5\xf6\x8a\xd7\xd7\xc1\x08\xe5`V!p\x13"\x01\x81A\xa6o\x95\x13M\x18I\xce\x02q\xcf\xd5l\xda{\xb2\x03\xa4\x07L\xc4\x86\xf6\x1f\x96\xdb+\xb0\xd9\x1e6\xd4\x9ed\xdfU\xd3\xf5S\x1a7\xe4\x03\xfd\xfc(\xda\x1f\x95\x01\xb7$af3\xcbs\xef\xda*\xcf1D\x88,[\xe3^#h\xd3\xd3-\x0bw&\n%\xae\x95Wf\xf9n\xed\xb3\x99\x082>\xa4XC\xbc3\x8d\x81?\x95\t\xe2c\xd8\xfc\xc0\x89\x1e\x8b wX\xbb\xe3!\x90RK\x17K\x03\xe5Y\x9c\x06\x18\xb9By\t\xb3-F\x90\xd5p\xf9@E\xf9\xf8\t\x1eAC\\\'\x16\xeaE:+zV\x93+\x1d\x92\xdc\x14}`!.t\xcf"\x14\xdb\xf1\xeb\xc1\xe4q`\xfa\x9b\xee\xa9(Vo]\x0b1zy\x80\xc4\x8d\xac?\xf5\r\xe8\xbc\xf8\x1c\x07,\xc4\xc6:\xb8\xab\x1d%\xaf\x9c\xfb*\xf6\xe4\xcc\xcai\x02.]\x7fB\x83\x13\xd9\xd6d\xe1Ew\xf2\xe5\x7f\xe0\x81\xdd\'\xa88c\x90L\x00\x05\x03\xec\xbeY\x00\x00\x00\x813\x07\xae\x0f\xd8\xb6\t\xe6*\x8cH\xa7\xca\xe4(yi\x85\x8a~\x01\x08\xdc\xa0\xc9\x02?v1\xa7\xd6\r\xb7\xaf\xd9\x03\xdet\xaa\x03a\xf6w0c9^\x19\xd8\x17\xa6pL\x01:\xaa\x8d\xda\xdcR\x8f\xa7\xa4\xb3C4\x94\xe7\xea\x12\\Q\x8f\x91\x0e\x99\xf6\xbcz\x16\xc0NP\xdc\x0cF&\x9fkB\xfa\x10\xba^T\xf4\xf1\t\xefy\'\x8d+}\x83Zt\x9d\xd4\x94\xd2\x88\xdcc\xf9\xfb^\xc3\x03\x15\x8c\xa6|\xc0\xf1O_/#o\xc3\xad\x06\xbe\x1c%\x04 &\x85\x1a\xa8p.\xc0(\x1b}\xb7\xfdD\x8b`+\xc7Z\xf5\x9dx\xc2i\x8eu\x0e\xb2\xea\x85"h\x96\x0b\x06\x8e\xe7\xc1C\xcd\xafU\\\x97&\x97*Q\xa6O\xbe\x17\x06\xd2\x8a>\x01\t\x80\xbf\x00\x07\x0b\x01\x00\x01#\x03\x01\x01\x05]\x00\x00\x10\x00\x0c\x81\xfa\n\x01\xcf\x12_\xe2\x00\x00'

    try:
        pickletools.dis(last_512_bytes)
    except Exception as e:
        print(f"excep={e}")

    try:
        pickle.loads(last_512_bytes)
    except Exception as e:
        print(f"excep={e}")

家中直接运行结果 (X)

test on uname_result(system='Windows', node='fzls', release='10', version='10.0.19041', machine='AMD64', processor='AMD64 Family 25 Model 80 Stepping 0, AuthenticAMD')
memory info: svmem(total=68564348928, available=51542294528, percent=24.8, used=17022054400, free=51542294528)
    0: ]    EMPTY_LIST
    1: r    LONG_BINPUT 2242751784
    6: \x91 FROZENSET  no MARK exists on stack
excep=no MARK exists on stack
memory info: svmem(total=68564348928, available=51076583424, percent=25.5, used=17487765504, free=51076583424)
memory info: svmem(total=68564348928, available=46252634112, percent=32.5, used=22311714816, free=46252634112)
memory info: svmem(total=68564348928, available=41432952832, percent=39.6, used=27131396096, free=41432952832)
memory info: svmem(total=68564348928, available=36606103552, percent=46.6, used=31958245376, free=36606103552)
memory info: svmem(total=68564348928, available=31856713728, percent=53.5, used=36707635200, free=31856713728)
memory info: svmem(total=68564348928, available=27049979904, percent=60.5, used=41514369024, free=27049979904)
memory info: svmem(total=68564348928, available=22150991872, percent=67.7, used=46413357056, free=22150991872)
memory info: svmem(total=68564348928, available=17196605440, percent=74.9, used=51367743488, free=17196605440)
memory info: svmem(total=68564348928, available=15552851968, percent=77.3, used=53011496960, free=15552851968)
memory info: svmem(total=68564348928, available=15554424832, percent=77.3, used=53009924096, free=15554424832)
excep=could not find MARK

家中wsl运行结果 (X)

test on uname_result(system='Linux', node='fzls', release='5.10.16.3-microsoft-standard-WSL2', version='#1 SMP Fri Apr 2 22:23:49 UTC 2021', machine='x86_64', processor='x86_64')
memory info: svmem(total=53790732288, available=52074532864, percent=3.2, used=809451520, free=51727347712)
    0: ]    EMPTY_LIST
    1: r    LONG_BINPUT 2242751784
    6: \x91 FROZENSET  no MARK exists on stack
excep=no MARK exists on stack
memory info: svmem(total=53790732288, available=52072976384, percent=3.2, used=811008000, free=51725791232)
memory info: svmem(total=53790732288, available=46075576320, percent=14.3, used=6808408064, free=45728391168)
memory info: svmem(total=53790732288, available=40057470976, percent=25.5, used=12826505216, free=39710277632)
memory info: svmem(total=53790732288, available=34268336128, percent=36.3, used=18615640064, free=33921142784)
memory info: svmem(total=53790732288, available=28672782336, percent=46.7, used=24211193856, free=28325588992)
memory info: svmem(total=53790732288, available=23094005760, percent=57.1, used=29789970432, free=22746812416)
memory info: svmem(total=53790732288, available=17712619520, percent=67.1, used=35171356672, free=17365426176)
memory info: svmem(total=53790732288, available=16119365632, percent=70.0, used=36764610560, free=15772172288)
memory info: svmem(total=53790732288, available=20140306432, percent=62.6, used=32747864064, free=19788918784)
excep=could not find MARK

家中docker运行 (X) -- docker run --rm fzls/debug

test on uname_result(system='Linux', node='d477019c0c3e', release='5.10.16.3-microsoft-standard-WSL2', version='#1 SMP Fri Apr 2 22:23:49 UTC 2021', machine='x86_64', processor='')
memory info: svmem(total=53790732288, available=52040724480, percent=3.3, used=844828672, free=51722014720)
    0: ]    EMPTY_LIST
    1: r    LONG_BINPUT 2242751784
    6: \x91 FROZENSET  no MARK exists on stack
excep=no MARK exists on stack
memory info: svmem(total=53790732288, available=52035772416, percent=3.3, used=849780736, free=51717062656)
memory info: svmem(total=53790732288, available=45943033856, percent=14.6, used=6943223808, free=45623209984)
memory info: svmem(total=53790732288, available=39834750976, percent=25.9, used=13051514880, free=39514918912)
memory info: svmem(total=53790732288, available=33987022848, percent=36.8, used=18899243008, free=33667190784)
memory info: svmem(total=53790732288, available=28351365120, percent=47.3, used=24534900736, free=28031533056)
memory info: svmem(total=53790732288, available=22656987136, percent=57.9, used=30229278720, free=22337155072)
memory info: svmem(total=53790732288, available=16946614272, percent=68.5, used=35939651584, free=16626774016)
memory info: svmem(total=53790732288, available=16087072768, percent=70.1, used=36799193088, free=15767232512)
memory info: svmem(total=53790732288, available=16087072768, percent=70.1, used=36799193088, free=15767232512)
excep=could not find MARK

公司电脑直接运行 (O)

test on uname_result(system='Windows', node='fzls', release='10', version='10.0.22000', machine='AMD64', processor='Intel64 Family 6 Model 158 Stepping 9, GenuineIntel')
memory info: svmem(total=34317340672, available=15854252032, percent=53.8, used=18463088640, free=15854252032)
    0: ]    EMPTY_LIST
    1: r    LONG_BINPUT 2242751784
    6: \x91 FROZENSET  no MARK exists on stack
excep=no MARK exists on stack
excep=

公司wsl运行 (X)

test on uname_result(system='Linux', node='fzls', release='5.10.43.3-microsoft-standard-WSL2', version='#1 SMP Wed Jun 16 23:47:55 UTC 2021', machine='x86_64', processor='x86_64')
memory info: svmem(total=16763052032, available=14863847424, percent=11.3, used=1476587520, free=13892059136)
    0: ]    EMPTY_LIST
    1: r    LONG_BINPUT 2242751784
    6: \x91 FROZENSET  no MARK exists on stack
excep=no MARK exists on stack
memory info: svmem(total=16763052032, available=14861516800, percent=11.3, used=1478402048, free=13890244608)
memory info: svmem(total=16763052032, available=7738986496, percent=53.8, used=8601235456, free=6766723072)
memory info: svmem(total=16763052032, available=2620260352, percent=84.4, used=13719961600, free=1647996928)
memory info: svmem(total=16763052032, available=438882304, percent=97.4, used=15917035520, free=157929472)
memory info: svmem(total=16763052032, available=308441088, percent=98.2, used=16064352256, free=136826880)
memory info: svmem(total=16763052032, available=15257268224, percent=9.0, used=1149415424, free=15325618176)
Killed

公司docker运行 (X) -- docker run --rm fzls/debug

test on uname_result(system='Linux', node='5239c35e8121', release='5.10.43.3-microsoft-standard-WSL2', version='#1 SMP Wed Jun 16 23:47:55 UTC 2021', machine='x86_64', processor='')
memory info: svmem(total=16763052032, available=14671990784, percent=12.5, used=1518010368, free=12482322432)
    0: ]    EMPTY_LIST
    1: r    LONG_BINPUT 2242751784
    6: \x91 FROZENSET  no MARK exists on stack
excep=no MARK exists on stack
memory info: svmem(total=16763052032, available=14663852032, percent=12.5, used=1526149120, free=12474183680)
memory info: svmem(total=16763052032, available=8072261632, percent=51.8, used=8117788672, free=5882290176)
memory info: svmem(total=16763052032, available=3471847424, percent=79.3, used=12718202880, free=1281875968)

fzls · 2021-09-08T14:55:53Z

根据pickletools的分析结果，应该是这段字节码被解析为创建一个2242751784大小的列表，导致最终消耗大量内存？如果内存足够的时候会进行创建，而内存不足时则会提前kill或者直接不分配内存直接抛异常

相关的opcode说明
https://juliahub.com/docs/Pickle/LAUNc/0.1.0/opcode/#Pickle.OpCodes.EMPTY_LIST
https://juliahub.com/docs/Pickle/LAUNc/0.1.0/opcode/#Pickle.OpCodes.LONG_BINPUT

zaxtyson · 2021-09-09T09:04:37Z

pickle 库确实存在反序列化漏洞，之前为了绕过官方检查才故意用这种 python 专属的序列化格式，没留意这一点。

试一试这个 https://github.com/zaxtyson/LanZouCloud-API/tree/fix-65
我们在反序列化之前检查一下二进制是不是有效的序列化数据。但是这样仍存在反序列化攻击的可能。

fzls · 2021-09-09T09:08:16Z

pickle 库确实存在反序列化漏洞，之前为了绕过官方检查才故意用这种 python 专属的序列化格式，没留意这一点。

试一试这个 https://github.com/zaxtyson/LanZouCloud-API/tree/fix-65
我们在反序列化之前检查一下二进制是不是有效的序列化数据。但是这样仍存在反序列化攻击的可能。

你这个分支好像还没提交东西哇-。-是准备先用pickletools去解析这段二进制，确保不会抛出异常，是正常的pickle序列化结果后再尝试使用pickle.loads吗0-0

fzls · 2021-09-09T09:11:04Z

如果是上面这样，至少可以把这种【非pickle序列化出来的内容，但是前面部分字节码可以正常解析，后面无法正常解析，而正常解析的这部分可能是任何行为】的情况给排除掉。一般不是pickle序列化出来的512字节，能完全符合pickle的协议的概率应该是微乎其微的，这样基本就能确保这512字节是我们上传时特地写进去的了

zaxtyson · 2021-09-09T10:54:06Z

又稍微改了一下，这种碰巧的情况应该不会遇到了

fzls closed this as completed Sep 8, 2021

fzls reopened this Sep 8, 2021

zaxtyson added the bug Something isn't working label Sep 9, 2021

zaxtyson added a commit that referenced this issue Dec 30, 2021

fix #65,#68,69

5b9624f

zaxtyson added a commit that referenced this issue Dec 30, 2021

fix #65,#68,#69

92fbe2e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

un_serialize增加开关以避免读取特定文件末尾512字节时会导致内存暴涨的问题 #65

un_serialize增加开关以避免读取特定文件末尾512字节时会导致内存暴涨的问题 #65

fzls commented Sep 5, 2021

fzls commented Sep 5, 2021 •

edited

Loading

zaxtyson commented Sep 7, 2021

fzls commented Sep 8, 2021

fzls commented Sep 8, 2021

fzls commented Sep 8, 2021 •

edited

Loading

fzls commented Sep 8, 2021

fzls commented Sep 8, 2021

fzls commented Sep 8, 2021

fzls commented Sep 8, 2021 •

edited

Loading

zaxtyson commented Sep 9, 2021 •

edited

Loading

fzls commented Sep 9, 2021 •

edited

Loading

fzls commented Sep 9, 2021

zaxtyson commented Sep 9, 2021

un_serialize增加开关以避免读取特定文件末尾512字节时会导致内存暴涨的问题 #65

un_serialize增加开关以避免读取特定文件末尾512字节时会导致内存暴涨的问题 #65

Comments

fzls commented Sep 5, 2021

问题说明

示例

修复办法

fzls commented Sep 5, 2021 • edited Loading

zaxtyson commented Sep 7, 2021

fzls commented Sep 8, 2021

fzls commented Sep 8, 2021

fzls commented Sep 8, 2021 • edited Loading

fzls commented Sep 8, 2021

fzls commented Sep 8, 2021

fzls commented Sep 8, 2021

fzls commented Sep 8, 2021 • edited Loading

zaxtyson commented Sep 9, 2021 • edited Loading

fzls commented Sep 9, 2021 • edited Loading

fzls commented Sep 9, 2021

zaxtyson commented Sep 9, 2021

fzls commented Sep 5, 2021 •

edited

Loading

fzls commented Sep 8, 2021 •

edited

Loading

fzls commented Sep 8, 2021 •

edited

Loading

zaxtyson commented Sep 9, 2021 •

edited

Loading

fzls commented Sep 9, 2021 •

edited

Loading