Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigation: Can numpy improve parsing performance #156

Open
cheahjs opened this issue Feb 21, 2024 · 3 comments
Open

Investigation: Can numpy improve parsing performance #156

cheahjs opened this issue Feb 21, 2024 · 3 comments
Labels
performance Issue related to improving performance of the library

Comments

@cheahjs
Copy link
Owner

cheahjs commented Feb 21, 2024

Investigate if using numpy to read and parse data structures could be faster than the current solution.

@cheahjs cheahjs added the performance Issue related to improving performance of the library label Feb 21, 2024
@magicbear
Copy link
Contributor

magicbear commented Feb 22, 2024

no, you may check for the way that I used to load it by multiprocessing.
MPMapPropertyProcess
MPMapProperty

@magicbear
Copy link
Contributor

24% performace from struct.Struct to int.from_bytes (like numpy)

def perf():
... t1 = time.time()
... for _ in range(100000000):
... struct.Struct("I").unpack(b"\x00\x00\x00\x00")
... print(time.time()-t1)
...

perf()
8.79693603515625
def perf():
... t1 = time.time()
... for _ in range(100000000):
... int.from_bytes(b"\x00\x00\x00\x01", byteorder="little")
... print(time.time()-t1)
...

perf()

@AntiMoron

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Issue related to improving performance of the library
Projects
None yet
Development

No branches or pull requests

3 participants