Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize model loader do load method #786

Merged
merged 1 commit into from
Jul 4, 2021

Conversation

Reskov
Copy link

@Reskov Reskov commented Jun 21, 2021

  • removed double iteration over self.columns at the _do_load method
  • prepare the property to column mapping only when the column value was changed
  • removed extra loop over row values to check if all values are none

@Reskov
Copy link
Author

Reskov commented Jun 22, 2021

It is still pretty slow, but results are better on my laptop

@contextmanager
def timer():
    real_time = time.time()
    cpu_time = time.process_time()
    yield
    real_time = time.time() - real_time
    cpu_time = time.process_time() - cpu_time

    print(f"CPU time: {cpu_time:.2f} s, Real time: {real_time:.2f} s")

async def test_model_alternative(user, event_loop):
    row = await User.query.gino.load(lambda r, c: r).first()

    # async def new_load():
    loader = ModelLoader(User)
    ctx = {}

    with timer():
        print("Full model")
        timeit.timeit(lambda: loader.do_load(row, ctx), number=20000)

    loader = ModelLoader(User, User.nickname)
    with timer():
        print("Single column loader")
        timeit.timeit(lambda: loader.do_load(row, ctx), number=20000)

    row = await db.select([User.nickname]).gino.load(lambda r, c: r).first()

    loader = ModelLoader(User)
    with timer():
        print("Partial loader")
        timeit.timeit(lambda: loader.do_load(row, ctx), number=20000)

Before:

Full model
CPU time: 1.01 s, Real time: 1.05 s
Single column loader
CPU time: 0.47 s, Real time: 0.49 s
Partial loader
CPU time: 0.74 s, Real time: 0.75 s

After:

Full model
CPU time: 0.60 s, Real time: 0.60 s
Single column loader
CPU time: 0.37 s, Real time: 0.37 s
Partial loader
CPU time: 0.49 s, Real time: 0.49 s

@wwwjfy
Copy link
Member

wwwjfy commented Jul 4, 2021

Looks good, thanks!

@wwwjfy wwwjfy merged commit 75fab19 into python-gino:master Jul 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants