Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use tensorstore async for writing out parameter group files #61

Closed
craffel opened this issue Nov 9, 2022 · 3 comments
Closed

use tensorstore async for writing out parameter group files #61

craffel opened this issue Nov 9, 2022 · 3 comments
Assignees

Comments

@craffel
Copy link
Contributor

craffel commented Nov 9, 2022

No description provided.

@blester125
Copy link
Collaborator

blester125 commented Nov 9, 2022

Mixing async and normal code can be tricky. Here is the skeleton of a common way to do it (sorry if I over-explain).

  1. Update the calls to tenorstore code to use the async versions of things (generally add an await and turn the function into an async def)
  2. Make a write all function who's job is to start all of the async tasks
  3. Execute this writer in a blocking way.
async write_tracked_file(...):
  """This is the update to make the tensorstore write async."""
  
 async write_all_tracked_files(...):
  """This is what schedules the async tasks to be done concurrently."""
   await aysncio.gather(
       write_tracked_file(x) for x in ...
  )

# From your real code. This is executing it in a blocking way in the main synchronous code
asyncio.run(write_all_tacked_files(...))

If we want to have a minimum python version of 3.7 we can just use this asyncio.run, otherwise we probably need a custom runner like

def async_run(to_run):
  loop = asyncio.get_event_loop()
  return loop.run_until_complete(write_all_tracked_files)

Let me know if anything is unclear

@blester125
Copy link
Collaborator

The new update class modules changes how this will need to be implemented, we probably need to have async versions of the write and apply methods on those objects. We will also need to see if multiple async methods accessing different git commits causes contention, but effective parallelism/concurrency could be a major win

@blester125
Copy link
Collaborator

This was done in #121

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants