Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

write result is inaccurate when multi-coroutine is executed #135

Closed
xiaoxinmiao opened this issue Mar 31, 2022 · 2 comments
Closed

write result is inaccurate when multi-coroutine is executed #135

xiaoxinmiao opened this issue Mar 31, 2022 · 2 comments

Comments

@xiaoxinmiao
Copy link

xiaoxinmiao commented Mar 31, 2022

Inaccurate results on windows when mode=a is set and Semaphore is 2

Environment:

system: windows
aiofiles version: 0.8.0

Test

test1:

sem = asyncio.Semaphore(2)
When the parameter is passed in 1, the result is correct
When the parameter is passed in 2, the result is incorrect

test2

Wrong result on windows, correct result on linux

Source

import asyncio
import os

import aiofiles


async def write_chunk_to_file(sem: asyncio.Semaphore, chunk: str) -> None:
    async with sem:
        async with aiofiles.open("test.txt", mode="a") as f:
            await f.write(f'{chunk}\n')


async def main() -> None:
    # Prepare the data to write in batches
    data_chunks: list[str] = [
        "Berlin", "Germany", "Madrid", "Spain", "Rome", "Italy",
        "Washington D.C.", "USA", "Ottawa", "Canada", "Mexico City", "Mexico",
        "Tokyo", "Japan", "Beijing", "China", "Manila", "Philippines"
    ]

    # Truncate existing file
    file_path = 'test.txt'
    if os.path.isfile(file_path):
        os.remove(file_path)

    # Launch coroutines to write batches to the file
    sem = asyncio.Semaphore(1)
    await asyncio.gather(*[write_chunk_to_file(sem, chunk) for chunk in data_chunks])

if __name__ == "__main__":
    asyncio.run(main())

Result

The result of running on windows when setting Semaphore to 2

expect data


1. Berlin
2. Germany
3. Madrid
4. Spain
5. Rome
6. Italy
7. Washington D.C.
8. USA
9. Ottawa
10. Canada
11. Mexico City
12. Mexico
13. Tokyo
14. Japan
15. Beijing
16. China
17. Manila
18. Philippines

actual data

1. Berlin
2. Germany
3. Madrid
4. Italy
5. Washington D.C.
6. Ottawa
7. Mexico City
8. Tokyo
9. Beijing
10. China
11. Philippines

动画2

@Tinche
Copy link
Owner

Tinche commented Mar 31, 2022

You're probably running into a race condition caused by the combination of Python and the operating system, I don't think I can fix this in aiofiles. I would suggest using a lock (or a semaphore with a value of 1) to mitigate the issue.

@Tinche Tinche closed this as completed Mar 31, 2022
@xiaoxinmiao
Copy link
Author

@Tinche ok,by the way, is it reliable to run on linux and macOs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants