UNIX - SystemError: (9, 'Bad File Descriptor') on writing to file by line

[T012m3n7oR]

I have a script that is working on my Windows 10 Pro machine, that is not working on my AWS Environment. Am I missing something simple?

MS Windows Version 2004 (OS Build 19041.685)
Python 3.8.0

UNIX Version:
uname -r: 4.14.214-160.339.amzn2.x86_64
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
Python 3.8.0 (Manually compiled, same issue on the stock python 3.7.9 that I could install with yum)

Requirements file:

aiofile==3.3.3
aiohttp==3.7.3
async-timeout==3.0.1
attrs==20.3.0
caio==0.6.3
cffi==1.14.4
chardet==3.0.4
discord.py==1.6.0
idna==3.1
multidict==5.1.0
pycparser==2.20
PyNaCl==1.4.0
python-dotenv==0.15.0
six==1.15.0
typing-extensions==3.7.4.3
yarl==1.6.3

The following code throws the error:

async def save_8ball(guild_id):
    async with aiofile.async_open(os.path.join(CONFIG['8Ball']['FileDirectory'], f'{guild_id}.txt'), 'w')\
            as create_8ball:
        for resp in EIGHT_BALL_RESPONSES[guild_id]:
            await create_8ball.write(f'{resp}\r\n')

I am using a relative path for the os.path.join, as the config file is currently pointing to "./8ball"

Error Listing:

Traceback (most recent call last):
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/caio/asyncio_base.py", line 43, in step
    self.context.submit(*operations.keys())
SystemError: (9, 'Bad file descriptor')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/discord/ext/commands/core.py", line 85, in wrapped
    ret = await coro(*args, **kwargs)
  File "./botGames.py", line 96, in prediction
    await create_guild_8ball(guild_id=ctx.guild.id)
  File "./botGames.py", line 66, in create_guild_8ball
    await save_8ball(guild_id=guild_id)
  File "./botGames.py", line 34, in save_8ball
    await create_8ball.write(f'{resp}\r\n')
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/aiofile/utils.py", line 183, in __aexit__
    await self.close()
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/aiofile/utils.py", line 176, in close
    await self.file.close()
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/aiofile/aio.py", line 166, in close
    await self.fsync()
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/aiofile/aio.py", line 261, in fsync
    return await self.__context.fdsync(self.fileno())
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/caio/asyncio_base.py", line 87, in submit
    await future
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/caio/asyncio_base.py", line 48, in step
    self.context.submit(operation)
SystemError: (9, 'Bad file descriptor')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/discord/ext/commands/bot.py", line 902, in invoke
    await ctx.command.invoke(ctx)
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/discord/ext/commands/core.py", line 1345, in invoke
    await super().invoke(ctx)
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/discord/ext/commands/core.py", line 864, in invoke
    await injected(*ctx.args, **ctx.kwargs)
  File "/home/ec2-user/python_discord_bot/lib/python3.8/site-packages/discord/ext/commands/core.py", line 94, in wrapped
    raise CommandInvokeError(exc) from exc
discord.ext.commands.errors.CommandInvokeError: Command raised an exception: SystemError: (9, 'Bad file descriptor')

When substituting the aiofile.async_open for a regular open this succeeds without issue on the Linux machine.
Reading online this error usually means that the file was closed and then attempted to be referenced, but I'm writing into the file, and then closing the file.

I've also tried to write into this using AIOFile with a writer instead of async_open, but I'm receiving the same result.

[T012m3n7oR]

This may be related to #42 and #53.
It looks similar, but different error.

[Natim]

What do you read for os.path.join(CONFIG['8Ball']['FileDirectory'], f'{guild_id}.txt')

[T012m3n7oR]

The os.path.join translates to:

os.path.join('./8ball', '806944440488296508.txt')

[T012m3n7oR]

The issue still exists, but I've transitioned to just using an async SQLite call so I don't need to worry about async file access.

[pymen]

short code which reproduces same issue

import asyncio
from aiofile import async_open


async def write_data(data):
    async with async_open('file-check-write.txt', 'w') as afp:
        await afp.write(data)


async def main():
    task = asyncio.create_task(write_data(data='some data'.encode()))
    await asyncio.gather(task)


if __name__ == '__main__':
    asyncio.run(main())

Ubutnu 16.04/ Python 3.7

aiofile 3.4.0
aiohttp 3.7.3

 File "/home/ant/projects/async/async/download.py", line 16, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/home/ant/projects/async/async/download.py", line 12, in main
    await asyncio.gather(task)
  File "/home/ant/projects/async/async/download.py", line 7, in write_data
    await afp.write(data)
  File "/home/ant/.virtualenvs/async/lib/python3.7/site-packages/aiofile/utils.py", line 183, in __aexit__
    await self.close()
  File "/home/ant/.virtualenvs/async/lib/python3.7/site-packages/aiofile/utils.py", line 176, in close
    await self.file.close()
  File "/home/ant/.virtualenvs/async/lib/python3.7/site-packages/aiofile/aio.py", line 166, in close
    await self.fsync()
  File "/home/ant/.virtualenvs/async/lib/python3.7/site-packages/aiofile/aio.py", line 261, in fsync
    return await self.__context.fdsync(self.fileno())
  File "/home/ant/.virtualenvs/async/lib/python3.7/site-packages/caio/asyncio_base.py", line 87, in submit
    await future
  File "/home/ant/.virtualenvs/async/lib/python3.7/site-packages/caio/asyncio_base.py", line 48, in step
    self.context.submit(operation)
SystemError: (9, 'Bad file descriptor')

[mosquito]

@pymen Please provide linux kernel version and the file system.

[pymen]

@mosquito
ext4 and 4.12.3-041203-generic

there is a warning, but it is not quite clear that everything will be broken

/usr/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: Linux supports fsync/fdsync with io_submit since 4.18 but current kernel 4.12.3-041203-generic doesn't support it. Related calls will have no effect.

and exception

  return f(*args, **kwds)
Traceback (most recent call last):
  File "/home/ant/.virtualenvs/async/lib/python3.7/site-packages/caio/asyncio_base.py", line 43, in step
    self.context.submit(*operations.keys())
SystemError: (9, 'Bad file descriptor')

[mosquito]

@pymen could you please upgrade kernel? Just for experiment? I think maybe restrict use linux_aio for kernels before 4.18.

[pymen]

@mosquito, unfortunately, the Nvidia driver have made my system not working after upgrade to 4.18, and the next upgrade to 18.04 finished it off, so I had to install new ubuntu 20.4, where the script works perfectly

[mosquito]

Probably fixed in 3.5.0