Broken large file support on AIX

[sable]

BPO	11184
Nosy	@birkenfeld, @rhettinger, @pitrou
Files	patch_aix_largefile.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/birkenfeld'
closed_at = <Date 2011-02-25.11:21:10.166>
created_at = <Date 2011-02-11.11:37:51.690>
labels = ['expert-IO']
title = 'Broken large file support on AIX'
updated_at = <Date 2011-02-25.11:21:10.165>
user = 'https://bugs.python.org/sable'

bugs.python.org fields:

activity = <Date 2011-02-25.11:21:10.165>
actor = 'georg.brandl'
assignee = 'georg.brandl'
closed = True
closed_date = <Date 2011-02-25.11:21:10.166>
closer = 'georg.brandl'
components = ['IO']
creation = <Date 2011-02-11.11:37:51.690>
creator = 'sable'
dependencies = []
files = ['20767']
hgrepos = []
issue_num = 11184
keywords = ['patch']
message_count = 19.0
messages = ['128374', '128376', '128377', '128385', '128386', '128387', '128388', '128390', '128391', '128392', '128394', '128395', '128643', '128682', '128683', '128709', '128836', '128837', '129356']
nosy_count = 4.0
nosy_names = ['georg.brandl', 'rhettinger', 'pitrou', 'sable']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue11184'
versions = ['Python 2.7']

[sable]

I get 2 errors when running test_io.py with trunk on AIX.

======================================================================
ERROR: test_large_file_ops (main.CIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
OverflowError: Python int too large to convert to C long

======================================================================
ERROR: test_large_file_ops (main.PyIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
OverflowError: Python int too large to convert to C long

---

Ran 395 tests in 27.958s

FAILED (errors=2, skipped=8)

thanks in advance

[pitrou]

Hmm, strange. Is it a 32-bit build? Is HAVE_LARGEFILE_SUPPORT defined in pyconfig.h?

[pitrou]

Apparently AIX needs a specific #define to enable large file support:
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/prg_lrg_files.htm

Python defines _LARGEFILE_SOURCE by default.

[sable]

OK, so the following patch should help:

Index: configure.in
===================================================================

--- configure.in        (revision 88393)
+++ configure.in        (working copy)
@@ -1375,6 +1375,8 @@
 
 if test "$use_lfs" = "yes"; then
 # Two defines needed to enable largefile support on various platforms
+AC_DEFINE(_LARGEFILES, 1, 
+[This must be defined on some systems to enable large file support.])
 # These may affect some typedefs
 AC_DEFINE(_LARGEFILE_SOURCE, 1, 
 [This must be defined on some systems to enable large file support.])

[sable]

the error is different now that _LARGEFILES is defined:

======================================================================
ERROR: test_large_file_ops (main.CIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
IOError: [Errno 22] Invalid argument

======================================================================
ERROR: test_large_file_ops (main.PyIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
IOError: [Errno 22] Invalid argument

[pitrou]

Hmm, interesting. Can you post the results of the two following snippets:

>>> f = open('foo', 'wb')
>>> f.seek(2**32)
# should be 4294967296

>>> f = open('foo', 'wb')
>>> f.truncate(2**32)
# should be 4294967296

[sable]

Sorry I made a mistake in my previous patch (_LARGEFILES instead of _LARGE_FILES).

Here is a better one:

Index: configure.in
===================================================================

--- configure.in	(révision 88395)
+++ configure.in	(copie de travail)
@@ -1376,6 +1376,14 @@
 if test "$use_lfs" = "yes"; then
 # Two defines needed to enable largefile support on various platforms
 # These may affect some typedefs
+    case $ac_sys_system/$ac_sys_release in
+    AIX*)
+        AC_DEFINE(_LARGE_FILES, 1, 
+        [This must be defined on AIX systems to enable large file support.])
+	;;
+    *)
+        ;;
+    esac
 AC_DEFINE(_LARGEFILE_SOURCE, 1, 
 [This must be defined on some systems to enable large file support.])
 AC_DEFINE(_FILE_OFFSET_BITS, 64,

The test fails in a different way now:

======================================================================
ERROR: test_large_file_ops (main.CIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 323, in large_file_ops
    self.assertEqual(f.write(b"xxx"), 3)
IOError: [Errno 27] File too large

======================================================================
ERROR: test_large_file_ops (main.PyIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 323, in large_file_ops
    self.assertEqual(f.write(b"xxx"), 3)
IOError: [Errno 27] File too large

---

Ran 395 tests in 27.958s

Here is your trace:
phenix:~/.buildbot/python-aix6/3.x.phenix.xlc/build\> ./python 
Python 3.2rc2+ (py3k:88393M, Feb 11 2011, 14:56:34) [C] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('foo', 'wb')
[55983 refs]
>>> f.seek(2**32)
4294967296
[55987 refs]
>>> f = open('foo', 'wb')
__main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
[55994 refs]
>>> f.truncate(2**32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 27] File too large
[56027 refs]

[pitrou]

Thanks for the patch.

The test fails in a different way now:
[...]
IOError: [Errno 27] File too large

This seems to mean that your file system isn't configured for large
files. According to
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/prg_lrg_files.htm :

    For the JFS, the maximum file size is determined by the
    parameters used at the time the file system was made. For JFS
    file systems that are enabled for large files, the maximum file
    size is slightly less than 64 gigabytes (0xff8400000). For all
    other JFS file systems, the maximum file size is 2Gb-1
    (0x7fffffff). Attempts to write a file in excess of the maximum
    file size in any file system format will fail, and errno will be
    set to EFBIG.

(I'm not under AIX, but EFBIG is 27 here)

What does test_largefile output?

[sable]

test_largefile complains about the filesystem having no largefile support.
It is probably the case, I will ask a sysadmin, and see if he can get me a file system with large file support so that I can test this feature.

> ./python -Wd -E -bb ./Lib/test/test_largefile.py 
Traceback (most recent call last):
  File "./Lib/test/test_largefile.py", line 163, in test_main
    f.write(b'x')
IOError: [Errno 27] File too large

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./Lib/test/test_largefile.py", line 192, in <module>
    test_main()
  File "./Lib/test/test_largefile.py", line 168, in test_main
    raise unittest.SkipTest("filesystem does not have largefile support")
unittest.case.SkipTest: filesystem does not have largefile support
[81125 refs]

In the meantime, this test should probably be skipped just like in test_largefile.py and the patch with _LARGE_FILES should probably be applied too.

Merci pour l'aide

[pitrou]

Yes, I think the skipping code in test_largefile should be factored out and used both in test_io and test_largefile (to be honest I don't know why test_io has large file tests as well; perhaps I should merge them together).

[sable]

Also:

is it OK if I open a new issue for each broken unit test on AIX even if I have not investigated them at the moment?

I have a dozen broken unit tests on AIX that need to be investigated, but I don't want to spam the bug tracker followers too much.
So tell me if it is a good thing to report each problem as soon as possible or if I should do it step by step.

[pitrou]

is it OK if I open a new issue for each broken unit test on AIX even
if I have not investigated them at the moment?

Yes. That way they get recorded somewhere and other people can chime in.
If you plan to investigate them you can add a sentence saying so.

Thanks!

[sable]

Here is the patch.
It only impacts AIX systems and is minimalist, so I think it should be safe for Python 3.2.