Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add zodbpickle to server_find_global. #109

Merged
merged 2 commits into from
Mar 27, 2018
Merged

Add zodbpickle to server_find_global. #109

merged 2 commits into from
Mar 27, 2018

Conversation

jamadden
Copy link
Member

Fixes #107

I also spent time trying to understand why so many of the tests are failing. I don't have answers for all of it, but I think this puts us in a better state.

  • Python 2 (both CPython and PyPy) is incredibly slow. The FileStorageTests, FileStorageSSLTests and BlobAdaptedFileStorageTests can take 8 to 15 minutes each. And we run them multiple times. Since Travis kills a test if there is no output in 10 minutes, Python 2 was just getting forcibly shutdown. Remedy: increase verbosity so we get some output. Now, they still fail due to some threads being left behind, but at least they're not terminated. And we can see that the 'Connection' errors that were prevelant in the logs for UnboundLocalError: local variable #107 aren't there anymore. I'm not sure why they're so slow---the Python 2 tests with msgpack aren't.

  • I can't directly replicate what Travis is doing: running tests in parallel on macOS results in a number of interpreter crashes (CPython 2.7.14 and 3.6.4):

crashed on child side of fork pre-exec
objc[87059]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
...
32  com.apple.CoreFoundation      	0x00007fff413e7473 CFPreferencesCopyAppValue + 99
33  com.apple.SystemConfiguration 	0x00007fff4d74c665 SCDynamicStoreCopyProxiesWithOptions + 155
34  _scproxy.so                   	0x000000010f78eade get_proxies + 14
35  org.python.python             	0x000000010eb76234 PyEval_EvalFrameEx + 16452
...
  • Also on macOS there are lots of failures like:
Error in test checkUndoZombie (ZEO.tests.testZEO.MappingStorageTests)
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 320, in run
    self.setUp()
  File "//ZEO/src/ZEO/tests/testZEO.py", line 244, in setUp
    StorageTestBase.StorageTestBase.setUp(self)
  File "//tmp-fab815c2b3ac8057/lib/python2.7/site-packages/ZODB/tests/util.py", line 87, in setUp
    setUp(self, name)
  File "//tmp-fab815c2b3ac8057/lib/python2.7/site-packages/ZODB/tests/util.py", line 66, in setUp
    d = tempfile.mkdtemp(prefix=name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tempfile.py", line 339, in mkdtemp
    _os.mkdir(file, 0700)
OSError: [Errno 63] File name too long: '/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/MappingStorageTestsWRZQtx/MappingStorageTestsOMvi3R/MappingStorageTestsAggQeU/MappingStorageTestsnrdFbw/MappingStorageTestsqN8xC8/MappingStorageTestsKiTxqI/MappingStorageTestszQZKB9/MappingStorageTestsqxGpnu/MappingStorageTests6lam8T/MappingStorageTestsN_4o4a/MappingStorageTestsOvsVMt/MappingStorageTests4pTysv/MappingStorageTestsV_aVcc/MappingStorageTestscVpkV7/MappingStorageTestswJTEJk/MappingStorageTestscphueF/MappingStorageTestsHYlPAR/MappingStorageTestsB2tuzg/MappingStorageTestsPN0Qkl/MappingStorageTestswgEWEY/MappingStorageTests6KM3mK/MappingStorageTestsK9XZoO/MappingStorageTestsL7QmGo/MappingStorageTestsdY0I2k/MappingStorageTestsTmZ8p2/MappingStorageTestshUymOQ/MappingStorageTestsVpLLIx/MappingStorageTests5ErtfQ/MappingStorageTests3zhjoQ/MappingStorageTestsJj3GiN/MappingStorageTestsbQyNXm/MappingStorageTestsAaPhk2/MappingStorageTestslegIUU/MappingStorageTestsO8JTJK/MappingStorageTestsvb9_gL/MappingStorageTestszA0o91/MappingStorageTestsnoVr5S/MappingStorageTestsS3e1N1'
  • It seems like we may be running way more tests on Python 2 than the other versions? I can't confirm that yet.

  • The ZEO4_SERVER tests still get terminated on Python 2. I don't know why there's no output.

  • The ZEO_MTACCEPTER tests fail on all tested versions of Python. These appear to be legit failures.

  • There were so many warnings about deprecations and unclosed files it was difficult to find the actual test failures, so I cleaned a large number of those up.

… travis

Some of the tests are being killed because we don't produce any output
when running parallel tests in a layer with -v1.

The amount of warnings made it hard to see real failures.

Also use more recent PyPy, and test on Py 3.6 (which produces the
'async' warnings we'll use to fix #104)
Copy link
Member

@jimfulton jimfulton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

I don't recall the tests being in such bad shape. :/

@jimfulton
Copy link
Member

Do you think we should reverse the ZODB change until this is sorted out?

@jamadden
Copy link
Member Author

I'm 90%+ confident that this change will fix the issue with ZODB 5.4. Reviewing the build history, I don't think that change is related to the extra failures, and we're probably safe to release as-is.

I may try to poke at the tests a bit more on travis and if I find anything I'll report back.

@jamadden jamadden merged commit 19db5a3 into master Mar 27, 2018
@jamadden jamadden deleted the issue107 branch March 27, 2018 13:45
@jimfulton
Copy link
Member

OK, shall I make a ZEO release?

@jamadden
Copy link
Member Author

I think #104 is low-hanging fruit, and possibly #106 and #105. I'd like to take a look at those and maybe get fixes in before a release, if that's OK. I'll try to look at those and open PRs within the next hour or two.

@jimfulton
Copy link
Member

Given that the ZODB changes break ZEO, I'd rather get the fixes out ASAP. I'm happy to release more tomorrow, or as soon as they're ready.

I'll go ahead and make a ZEO release, unless you think the ZEO changes could have broken something.

@jamadden
Copy link
Member Author

No, I don't think the ZEO change here could break anything (well, ensure_future requires Python 3.4.4 or above, so it would "break" on older point releases of Python 3.4). I'm happy with a release if you are!

@jamadden
Copy link
Member Author

FWIW, it's one test, checkPackAfterUndoManyTimes that's insanely slow. I've been staring at it for 10 minutes.

@jimfulton
Copy link
Member

5.1.2 released.

@jamadden
Copy link
Member Author

Ah, debugging this it looks like the hangs are related to #113

@jamadden
Copy link
Member Author

(Using the python.org installers instead of custom macports Python installers fixed the interpreter fork crash.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

UnboundLocalError: local variable
2 participants