python3Packages.s3transfer: enable tests

[fabaff]

Motivation for this change

Enable tests

Related #154264

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.05 Release Notes (or backporting 21.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[fabaff]

Result of nixpkgs-review pr 154280 run on x86_64-linux 1

1 package marked as broken and skipped:

• mlflow-server

51 packages failed to build:

• aws-sam-cli
• awslimitchecker
• home-assistant
• lexicon
• mapproxy
• python310Packages.buildbot
• python310Packages.buildbot-full
• python310Packages.buildbot-ui
• python310Packages.cartopy
• python310Packages.celery
• python310Packages.certbot-dns-route53
• python310Packages.cloudsplaining
• python310Packages.dependency-injector
• python310Packages.django-raster
• python310Packages.django-rq
• python310Packages.djmail
• python310Packages.fiona
• python310Packages.flower
• python310Packages.flowlogs_reader
• python310Packages.gensim
• python310Packages.geopandas
• python310Packages.gradient
• python310Packages.hass-nabucasa
• python310Packages.mapbox
• python310Packages.moto
• python310Packages.osmnx
• python310Packages.parsedmarc
• python310Packages.pathy
• python310Packages.pyhiveapi
• python310Packages.sentry-sdk
• python310Packages.serverlessrepo
• python310Packages.slack-sdk
• python310Packages.slicedimage
• python310Packages.smart-open
• python310Packages.snowflake-connector-python
• python310Packages.snowflake-sqlalchemy
• python310Packages.spacy
• python310Packages.spacy-transformers
• python310Packages.tensorboardx
• python310Packages.textacy
• python39Packages.cartopy
• python39Packages.geopandas
• python39Packages.osmnx
• python39Packages.serverlessrepo
• python39Packages.snowflake-connector-python
• python39Packages.snowflake-sqlalchemy
• python39Packages.spacy
• python39Packages.spacy-transformers
• python39Packages.textacy
• sourcehut.metasrht
• tts

102 packages built:

• alerta-server
• aws-google-auth
• aws-mfa
• awscli
• awscli2
• awslogs
• barman
• bash-my-aws
• buildbot (python39Packages.buildbot)
• buildbot-full (python39Packages.buildbot-full)
• buildbot-ui (python39Packages.buildbot-ui)
• certbot-full
• checkov
• cloud-custodian
• credstash (python39Packages.credstash)
• cura
• deja-dup
• duplicity
• duply
• dvc-with-remotes
• elasticsearch-curator
• git-remote-codecommit
• gns3-gui
• gns3-server
• iceshelf
• luigi
• moodle-dl
• moz-phab
• nimbo
• ntfy
• octoprint
• parquet-tools
• python310Packages.aws-sam-translator
• python310Packages.boto3
• python310Packages.cfn-lint
• python310Packages.credstash
• python310Packages.django-anymail
• python310Packages.ec2instanceconnectcli
• python310Packages.fs-s3fs
• python310Packages.mypy-boto3-builder
• python310Packages.mypy-boto3-s3
• python310Packages.pyathena
• python310Packages.pycognito
• python310Packages.rasterio
• python310Packages.s3transfer
• python310Packages.sagemaker
• python310Packages.sapi-python-client
• python310Packages.warrant
• python39Packages.aws-sam-translator
• python39Packages.boto3
• python39Packages.celery
• python39Packages.certbot-dns-route53
• python39Packages.cfn-lint
• python39Packages.cloudsplaining
• python39Packages.dependency-injector
• python39Packages.django-anymail
• python39Packages.django-raster
• python39Packages.django-rq
• python39Packages.djmail
• python39Packages.ec2instanceconnectcli
• python39Packages.fiona
• python39Packages.flower
• python39Packages.flowlogs_reader
• python39Packages.fs-s3fs
• python39Packages.gensim
• python39Packages.gradient
• python39Packages.hass-nabucasa
• python39Packages.mapbox
• python39Packages.moto
• python39Packages.mypy-boto3-builder
• python39Packages.mypy-boto3-s3
• python39Packages.parsedmarc
• python39Packages.pathy
• python39Packages.pyathena
• python39Packages.pycognito
• python39Packages.pyhiveapi
• python39Packages.rasterio
• python39Packages.s3transfer
• python39Packages.sagemaker
• python39Packages.sapi-python-client
• python39Packages.sentry-sdk
• python39Packages.slack-sdk
• python39Packages.slicedimage
• python39Packages.smart-open
• python39Packages.tensorboardx
• python39Packages.warrant
• s3bro
• s4cmd
• snakemake
• sourcehut.buildsrht
• sourcehut.coresrht
• sourcehut.dispatchsrht
• sourcehut.gitsrht
• sourcehut.hgsrht
• sourcehut.hubsrht
• sourcehut.listssrht
• sourcehut.mansrht
• sourcehut.pastesrht
• sourcehut.todosrht
• ssm-session-manager-plugin
• streamlit
• tribler

[lamdor]

@fabaff This seems to be failing for me on aarch64-darwin on MacOS. The test failure I'm getting is:

error: builder for '/nix/store/w5nwvfn71y90b2nm6cfgqv5q1kvwhpq9-python3.9-s3transfer-0.5.0.drv' failed with exit code 1;
       last 10 log lines:
       >     with self.assertRaisesRegexp(
       >
       > tests/unit/test_subscribers.py::TestSubscribers::test_not_callable_in_subclass_subscriber_method
       >   /private/tmp/nix-build-python3.9-s3transfer-0.5.0.drv-0/source/tests/unit/test_subscribers.py:81: DeprecationWarning: Please use assertRaisesRegex instead.
       >     with self.assertRaisesRegexp(
       >
       > -- Docs: https://docs.pytest.org/en/stable/warnings.html
       > =========================== short test summary info ============================
       > FAILED tests/unit/test_compat.py::TestBaseManager::test_can_provide_signal_handler_initializers_to_start
       > =========== 1 failed, 601 passed, 28 skipped, 44 warnings in 23.11s ============
       For full logs, run 'nix log /nix/store/w5nwvfn71y90b2nm6cfgqv5q1kvwhpq9-python3.9-s3transfer-0.5.0.drv'.

It fails for me consistently.

Here's the full build log s3transfer-build-log.txt.

Please let me know if I can help run a specific branch/test to help or if maybe tests/unit/test_compat.py needs added to disabledTestPaths? I'm sort of at a loss on where to start to fix it.

[duckpuppy]

@fabaff I'm getting the error on x86_64-darwin as well. The actual test failure is below:

____ TestBaseManager.test_can_provide_signal_handler_initializers_to_start _____

self = <tests.unit.test_compat.TestBaseManager testMethod=test_can_provide_signal_handler_initializers_to_start>

    @skip_if_windows('os.kill() with SIGINT not supported on Windows')
    def test_can_provide_signal_handler_initializers_to_start(self):
        manager = self.create_pid_manager()
>       manager.start(signal.signal, (signal.SIGINT, signal.SIG_IGN))

tests/unit/test_compat.py:100:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/managers.py:554: in start
    self._process.start()
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/context.py:284: in _Popen
    return Popen(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <SpawnProcess name='PIDManager-43' parent=65299 initial>
file = <_io.BytesIO object at 0x1131d6630>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'TestBaseManager.create_pid_manager.<locals>.PIDManager'

/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/reduction.py:60: AttributeError

[YorikSar]

I'm also affected by this.

@lamdor @duckpuppy Did you happen to open an issue about this?

[lamdor]

@YorikSar I haven't yet had time to look more into it and create an issue. If someone beats me to it, that would be great.

[fabaff]

It still fails (https://hydra.nixos.org/eval/1738910?filter=s3transfer&compare=1738843&full=#tabs-still-fail).

Looks like that a condition for tests is needed for Darwin.

[fabaff]

Perhaps somebody from @NixOS/darwin-maintainers could help.

[thefloweringash]

This passes for me on python 3.7, but fails on 3.8 and 3.9 (and probably any later version). The stack trace implicates multiprocessing. There was a change in python 3.8 that defaults multiprocessing to spawn instead of fork on macOS (bpo-33725, python/cpython#13603). I suspect the underlying issue here is that upstream tests aren't compatible with spawn multiprocessing, and pass on linux where the default is still fork.

[lamdor]

I submitted a PR which ignores that test_compat on darwin. I've verified that then it is able to build s3transfer without any issue: #155896