python37Packages.tensorflow-bin_2: fix tensorflow-bin_2 on mac #116496

nbren12 · 2021-03-16T03:16:36Z

This package was broken on mac with some wildcarding issues.
Some more constraints in the tensorflow wheel needed to be relaxed.

This is my first PR, so I apologize if I am missing anything.

Motivation for this change

The package was broken, and the source builds don't work on Mac see e.g.: https://discourse.nixos.org/t/tensorflow-on-20-09-mac/11931/2.

I plan to backport this to 20.09.

Resolves #116364.
Closes #108614

Things done

I did import tensorflow in python and try some simple operations.

pkgs/development/python-modules/tensorflow/2/bin.nix

SuperSandro2000 · 2021-03-16T11:46:39Z

The only package that evals for me is nix-build -A python37Packages.tensorflow-bin_2. The others are disabled due to version constraints.

And the python37 package then fails to build with:

  ERROR: Could not find a version that satisfies the requirement tensorboard<2.2.0,>=2.1.0 (from tensorflow-cpu)
  ERROR: No matching distribution found for tensorboard<2.2.0,>=2.1

nbren12 · 2021-03-18T05:44:56Z

Thanks for the review @SuperSandro2000. I apologize for it not being cleaner. I was working off an old master before rebasing 285b260. It looks like this package is now broken on master (4307699) becase httplib2 has test failures.

nbren12 · 2021-03-18T07:29:05Z

I fixed the remaining version conflict issues and the ignored the failing network related tests in the upstream packages.

These upstream packages mostly had some mocked local webserver tests. It is interesting that these were building on Linux, but not Mac. Does the linux sandboxing somehow allow for local network based tests?

nbren12 · 2021-03-18T07:41:01Z

Also let me know if I should split the commits into separate PRs. I was trying to minimize the red tape.

nixos-discourse · 2021-03-19T02:18:08Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/482

jonringer

otherwise LGTM

jonringer · 2021-03-19T06:30:46Z

pkgs/development/python-modules/google-auth-oauthlib/default.nix

@@ -28,6 +29,8 @@ buildPythonPackage rec {
    pytestCheckHook
  ];

+  pytestFlagsArray = lib.optionals stdenv.isDarwin ["-k 'not test_run_local_server'"];


Suggested change

pytestFlagsArray = lib.optionals stdenv.isDarwin ["-k 'not test_run_local_server'"];

disabledTests = lib.optionals stdenv.isDarwin [ "test_run_local_server" ];

I added this to the google-auth-oauthlib commit.

pkgs/development/python-modules/httplib2/default.nix

Most of the tests were failing on Mac OS, and tests were only recently enabled for this package.

nbren12 · 2021-03-19T15:02:31Z

Okay. I replied to the reviews. I modified the relevant commits with git rebase -i, not sure if this is how PRs are supposed to work on this repo.

SuperSandro2000 · 2021-03-20T19:47:27Z

Okay. I replied to the reviews. I modified the relevant commits with git rebase -i, not sure if this is how PRs are supposed to work on this repo.

Thats fine :)

SuperSandro2000

Did you try __darwinAllowLocalNetworking = true;?

pkgs/development/python-modules/google-auth/default.nix

SuperSandro2000 · 2021-03-21T00:14:41Z

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 116496 run on x86_64-linux 1

38 packages marked as broken and skipped:

python38Packages.arviz
python38Packages.dm-sonnet
python38Packages.graph_nets
python38Packages.optuna
python38Packages.pymc3
python38Packages.rl-coach
python38Packages.shap
python38Packages.skorch
python39Packages.dm-sonnet
python39Packages.google-cloud-asset
python39Packages.google-cloud-automl
python39Packages.google-cloud-bigquery
python39Packages.google-cloud-bigquery-datatransfer
python39Packages.google-cloud-container
python39Packages.google-cloud-dataproc
python39Packages.google-cloud-datastore
python39Packages.google-cloud-dlp
python39Packages.google-cloud-error-reporting
python39Packages.google-cloud-iam
python39Packages.google-cloud-iot
python39Packages.google-cloud-kms
python39Packages.google-cloud-language
python39Packages.google-cloud-monitoring
python39Packages.google-cloud-os-config
python39Packages.google-cloud-pubsub
python39Packages.google-cloud-redis
python39Packages.google-cloud-secret-manager
python39Packages.google-cloud-securitycenter
python39Packages.google-cloud-spanner
python39Packages.google-cloud-speech
python39Packages.google-cloud-storage
python39Packages.google-cloud-tasks
python39Packages.google-cloud-texttospeech
python39Packages.google-cloud-translate
python39Packages.google-cloud-vision
python39Packages.google-cloud-websecurityscanner
python39Packages.graph_nets
python39Packages.rl-coach

2 packages failed to build and already failed to build on hydra master:

python38Packages.pywick: log was empty
s3ql: log was empty

103 packages built:

beancount (python38Packages.beancount)
deja-dup
duplicity
duply
dvc-with-remotes
fava
gcalcli
kmymoney
ledger2beancount
lieer
patroni
python38Packages.beancount_docverif
python38Packages.chirpstack-api
python38Packages.cirq
python38Packages.goobook
python38Packages.google-api-core
python38Packages.google-api-python-client
python38Packages.google-auth
python38Packages.google-auth-httplib2
python38Packages.google-auth-oauthlib
python38Packages.google-cloud-access-context-manager
python38Packages.google-cloud-asset
python38Packages.google-cloud-automl
python38Packages.google-cloud-bigquery
python38Packages.google-cloud-bigquery-datatransfer
python38Packages.google-cloud-bigtable
python38Packages.google-cloud-container
python38Packages.google-cloud-core
python38Packages.google-cloud-dataproc
python38Packages.google-cloud-datastore
python38Packages.google-cloud-dlp
python38Packages.google-cloud-dns
python38Packages.google-cloud-error-reporting
python38Packages.google-cloud-firestore
python38Packages.google-cloud-iam
python38Packages.google-cloud-iot
python38Packages.google-cloud-kms
python38Packages.google-cloud-language
python38Packages.google-cloud-logging
python38Packages.google-cloud-monitoring
python38Packages.google-cloud-org-policy
python38Packages.google-cloud-os-config
python38Packages.google-cloud-pubsub
python38Packages.google-cloud-redis
python38Packages.google-cloud-resource-manager
python38Packages.google-cloud-runtimeconfig
python38Packages.google-cloud-secret-manager
python38Packages.google-cloud-securitycenter
python38Packages.google-cloud-spanner
python38Packages.google-cloud-speech
python38Packages.google-cloud-storage
python38Packages.google-cloud-tasks
python38Packages.google-cloud-testutils
python38Packages.google-cloud-texttospeech
python38Packages.google-cloud-trace
python38Packages.google-cloud-translate
python38Packages.google-cloud-videointelligence
python38Packages.google-cloud-vision
python38Packages.google-cloud-websecurityscanner
python38Packages.google-resumable-media
python38Packages.gspread
python38Packages.ignite
python38Packages.kubernetes
python38Packages.labelbox
python38Packages.oauthenticator
python38Packages.pydrive
python38Packages.rising
python38Packages.slicer
python38Packages.tensorboardx
python38Packages.test-tube
python38Packages.weboob
python39Packages.beancount
python39Packages.beancount_docverif
python39Packages.chirpstack-api
python39Packages.cirq
python39Packages.goobook
python39Packages.google-api-core
python39Packages.google-api-python-client
python39Packages.google-auth
python39Packages.google-auth-httplib2
python39Packages.google-auth-oauthlib
python39Packages.google-cloud-access-context-manager
python39Packages.google-cloud-bigtable
python39Packages.google-cloud-core
python39Packages.google-cloud-dns
python39Packages.google-cloud-firestore
python39Packages.google-cloud-logging
python39Packages.google-cloud-org-policy
python39Packages.google-cloud-resource-manager
python39Packages.google-cloud-runtimeconfig
python39Packages.google-cloud-testutils
python39Packages.google-cloud-trace
python39Packages.google-cloud-videointelligence
python39Packages.google-resumable-media
python39Packages.gspread
python39Packages.kubernetes
python39Packages.oauthenticator
python39Packages.pydrive
python39Packages.slicer
python39Packages.tensorboardx
python39Packages.test-tube
python39Packages.weboob
wal_e

nbren12 · 2021-03-21T05:40:35Z

Did you try __darwinAllowLocalNetworking = true;?

I did not. Not sure how to do that.

nbren12 · 2021-03-28T16:08:41Z

@SuperSandro2000 Does this PR require any more action from me?

nbren12 · 2021-04-04T07:34:07Z

@jonringer @SuperSandro2000 This fixes several important python packages on darwin, and your last review comments, which I addressed, were mostly of an aesthetic nature...I would’ve been okay with you manually applying those changes when I had your attention.

This is my first contribution so would appreciate some feedback. Would splitting this into multiple PRs have made it easier for you? Just wondering for future contributions...seems like “I made some small change and merged your PR” is pretty common in the guix mailing lists, but if each iteration in nixpkgs takes weeks before re-review it puts a lot of emphasis on simplifying the PRs I think.

nixos-discourse · 2021-04-05T20:41:47Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/python-package-rodeo/12282/14

nbren12 · 2021-04-05T20:42:26Z

I am pretty sure this resolves #108614

alexvorobiev · 2021-04-06T03:42:28Z

That is correct. I reported #108614, now after applying the change tensorflow-bin_2 installs on Linux again.

pkgs/development/python-modules/google-auth/default.nix

pkgs/development/python-modules/google-auth-oauthlib/default.nix

pkgs/development/python-modules/google-auth/default.nix

pkgs/development/python-modules/httplib2/default.nix

This disables some tests that using a mocked webserver.

These tests fail in a sandboxed build on mac

This package was broken on mac with some wildcarding issues. Some more constraints in the tensorflow wheel needed to be relaxed.

SuperSandro2000 · 2021-04-12T00:18:33Z

I did not. Not sure how to do that.

See

nixpkgs/pkgs/applications/networking/dyndns/dyndnsc/default.nix

Line 40 in 7248f89

__darwinAllowLocalNetworking = true;

for an example. Please use the search in the future because it can take some days until I have the time to answer questions.

nbren12 · 2021-04-12T01:59:33Z

Thanks. @SuperSandro2000. I was just confused bc you "approved" the PR and then asked a question w/o saying it was a requirement for merging. I didn't know what you wanted me to do nor why. I would've appreciated some reasoning for this since _darwinAllowLocalNetworking is not used in all python packages, is not documented online, and starts with a "_" which coming from python typically implies it is a private API to be avoided.

nbren12 requested review from FRidh and jonringer as code owners March 16, 2021 03:16

nbren12 commented Mar 16, 2021

View reviewed changes

pkgs/development/python-modules/tensorflow/2/bin.nix Outdated Show resolved Hide resolved

ofborg bot added 6.topic: python 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Mar 16, 2021

veprbl added the 2.status: merge conflict label Mar 16, 2021

nbren12 force-pushed the fix-tensorflow-bin_2-master branch from fa5110c to e30b872 Compare March 16, 2021 07:21

ofborg bot removed the 2.status: merge conflict label Mar 16, 2021

SuperSandro2000 changed the title ~~(tensorflow-bin_2): fix broken package~~ python3Packages.tensorflow-bin_2: fix broken package Mar 16, 2021

SuperSandro2000 changed the title ~~python3Packages.tensorflow-bin_2: fix broken package~~ python37Packages.tensorflow-bin_2: fix broken package Mar 16, 2021

SuperSandro2000 changed the title ~~python37Packages.tensorflow-bin_2: fix broken package~~ python37Packages.tensorflow-bin_2: soften dependency restraints Mar 16, 2021

SuperSandro2000 requested a review from Mic92 March 16, 2021 11:46

SuperSandro2000 marked this pull request as draft March 16, 2021 11:46

nbren12 force-pushed the fix-tensorflow-bin_2-master branch from e30b872 to e44be29 Compare March 18, 2021 07:25

nbren12 marked this pull request as ready for review March 18, 2021 07:26

ofborg bot requested review from SuperSandro2000, terlar and fabaff March 18, 2021 07:35

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 501-1000 10.rebuild-linux: 101-500 and removed 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Mar 18, 2021

jonringer reviewed Mar 19, 2021

View reviewed changes

python37Packages.httplib: disable tests on darwin

c2cb457

Most of the tests were failing on Mac OS, and tests were only recently enabled for this package.

nbren12 force-pushed the fix-tensorflow-bin_2-master branch from e44be29 to bf8b375 Compare March 19, 2021 14:59

SuperSandro2000 approved these changes Mar 20, 2021

View reviewed changes

pkgs/development/python-modules/google-auth/default.nix Outdated Show resolved Hide resolved

ofborg bot requested a review from SuperSandro2000 March 21, 2021 05:47

nbren12 requested a review from jonringer March 29, 2021 19:11

nbren12 changed the title ~~python37Packages.tensorflow-bin_2: soften dependency restraints~~ python37Packages.tensorflow-bin_2: fix tensorflow-bin_2 on mac Apr 4, 2021

nbren12 mentioned this pull request Apr 5, 2021

pythonPackages.tensorflow-bin_2 fails to install with the updated gast #108614

Closed

SuperSandro2000 reviewed Apr 8, 2021

View reviewed changes

ofborg bot requested a review from SuperSandro2000 April 8, 2021 07:48

nbren12 added 3 commits April 8, 2021 07:52

python37Packages.google-auth: disable network tests

db242e1

This disables some tests that using a mocked webserver.

python37Packages.google-auth-oauthlib: disable webserver tests

1e48ad0

These tests fail in a sandboxed build on mac

python37Packages.tensorflow-bin_2: soften additional versions

5542995

This package was broken on mac with some wildcarding issues. Some more constraints in the tensorflow wheel needed to be relaxed.

nbren12 force-pushed the fix-tensorflow-bin_2-master branch from 43c7cd7 to 5542995 Compare April 8, 2021 07:53

SuperSandro2000 merged commit 28f381b into NixOS:master Apr 22, 2021

nbren12 deleted the fix-tensorflow-bin_2-master branch April 22, 2021 03:39

nbren12 mentioned this pull request May 28, 2021

tensorflow-bin build fails #73703

Closed

Janik-Haag added the 12. first-time contribution label Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python37Packages.tensorflow-bin_2: fix tensorflow-bin_2 on mac #116496

python37Packages.tensorflow-bin_2: fix tensorflow-bin_2 on mac #116496

nbren12 commented Mar 16, 2021 •

edited by SuperSandro2000

Loading

SuperSandro2000 commented Mar 16, 2021

nbren12 commented Mar 18, 2021

nbren12 commented Mar 18, 2021

nbren12 commented Mar 18, 2021

nixos-discourse commented Mar 19, 2021

jonringer left a comment

jonringer Mar 19, 2021 •

edited by SuperSandro2000

Loading

nbren12 Mar 19, 2021

nbren12 commented Mar 19, 2021

SuperSandro2000 commented Mar 20, 2021

SuperSandro2000 left a comment

SuperSandro2000 commented Mar 21, 2021

nbren12 commented Mar 21, 2021

nbren12 commented Mar 28, 2021 •

edited

Loading

nbren12 commented Apr 4, 2021 •

edited

Loading

nixos-discourse commented Apr 5, 2021

nbren12 commented Apr 5, 2021

alexvorobiev commented Apr 6, 2021

SuperSandro2000 commented Apr 12, 2021

nbren12 commented Apr 12, 2021 •

edited

Loading

	pytestFlagsArray = lib.optionals stdenv.isDarwin ["-k 'not test_run_local_server'"];
	disabledTests = lib.optionals stdenv.isDarwin [ "test_run_local_server" ];

python37Packages.tensorflow-bin_2: fix tensorflow-bin_2 on mac #116496

python37Packages.tensorflow-bin_2: fix tensorflow-bin_2 on mac #116496

Conversation

nbren12 commented Mar 16, 2021 • edited by SuperSandro2000 Loading

Motivation for this change

Things done

SuperSandro2000 commented Mar 16, 2021

nbren12 commented Mar 18, 2021

nbren12 commented Mar 18, 2021

nbren12 commented Mar 18, 2021

nixos-discourse commented Mar 19, 2021

jonringer left a comment

Choose a reason for hiding this comment

jonringer Mar 19, 2021 • edited by SuperSandro2000 Loading

Choose a reason for hiding this comment

nbren12 Mar 19, 2021

Choose a reason for hiding this comment

nbren12 commented Mar 19, 2021

SuperSandro2000 commented Mar 20, 2021

SuperSandro2000 left a comment

Choose a reason for hiding this comment

SuperSandro2000 commented Mar 21, 2021

nbren12 commented Mar 21, 2021

nbren12 commented Mar 28, 2021 • edited Loading

nbren12 commented Apr 4, 2021 • edited Loading

nixos-discourse commented Apr 5, 2021

nbren12 commented Apr 5, 2021

alexvorobiev commented Apr 6, 2021

SuperSandro2000 commented Apr 12, 2021

nbren12 commented Apr 12, 2021 • edited Loading

nbren12 commented Mar 16, 2021 •

edited by SuperSandro2000

Loading

jonringer Mar 19, 2021 •

edited by SuperSandro2000

Loading

nbren12 commented Mar 28, 2021 •

edited

Loading

nbren12 commented Apr 4, 2021 •

edited

Loading

nbren12 commented Apr 12, 2021 •

edited

Loading