Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

minimal pkg pipeline #3152

Merged
merged 44 commits into from May 8, 2020
Merged

Conversation

guihao-liang
Copy link
Collaborator

@guihao-liang guihao-liang commented Apr 29, 2020

The successor of #3151. Sub-pr: #3171.

The minimal build wheel won't have dependencies listed below.

19 NON_MINIMAL_LIST = [
 20     "coremltools",
 21     "pandas",
 22     "tensorflow",
 23     "resampy",
 24     "scipy",
 25 ]

And the minimal pkg will run relevant tests, which are not considered as part of the toolkits components.

minimal_test=($(grep -L "toolkits" *.py))

The internal runner will run tests on minimal pkg with 2.7 (Linux) and 3.7 (OSX). The reason to do so is that minimal pkg is the subset of full pkg. As long as full pkg succeeds in all platforms we care about, we should be confident minimal pkg will succeed too.


update 07/05/2020

The feature shown below is from #3175. The package produced by this PR doesn't have any prompt for the user to install the full package. That is to say, this PR is now not related to #3175 now.

Update: 01/05/2020

in the minimal package, if you use lazy-import wrapped tensorflow,

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    tc._deps.tensorflow.get_module()
  File "/Users/guihaoliang/Work/guicreate-2/target/env36/lib/python3.6/site-packages/turicreate/_deps/minimal_import.py", line 307, in get_module
    self._load_module()
  File "/Users/guihaoliang/Work/guicreate-2/target/env36/lib/python3.6/site-packages/turicreate/_deps/minimal_import.py", line 326, in _load_module
    self._module = self._init_func(self._name)
  File "/Users/guihaoliang/Work/guicreate-2/target/env36/lib/python3.6/site-packages/turicreate/_deps/minimal_import.py", line 95, in default_init_func
    raise e
  File "/Users/guihaoliang/Work/guicreate-2/target/env36/lib/python3.6/site-packages/turicreate/_deps/minimal_import.py", line 72, in default_init_func
    return importlib.import_module(name)
  File "/Users/guihaoliang/Work/guicreate-2/target/env36/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tensorflow'. This is a minimal package for SFrame only, without tensorflow pinned as a dependency. You can try install all required packages by installingfull package by:
**pip install turicreate==6.2**

@guihao-liang guihao-liang added this to the 6.3 milestone Apr 29, 2020
@guihao-liang guihao-liang self-assigned this Apr 29, 2020
@guihao-liang
Copy link
Collaborator Author

@brtal This is a good place for you to get familiar with our pipelines. Feel free to review.

@guihao-liang guihao-liang requested a review from hoytak May 5, 2020 19:37
.gitlab-ci.yml Outdated Show resolved Hide resolved
.gitlab-ci.yml Outdated Show resolved Hide resolved
scripts/make_documentation.sh Outdated Show resolved Hide resolved
scripts/make_wheel.sh Outdated Show resolved Hide resolved
scripts/make_wheel.sh Show resolved Hide resolved
src/python/turicreate/test/test_models.py Outdated Show resolved Hide resolved
src/python/turicreate/test/test_recsys_api.py Outdated Show resolved Hide resolved
src/python/turicreate/test/test_regression.py Outdated Show resolved Hide resolved
scripts/test_wheel.sh Show resolved Hide resolved
src/python/turicreate/test/test_linear_regression.py Outdated Show resolved Hide resolved
@guihao-liang
Copy link
Collaborator Author

adding minimal_packge.py is aimed to make this PR standalone so that this PR can produce minimal pkg without any feature relying on #3175. All the variable-, package-, and function- names are self-explanatory in minimal_package.py and _deps/__init__.py.

@guihao-liang guihao-liang requested review from hoytak and brtal May 7, 2020 20:09
@guihao-liang guihao-liang merged commit d862948 into apple:master May 8, 2020
@guihao-liang guihao-liang added this to Done in 6.3 via automation May 8, 2020
@guihao-liang
Copy link
Collaborator Author

passed internally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
6.3
  
Done
Development

Successfully merging this pull request may close these issues.

None yet

3 participants