Skip to content

Commit

Permalink
Resolve issue #2018 (#2022)
Browse files Browse the repository at this point in the history
* Issue with TF

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Comment out the PR gate affected tests with the upgrade to TF>2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Comment out the nightly builds affected tests with the upgrade to TF>2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Comment out the nightly builds affected tests with the upgrade to TF>2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* revert the breaking tests with TF 2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* temporary pin to TF=2.8.4

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update security tests

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Try to resolve #2018

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Exclude tensorflow versions that are not supported

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Correct version comparison using packaging.version.Version

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Capture importerror

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Restrict tensorflow < 2.13

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Set tensorflow < 2.12

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Not triggering unit tests on Draft PR (#2033)

* Not triggering unit tests on Draft PR

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Change a PR-triggering file to test

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

---------

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Refactor ranking metric `map` to be the same as Spark's (#2004)

* Announcement LF

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update email

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update README.md

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* security

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* license and contribution notice

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* update author link

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Add new code of conduct from LF

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Replacing references GRU4Rec to GRU

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Replacing references GRU4Rec to GRU

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Replacing references GRU4Rec in config files

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update references

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Delete conda.md

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* refactor map_at_k and map to be the same as Spark's

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* list of test failing to fix

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update readme LF feedback @wutaomsft

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update NEWS.md

Co-authored-by: Andreas Argyriou <anargyri@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update README.md

Co-authored-by: Andreas Argyriou <anargyri@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Fix test errors, Refactor column check utils to be simpler

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Rename ranking tests to be _at_k suffixed

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Change test names in the test group

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* add comment to mocked fn in a test

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* remove unused input

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* no need to output the logs twice

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* packages

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* skipping flaky test

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Issue with TF

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Comment out the PR gate affected tests with the upgrade to TF>2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Comment out the nightly builds affected tests with the upgrade to TF>2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Comment out the nightly builds affected tests with the upgrade to TF>2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* revert the breaking tests with TF 2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* temporary pin to TF=2.8.4

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update security tests

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update expected values to not use fixture

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* list of test failing to fix

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Fix missing fixture error

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

---------

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>
Co-authored-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Co-authored-by: Andreas Argyriou <anargyri@users.noreply.github.com>
Co-authored-by: Miguel Fierro <3491412+miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Add missing kernelspec language

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Remove scrapbook and papermill deps

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* notebook utils programmatic execution

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Test notebook programmatic

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Added test notebook for utils

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* data notebooks

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Replace papermill and scrapbook for new internal function

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Replace papermill and scrapbook for new internal function

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update new programmatic execution code

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update new programmatic execution code

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update notebooks with new utility

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Issue with xDeepFM WIP

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Document the tests in programmatic notebook

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* WIP

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* WIP

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Import missing store_metadata

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Correct pattern matching and substitution

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Merge multiline parameters into one line

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Increase timeout

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Fix nightly test errors (#2045)

* Revert tests tolerance
* Fix notebook parameter parsing
* Add notebook utils tests to test groups
* Fix notebooks
* Fix notebook unit tests
* Update evaluation metrics name map. Handle None for exp_var
* Fix smoke tests
* cleanup
* Fix functional test errors
* make notebook parameter update function to be private
* Fix benchmark notebook bug
* fix remaining bugs
---------

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Fix benchmarks last cell to store value, not [value]

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝 remove papermill and scrapbook references

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝 remove papermill and scrapbook references

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝 remove papermill and scrapbook references

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝 remove papermill and scrapbook references

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝 remove papermill and scrapbook references

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Updated PR template

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Updated contributing

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Updated PR template and contributing

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Updated contributing

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* [Fix] correct MIND data construction of user behavior history

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* change path hybrid

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update hybrid to CF

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* change path hybrid

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* change path hybrid

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Replace LayerRNNCell with AbstractRNNCell

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Stop testing for deeprec

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Refactor ranking metric `map` to be the same as Spark's (#2004)

* Announcement LF

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update email

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update README.md

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* security

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* license and contribution notice

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* update author link

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Add new code of conduct from LF

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Replacing references GRU4Rec to GRU

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Replacing references GRU4Rec to GRU

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Replacing references GRU4Rec in config files

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update references

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Delete conda.md

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* refactor map_at_k and map to be the same as Spark's

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* list of test failing to fix

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update readme LF feedback @wutaomsft

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update NEWS.md

Co-authored-by: Andreas Argyriou <anargyri@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update README.md

Co-authored-by: Andreas Argyriou <anargyri@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Fix test errors, Refactor column check utils to be simpler

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Rename ranking tests to be _at_k suffixed

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Change test names in the test group

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* add comment to mocked fn in a test

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* remove unused input

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* no need to output the logs twice

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* packages

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* skipping flaky test

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Issue with TF

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Comment out the PR gate affected tests with the upgrade to TF>2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Comment out the nightly builds affected tests with the upgrade to TF>2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Comment out the nightly builds affected tests with the upgrade to TF>2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* revert the breaking tests with TF 2.10.1

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* temporary pin to TF=2.8.4

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update security tests

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Update expected values to not use fixture

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* list of test failing to fix

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Fix missing fixture error

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

---------

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>
Co-authored-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Co-authored-by: Andreas Argyriou <anargyri@users.noreply.github.com>
Co-authored-by: Miguel Fierro <3491412+miguelgfierro@users.noreply.github.com>

* notebook utils programmatic execution

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>

* Test notebook programmatic

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>

* Added test notebook for utils

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>

* Replace papermill and scrapbook for new internal function

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>

* Replace papermill and scrapbook for new internal function

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update new programmatic execution code

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>

* Update new programmatic execution code

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>

* Update notebooks with new utility

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Issue with xDeepFM WIP

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Document the tests in programmatic notebook

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* WIP

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* WIP

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Import missing store_metadata

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Correct pattern matching and substitution

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Increase timeout

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Fix nightly test errors (#2045)

* Revert tests tolerance
* Fix notebook parameter parsing
* Add notebook utils tests to test groups
* Fix notebooks
* Fix notebook unit tests
* Update evaluation metrics name map. Handle None for exp_var
* Fix smoke tests
* cleanup
* Fix functional test errors
* make notebook parameter update function to be private
* Fix benchmark notebook bug
* fix remaining bugs
---------

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Fix benchmarks last cell to store value, not [value]

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝 remove papermill and scrapbook references

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝 remove papermill and scrapbook references

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝 remove papermill and scrapbook references

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Updated PR template

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>

* Updated contributing

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>

* Updated PR template and contributing

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Updated contributing

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* change path hybrid

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>

* change path hybrid

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Creating a jupyter book

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Creating documentation

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* WIP

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Added rst files

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* license

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Weird warning with a link in the docstrings

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Fix docstring errors and replace .. note:: with Note:

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Automatic build of documentation

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Automatic build of documentation dev

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Automatic build of documentation deps

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Automatic build of documentation deps

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Automatic build of documentation deps

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Delete workflow and try via UI

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Added again the workflow

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* git add * -rf

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* git add * -f

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* add git info

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* actions to automatically update documentation

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* actions to automatically update documentation

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* actions to automatically update documentation 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* actions to automatically update documentation 🐛

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* trying github token

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* trying github token

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* trying github token and pull before pushing

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* pull rebase

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* pull rebase and -Xtheirs

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* clean

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update documentation badge

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* install all deps

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* try adding other sphinx extensions

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Refact model rst

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* comment geoimc and rlrmc docs until issue is fixed

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* 📝

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Adding init and other special members

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Adding init and other special members

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Reviewing other rst

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Change sphinx version

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Change sphinx version and jupyter book

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Change the way we compile the documentation

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Using the latest JB release

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Documentation working

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update docs/_config.yml

Co-authored-by: Simon Zhao <simonyansenzhao@gmail.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update docs/requirements-doc.txt

Co-authored-by: Simon Zhao <simonyansenzhao@gmail.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update docs/_config.yml

Co-authored-by: Simon Zhao <simonyansenzhao@gmail.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Added comments by @SimonYansenZhao

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Upgrade versions of GitHub Actions

See https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update setup.py

Co-authored-by: Miguel Fierro <3491412+miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Try to disable sum and sum_component only

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Upgrade AzureML docker image

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Correct variable names

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Install git in the Conda env

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Disable test_xdeepfm_component_definition

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Use latest CUDA

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Correct GPU selection

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Remove leading whitespaces in Dockerfile

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Simplify azureml-test/action.yml

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Install wget in Docker image

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

* Update

Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>

---------

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: Simon Zhao <simonyansenzhao@gmail.com>
Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>
Co-authored-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Co-authored-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>
Co-authored-by: Andreas Argyriou <anargyri@users.noreply.github.com>
Co-authored-by: Miguel Fierro <3491412+miguelgfierro@users.noreply.github.com>
Co-authored-by: thaiminhpv <thaiminhpv@gmail.com>
  • Loading branch information
6 people committed Feb 19, 2024
1 parent 4679844 commit c736241
Show file tree
Hide file tree
Showing 13 changed files with 92 additions and 88 deletions.
49 changes: 19 additions & 30 deletions .github/actions/azureml-test/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ runs:
using: "composite"
steps:
- name: Setup python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.8"
- name: Install azureml-core and azure-cli on a GitHub hosted server
Expand All @@ -82,43 +82,32 @@ runs:
- name: Install wheel package
shell: bash
run: pip install --quiet wheel
- name: Submit CPU tests to AzureML
- name: Submit tests to AzureML
shell: bash
if: contains(inputs.TEST_GROUP, 'cpu')
run: >-
python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py --clustername ${{inputs.CPU_CLUSTER_NAME}}
--subid ${{inputs.AZUREML_TEST_SUBID}} --reponame "recommenders" --branch ${{ github.ref }}
--rg ${{inputs.RG}} --wsname ${{inputs.WS}} --expname ${{inputs.EXP_NAME}}_${{inputs.TEST_GROUP}}
--testlogs ${{inputs.TEST_LOGS_PATH}} --testkind ${{inputs.TEST_KIND}}
--conda_pkg_python ${{inputs.PYTHON_VERSION}} --testgroup ${{inputs.TEST_GROUP}}
--disable-warnings --sha "${GITHUB_SHA}"
- name: Submit GPU tests to AzureML
shell: bash
if: contains(inputs.TEST_GROUP, 'gpu')
run: >-
python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py --clustername ${{inputs.GPU_CLUSTER_NAME}}
--subid ${{inputs.AZUREML_TEST_SUBID}} --reponame "recommenders" --branch ${{ github.ref }}
--rg ${{inputs.RG}} --wsname ${{inputs.WS}} --expname ${{inputs.EXP_NAME}}_${{inputs.TEST_GROUP}}
--testlogs ${{inputs.TEST_LOGS_PATH}} --add_gpu_dependencies --testkind ${{inputs.TEST_KIND}}
--conda_pkg_python ${{inputs.PYTHON_VERSION}} --testgroup ${{inputs.TEST_GROUP}}
--disable-warnings --sha "${GITHUB_SHA}"
- name: Submit PySpark tests to AzureML
shell: bash
if: contains(inputs.TEST_GROUP, 'spark')
run: >-
python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py --clustername ${{inputs.CPU_CLUSTER_NAME}}
--subid ${{inputs.AZUREML_TEST_SUBID}} --reponame "recommenders" --branch ${{ github.ref }}
--rg ${{inputs.RG}} --wsname ${{inputs.WS}} --expname ${{inputs.EXP_NAME}}_${{inputs.TEST_GROUP}}
--testlogs ${{inputs.TEST_LOGS_PATH}} --add_spark_dependencies --testkind ${{inputs.TEST_KIND}}
--conda_pkg_python ${{inputs.PYTHON_VERSION}} --testgroup ${{inputs.TEST_GROUP}}
--disable-warnings --sha "${GITHUB_SHA}"
python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py \
--subid ${{inputs.AZUREML_TEST_SUBID}} \
--reponame "recommenders" \
--branch ${{ github.ref }} \
--rg ${{inputs.RG}} \
--wsname ${{inputs.WS}} \
--expname ${{inputs.EXP_NAME}}_${{inputs.TEST_GROUP}} \
--testlogs ${{inputs.TEST_LOGS_PATH}} \
--testkind ${{inputs.TEST_KIND}} \
--conda_pkg_python ${{inputs.PYTHON_VERSION}} \
--testgroup ${{inputs.TEST_GROUP}} \
--disable-warnings \
--sha "${GITHUB_SHA}" \
--clustername $(if [[ ${{inputs.TEST_GROUP}} =~ "gpu" ]]; then echo "${{inputs.GPU_CLUSTER_NAME}}"; else echo "${{inputs.CPU_CLUSTER_NAME}}"; fi) \
$(if [[ ${{inputs.TEST_GROUP}} =~ "gpu" ]]; then echo "--add_gpu_dependencies"; fi) \
$(if [[ ${{inputs.TEST_GROUP}} =~ "spark" ]]; then echo "--add_spark_dependencies"; fi)
- name: Get exit status
shell: bash
id: exit_status
run: echo "code=$(cat ${{inputs.PYTEST_EXIT_CODE}})" >> $GITHUB_OUTPUT
- name: Check Success/Failure
if: ${{ steps.exit_status.outputs.code != 0 }}
uses: actions/github-script@v3
uses: actions/github-script@v7
with:
script: |
core.setFailed('All tests did not pass!')
4 changes: 2 additions & 2 deletions .github/workflows/azureml-cpu-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Get test group names
id: get_test_groups
uses: ./.github/actions/get-test-groups
Expand All @@ -71,7 +71,7 @@ jobs:
test-group: ${{ fromJSON(needs.get-test-groups.outputs.test_groups) }}
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Execute tests
uses: ./.github/actions/azureml-test
id: execute_tests
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/azureml-gpu-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Get test group names
id: get_test_groups
uses: ./.github/actions/get-test-groups
Expand All @@ -71,7 +71,7 @@ jobs:
test-group: ${{ fromJSON(needs.get-test-groups.outputs.test_groups) }}
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Execute tests
uses: ./.github/actions/azureml-test
id: execute_tests
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/azureml-release-pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ jobs:
needs: [unit-test-workflow, cpu-nightly-workflow, gpu-nightly-workflow, spark-nightly-workflow]
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Setup python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.8"
- name: Install wheel package
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/azureml-spark-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Get test group names
id: get_test_groups
uses: ./.github/actions/get-test-groups
Expand All @@ -70,7 +70,7 @@ jobs:
test-group: ${{ fromJSON(needs.get-test-groups.outputs.test_groups) }}
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Execute tests
uses: ./.github/actions/azureml-test
id: execute_tests
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/azureml-unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Get test group names
id: get_test_groups
uses: ./.github/actions/get-test-groups
Expand All @@ -60,7 +60,7 @@ jobs:
test-group: ${{ fromJSON(needs.get-test-groups.outputs.test_groups) }}
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Execute tests
uses: ./.github/actions/azureml-test
id: execute_tests
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/sarplus.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,10 @@ jobs:
matrix:
python-version: ["3.8", "3.9"]
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

Expand Down Expand Up @@ -96,15 +96,15 @@ jobs:
- name: Upload Python wheel as GitHub artifact when merged into main
# Upload the whl file of the specific python version
if: github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: pysarplus-${{ env.sarplus_version }}-cp${{ matrix.python-version }}-wheel
path: ${{ env.PYTHON_ROOT }}/dist/*.whl

- name: Upload Python source as GitHub artifact when merged into main
# Only one pysarplus source tar file is needed
if: github.ref == 'refs/heads/main' && matrix.python-version == '3.10'
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: pysarplus-${{ env.sarplus_version }}-source
path: ${{ env.PYTHON_ROOT }}/dist/*.tar.gz
Expand All @@ -131,7 +131,7 @@ jobs:
hadoop-version: "3.3.1"

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Test
run: |
Expand Down Expand Up @@ -180,7 +180,7 @@ jobs:
- name: Upload Scala bundle as GitHub artifact when merged into main
if: github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: sarplus-${{ env.sarplus_version }}-bundle_2.12-spark-${{ matrix.spark-version }}-jar
path: ${{ env.SCALA_ROOT }}/target/scala-2.12/*bundle*.jar
4 changes: 2 additions & 2 deletions .github/workflows/update_documentation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ jobs:

steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.10"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -601,9 +601,9 @@ def __init__(
):
self._build_bias = build_bias

if args is None or (nest.is_sequence(args) and not args):
if args is None or (nest.is_nested(args) and not args):
raise ValueError("`args` must be specified")
if not nest.is_sequence(args):
if not nest.is_nested(args):
args = [args]
self._is_sequence = False
else:
Expand Down
3 changes: 2 additions & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@
extras_require = {
"gpu": [
"nvidia-ml-py3>=7.352.0",
"tensorflow==2.8.4", # FIXME: Temporarily pinned due to issue with TF version > 2.10.1 See #2018
"tensorflow>=2.8.4,!=2.9.0.*,!=2.9.1,!=2.9.2,!=2.10.0.*,<3",
"tf-slim>=1.1.0",
"torch>=1.13.1", # for CUDA 11 support
"fastai>=1.0.46,<2",
Expand All @@ -73,6 +73,7 @@
"pytest>=3.6.4",
"pytest-cov>=2.12.1",
"pytest-mock>=3.6.1", # for access to mock fixtures in pytest
"packaging>=20.9", # for version comparison in test_dependency_security.py
],
}
# For the brave of heart
Expand Down
56 changes: 34 additions & 22 deletions tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@
"""
import argparse
import logging
import glob

from azureml.core.authentication import AzureCliAuthentication
from azureml.core import Workspace
Expand Down Expand Up @@ -146,7 +145,6 @@ def setup_persistent_compute_target(workspace, cluster_name, vm_size, max_nodes)

def create_run_config(
cpu_cluster,
docker_proc_type,
add_gpu_dependencies,
add_spark_dependencies,
conda_pkg_jdk,
Expand All @@ -165,7 +163,6 @@ def create_run_config(
the following:
- Reco_cpu_test
- Reco_gpu_test
docker_proc_type (str) : processor type, cpu or gpu
add_gpu_dependencies (bool) : True if gpu packages should be
added to the conda environment, else False
add_spark_dependencies (bool) : True if PySpark packages should be
Expand All @@ -179,7 +176,39 @@ def create_run_config(
run_azuremlcompute = RunConfiguration()
run_azuremlcompute.target = cpu_cluster
run_azuremlcompute.environment.docker.enabled = True
run_azuremlcompute.environment.docker.base_image = docker_proc_type
if not add_gpu_dependencies:
# https://github.com/Azure/AzureML-Containers/blob/master/base/cpu/openmpi4.1.0-ubuntu22.04
run_azuremlcompute.environment.docker.base_image = "mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04"
else:
run_azuremlcompute.environment.docker.base_image = None
# Use the latest CUDA
# See
# * https://learn.microsoft.com/en-us/azure/machine-learning/how-to-train-with-custom-image?view=azureml-api-1#use-a-custom-dockerfile-optional
# * https://github.com/Azure/AzureML-Containers/blob/master/base/gpu/openmpi4.1.0-cuda11.8-cudnn8-ubuntu22.04
run_azuremlcompute.environment.docker.base_dockerfile = r"""
FROM nvcr.io/nvidia/cuda:12.3.1-devel-ubuntu22.04
USER root:root
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && \
apt-get install -y wget git-all && \
apt-get clean -y && \
rm -rf /var/lib/apt/lists/*
# Conda Environment
ENV MINICONDA_VERSION py38_23.3.1-0
ENV PATH /opt/miniconda/bin:$PATH
ENV CONDA_PACKAGE 23.5.0
RUN wget -qO /tmp/miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh && \
bash /tmp/miniconda.sh -bf -p /opt/miniconda && \
conda install conda=${CONDA_PACKAGE} -y && \
conda update --all -c conda-forge -y && \
conda clean -ay && \
rm -rf /opt/miniconda/pkgs && \
rm /tmp/miniconda.sh && \
find / -type d -name __pycache__ | xargs rm -rf
"""

# Use conda_dependencies.yml to create a conda environment in
# the Docker image for execution
Expand All @@ -195,6 +224,7 @@ def create_run_config(

# install recommenders
reco_extras = "dev"
conda_dep.add_conda_package("anaconda::git")
if add_gpu_dependencies and add_spark_dependencies:
conda_dep.add_channel("conda-forge")
conda_dep.add_conda_package(conda_pkg_jdk)
Expand Down Expand Up @@ -326,13 +356,6 @@ def create_arg_parser():
default="STANDARD_D3_V2",
help="Set the size of the VM either STANDARD_D3_V2",
)
# cpu or gpu
parser.add_argument(
"--dockerproc",
action="store",
default="cpu",
help="Base image used in docker container",
)
# Azure subscription id, when used in a pipeline, it is stored in keyvault
parser.add_argument(
"--subid", action="store", default="123456", help="Azure Subscription ID"
Expand Down Expand Up @@ -421,16 +444,6 @@ def create_arg_parser():

logger = logging.getLogger("submit_groupwise_azureml_pytest.py")
args = create_arg_parser()

if args.dockerproc == "cpu":
from azureml.core.runconfig import DEFAULT_CPU_IMAGE

docker_proc_type = DEFAULT_CPU_IMAGE
else:
from azureml.core.runconfig import DEFAULT_GPU_IMAGE

docker_proc_type = DEFAULT_GPU_IMAGE

cli_auth = AzureCliAuthentication()

workspace = setup_workspace(
Expand All @@ -450,7 +463,6 @@ def create_arg_parser():

run_config = create_run_config(
cpu_cluster=cpu_cluster,
docker_proc_type=docker_proc_type,
add_gpu_dependencies=args.add_gpu_dependencies,
add_spark_dependencies=args.add_spark_dependencies,
conda_pkg_jdk=args.conda_pkg_jdk,
Expand Down
Loading

0 comments on commit c736241

Please sign in to comment.