Merging all the codes of dev to main branch. #9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds some more definite types for our NLP tasks and tokenization configurations. This is the first step in allowing users to more easily import their own transformer models via something other than hugging face. Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Adds support for `question_answering` NLP models within the pytorch model uploader. Related: elastic/elasticsearch#85958 Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
This improves the user consumed functions and classes for PyTorch NLP model upload to Elasticsearch. Previously it was difficult to wrap your own module for uploading to Elasticsearch. This commit splits some classes out, adds new ones, and adds tests showing how to wrap some simple modules. Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
…es (#465) This switches our sklearn.DecisionTreeClassifier serialization logic to account for multi-valued leaves in the tree. The key difference between our inference and DecisionTreeClassifier, is that we run a softMax over the leaf where sklearn simply normalizes the results. This means that our "probabilities" returned will be different than sklearn. Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Co-authored-by: Seth Michael Larson <seth.larson@elastic.co> Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
… NLP task type from model config (#475) For many model types, we don't need to require the task requested. We can infer the task type based on the model configuration and architecture. This commit makes the `task-type` parameter optional for the model up load script and adds logic for auto-detecting the task type based on the 🤗 model. Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
…n elastic vs open search Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
…arning Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>
|
See several files contains "elastic" like this one |
ylwu-amzn
reviewed
Oct 19, 2022
| * Added support for XGBoost 1.6 (`#458`_) | ||
| * Added support for ``question_answering`` NLP tasks (`#457`_) | ||
|
|
||
| .. _#457: https://github.com/elastic/eland/pull/457 |
There was a problem hiding this comment.
Should we keep this change log file? Seems most content is about eland
ylwu-amzn
reviewed
Oct 19, 2022
| @@ -57,3 +57,100 @@ If you discover a potential security issue in this project we ask that you notif | |||
| ## Licensing | |||
|
|
|||
| See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. | |||
| **Repository:** <https://github.com/elastic/eland> | |||
There was a problem hiding this comment.
Change this part by referring to https://github.com/opensearch-project/opensearch-py/blob/main/CONTRIBUTING.md ?
ylwu-amzn
reviewed
Oct 19, 2022
| - Support for Docker | ||
| - Support for continuous integration | ||
| - Regenerating Sphinx docs | ||
| - Creating tutorials for `opensearch-py-ml` in both notebook and video form |
There was a problem hiding this comment.
How about follow the readme template of https://github.com/opensearch-project/opensearch-py, add "Code of Conduct", "Liscense" and "Copyright"?
There was a problem hiding this comment.
Yeah, I was planning to add those in the next PR.
ylwu-amzn
reviewed
Oct 19, 2022
| @@ -0,0 +1,2319 @@ | |||
| { | |||
There was a problem hiding this comment.
Not sure how to use this demo notebook for community user. Does it depend on some data set? How to prepare the environment to run this demo? I think explain this in some doc could help, like readme or some other doc
ylwu-amzn
reviewed
Oct 19, 2022
| @@ -0,0 +1,1213 @@ | |||
| { | |||
There was a problem hiding this comment.
The file name seems not intuitive, we have "demo.ipynb" and this "demo_notebook.ipynb". How about change to other name to reflect the things it want to demo?
There was a problem hiding this comment.
I'm planning to remove both of those in fact. And then add a detailed read me to follow.
ylwu-amzn
approved these changes
Oct 19, 2022
mingshl
approved these changes
Oct 19, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Merging all the codes of dev to main branch
Issues Resolved
Merging all the codes of dev to main branch
--signoff::dhrubo@amazon.com
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.