Merge branch 'master' into master

README.md

@@ -89,11 +89,15 @@ After it starts up, the container will run in the background, at http://127.0.0.
 You can test the RESTful API using `curl`:
 
 ```bash
-curl 'http://127.0.0.1:5000/yake/' \
--XPOST \
--H 'Accept: application/json' \
--H 'Content-Type: application/x-www-form-urlencoded' \
---data 'text=Coffee%20is%20a%20brewed%20drink%20prepared%20from%20roasted%20coffee%20beans%2C%20the%20seeds%20of%20berries%20from%20certain%20Coffea%20species.%20The%20genus%20Coffea%20is%20native%20to%20tropical%20Africa%20(specifically%20having%20its%20origin%20in%20Ethiopia%20and%20Sudan)%20and%20Madagascar%2C%20the%20Comoros%2C%20Mauritius%2C%20and%20R%C3%A9union%20in%20the%20Indian%20Ocean.%5B2%5D%20Coffee%20plants%20are%20now%20cultivated%20in%20over%2070%20countries%2C%20primarily%20in%20the%20equatorial%20regions%20of%20the%20Americas%2C%20Southeast%20Asia%2C%20Indian%20subcontinent%2C%20and%20Africa.%20The%20two%20most%20commonly%20grown%20are%20C.%20arabica%20and%20C.%20robusta.%20Once%20ripe%2C%20coffee%20berries%20are%20picked%2C%20processed%2C%20and%20dried.%20Dried%20coffee%20seeds%20(referred%20to%20as%20%22beans%22)%20are%20roasted%20to%20varying%20degrees%2C%20depending%20on%20the%20desired%20flavor.%20Roasted%20beans%20are%20ground%20and%20then%20brewed%20with%20near-boiling%20water%20to%20produce%20the%20beverage%20known%20as%20coffee.&language=en&max_ngram_size=4&number_of_keywords=10'
+curl -X POST "http://localhost:5000/yake/" -H "accept: application/json" -H "Content-Type: application/json" \
+-d @- <<'EOF'
+{
+  "language": "en",
+  "max_ngram_size": 2,
+  "number_of_keywords": 10,
+  "text": "Sources tell us that Google is acquiring Kaggle, a platform that hosts data science and machine learning competitions. Details about the transaction remain somewhat vague , but given that Google is hosting its Cloud Next conference in San Francisco this week, the official announcement could come as early as tomorrow. Reached by phone, Kaggle co-founder CEO Anthony Goldbloom declined to deny that the acquisition is happening. Google itself declined 'to comment on rumors'. Kaggle, which has about half a million data scientists on its platform, was founded by Goldbloom and Ben Hamner in 2010. The service got an early start and even though it has a few competitors like DrivenData, TopCoder and HackerRank, it has managed to stay well ahead of them by focusing on its specific niche. The service is basically the de facto home for running data science and machine learning competitions. With Kaggle, Google is buying one of the largest and most active communities for data scientists ..."
+}
+EOF
 ```
 *Example text from Wikipedia*
 
@@ -166,9 +170,6 @@ How to use it on Python
 
 ### Dockerfiles
 
-https://github.com/feup-infolab/yake-dockerfile - Dockerfile for building an image for this package.
-https://github.com/feup-infolab/yake-rest-dockerfile - Dockerfile for building an image of the RESTful API version of this package.
-
 Credits to https://github.com/silvae86
 
 ### `pke` - python keyphrase extraction

docker/Dockerfiles/yake-server/yake-rest-api.py

@@ -17,66 +17,123 @@
 from flask import Flask, jsonify, request
 
 from flasgger import Swagger
+
+try:
+    import simplejson as json
+except ImportError:
+    import json
+try:
+    from http import HTTPStatus
+except ImportError:
+    import httplib as HTTPStatus
+
 import yake
 
 app = Flask(__name__)
 app.config['SWAGGER'] = {
-    'title': 'Yake API sample'
+    'title': 'Yake API sample',
+    'uiversion': 3
 }
-Swagger(app)
+swagger = Swagger(app)
 
+@swagger.validate('content')
 @app.route('/yake/',methods=['POST'])
 def handle_yake():
     """Example endpoint return a list of keywords using YAKE
     ---
+    consumes:
+      - application/json
+    produces:
+      - application/json
     parameters:
-      - name: text
-        in: formData
-        type: string
-        description: text
-        required: true
-      - name: language
-        in: formData
-        type: string
-        description: language
-        required: true
-        default: "en"
-        enum: ["pt", "en", "es", "fr", "it", "de" ]
-      - name: max_ngram_size
-        in: formData
-        type: integer
-        description: max size of ngram
-        required: true
-        default: 4
-      - name: number_of_keywords
-        in: formData
-        type: integer
-        description: number of keywords to return
+        - name: content
+          in: body
+          description: content object
+          required: true
+          schema:
+            $ref: '#/definitions/content'
+
+    requestBody:
+        description: Optional description in *Markdown*
         required: true
-        default: 10
+        content:
+          application/json:
+            schema:
+              id: content
+              type: object
+
 
     responses:
       200:
         description: Extract keywords from input text
+        schema:
+            $ref: '#/definitions/result'
+
+    definitions:
+      content:
+        description: content object
+        properties:
+          text:
+            type: string
+          language:
+            type: string
+          max_ngram_size:
+            type: integer
+            minimum: 1
+          number_of_keywords:
+            type: integer
+            minimum: 1
+        required:
+          - text
+          - language
+          - max_ngram_size
+          - number_of_keywords
+        example:   # Sample object
+            text: Sources tell us that Google is acquiring Kaggle, a platform that hosts data science and machine learning   competitions. Details about the transaction remain somewhat vague , but given that Google is hosting   its Cloud Next conference in San Francisco this week, the official announcement could come as early   as tomorrow.  Reached by phone, Kaggle co-founder CEO Anthony Goldbloom declined to deny that the   acquisition is happening. Google itself declined 'to comment on rumors'.   Kaggle, which has about half a million data scientists on its platform, was founded by Goldbloom   and Ben Hamner in 2010. The service got an early start and even though it has a few competitors   like DrivenData, TopCoder and HackerRank, it has managed to stay well ahead of them by focusing on its   specific niche. The service is basically the de facto home for running data science  and machine learning   competitions.  With Kaggle, Google is buying one of the largest and most active communities for   data scientists - and with that, it will get increased mindshare in this community, too   (though it already has plenty of that thanks to Tensorflow and other projects).   Kaggle has a bit of a history with Google, too, but that's pretty recent. Earlier this month,   Google and Kaggle teamed up to host a $100,000 machine learning competition around classifying   YouTube videos. That competition had some deep integrations with the Google Cloud Platform, too.   Our understanding is that Google will keep the service running - likely under its current name.   While the acquisition is probably more about Kaggle's community than technology, Kaggle did build   some interesting tools for hosting its competition and 'kernels', too. On Kaggle, kernels are   basically the source code for analyzing data sets and developers can share this code on the   platform (the company previously called them 'scripts').  Like similar competition centric sites,   Kaggle also runs a job board, too. It's unclear what Google will do with that part of the service.   According to Crunchbase, Kaggle raised $12.5 million (though PitchBook says it's $12.75) since its   launch in 2010. Investors in Kaggle include Index Ventures, SV Angel, Max Levchin, Naval Ravikant,   Google chief economist Hal Varian, Khosla Ventures and Yuri Milner
+            language: en
+            max_ngram_size: 3
+            number_of_keywords: 10
+      result:
+        type: array
+        items:
+          minItems: 0
+          type: object
+          required:
+            - name
+            - value
+          properties:
+            ngram:
+              type: string
+            score:
+              type: number
     """
-    print(request.form)
-    text = request.form["text"]
-    language = request.form["language"]
-    max_ngram_size = int(request.form["max_ngram_size"])
-    number_of_keywords = int(request.form["number_of_keywords"])
 
-    my_yake = yake.KeywordExtractor(lan=language,
-                                    n=max_ngram_size,
-                                    top=number_of_keywords,
-                                    dedupLim=0.8,
-                                    windowsSize=2
-                                    )
+    try:
+        assert request.json["text"] , "Invalid text"
+        assert len(request.json["language"]) == 2, "Invalid language code"
+        assert int(request.json["max_ngram_size"]) , "Invalid max_ngram_size"
+        assert int(request.json["number_of_keywords"]) , "Invalid number_of_keywords"
+
+        text = request.json["text"]
+        language = request.json["language"]
+        max_ngram_size = int(request.json["max_ngram_size"])
+        number_of_keywords = int(request.json["number_of_keywords"])
 
-    keywords = my_yake.extract_keywords(text)
-    result  = [{"ngram":x[1] ,"score":x[0]} for x in keywords]
-    return jsonify(result)
+        my_yake = yake.KeywordExtractor(lan=language,
+                                        n=max_ngram_size,
+                                        top=number_of_keywords,
+                                        dedupLim=0.8,
+                                        windowsSize=2
+                                        )
 
+        keywords = my_yake.extract_keywords(text)
+        result  = [{"ngram":x[1] ,"score":x[0]} for x in keywords]
 
+        return jsonify(result), HTTPStatus.OK
+    except IOError as e:
+        return jsonify("Language not supported"), HTTPStatus.BAD_REQUEST
+    except Exception as e:
+        return jsonify(str(e)), HTTPStatus.BAD_REQUEST
 
 if __name__ == "__main__":
     app.run(host='0.0.0.0', debug=True)

docker/test_image_calls.sh

@@ -41,11 +41,15 @@ docker run -d -p $YAKE_PORT:$YAKE_PORT "$YAKE_SERVER_IMAGE:$TAG"
 
 wait_for_server_to_boot_on_port "127.0.0.1" "$YAKE_PORT" "<title>404 Not Found</title>"
 
-curl 'http://127.0.0.1:5000/yake/' \
--XPOST \
--H 'Accept: application/json' \
--H 'Content-Type: application/x-www-form-urlencoded' \
---data "text=Coffee%20is%20a%20brewed%20drink%20prepared%20from%20roasted%20coffee%20beans%2C%20the%20seeds%20of%20berries%20from%20certain%20Coffea%20species.%20The%20genus%20Coffea%20is%20native%20to%20tropical%20Africa%20(specifically%20having%20its%20origin%20in%20Ethiopia%20and%20Sudan)%20and%20Madagascar%2C%20the%20Comoros%2C%20Mauritius%2C%20and%20R%C3%A9union%20in%20the%20Indian%20Ocean.%5B2%5D%20Coffee%20plants%20are%20now%20cultivated%20in%20over%2070%20countries%2C%20primarily%20in%20the%20equatorial%20regions%20of%20the%20Americas%2C%20Southeast%20Asia%2C%20Indian%20subcontinent%2C%20and%20Africa.%20The%20two%20most%20commonly%20grown%20are%20C.%20arabica%20and%20C.%20robusta.%20Once%20ripe%2C%20coffee%20berries%20are%20picked%2C%20processed%2C%20and%20dried.%20Dried%20coffee%20seeds%20(referred%20to%20as%20%22beans%22)%20are%20roasted%20to%20varying%20degrees%2C%20depending%20on%20the%20desired%20flavor.%20Roasted%20beans%20are%20ground%20and%20then%20brewed%20with%20near-boiling%20water%20to%20produce%20the%20beverage%20known%20as%20coffee.&language=en&max_ngram_size=4&number_of_keywords=10"
+curl -X POST "http://127.0.0.1:5000/yake/" -H "accept: application/json" -H "Content-Type: application/json" \
+-d @- <<'EOF'
+{
+  "language": "en",
+  "max_ngram_size": 2,
+  "number_of_keywords": 10,
+  "text": "Sources tell us that Google is acquiring Kaggle, a platform that hosts data science and machine learning competitions. Details about the transaction remain somewhat vague , but given that Google is hosting its Cloud Next conference in San Francisco this week, the official announcement could come as early as tomorrow. Reached by phone, Kaggle co-founder CEO Anthony Goldbloom declined to deny that the acquisition is happening. Google itself declined 'to comment on rumors'. Kaggle, which has about half a million data scientists on its platform, was founded by Goldbloom and Ben Hamner in 2010. The service got an early start and even though it has a few competitors like DrivenData, TopCoder and HackerRank, it has managed to stay well ahead of them by focusing on its specific niche. The service is basically the de facto home for running data science and machine learning competitions. With Kaggle, Google is buying one of the largest and most active communities for data scientists - and with that, it will get increased mindshare in this community, too (though it already has plenty of that thanks to Tensorflow and other projects). Kaggle has a bit of a history with Google, too, but that's pretty recent. Earlier this month, Google and Kaggle teamed up to host a $100,000 machine learning competition around classifying YouTube videos. That competition had some deep integrations with the Google Cloud Platform, too. Our understanding is that Google will keep the service running - likely under its current name. While the acquisition is probably more about Kaggle's community than technology, Kaggle did build some interesting tools for hosting its competition and 'kernels', too. On Kaggle, kernels are basically the source code for analyzing data sets and developers can share this code on the platform (the company previously called them 'scripts'). Like similar competition centric sites, Kaggle also runs a job board, too. It's unclear what Google will do with that part of the service. According to Crunchbase, Kaggle raised $12.5 million (though PitchBook says it's $12.75) since its launch in 2010. Investors in Kaggle include Index Ventures, SV Angel, Max Levchin, Naval Ravikant, Google chief economist Hal Varian, Khosla Ventures and Yuri Milner"
+}
+EOF
 
 # test
 docker run "$YAKE_IMAGE:$TAG" -ti "Caffeine is a central nervous system (CNS) stimulant of the methylxanthine class.[10] It is the world's most widely consumed psychoactive drug. Unlike many other psychoactive substances, it is legal and unregulated in nearly all parts of the world. There are several known mechanisms of action to explain the effects of caffeine. The most prominent is that it reversibly blocks the action of adenosine on its receptor and consequently prevents the onset of drowsiness induced by adenosine. Caffeine also stimulates certain portions of the autonomic nervous system."