Skip to content

Commit

Permalink
Merge pull request #197 from aapa/urltitle-fix
Browse files Browse the repository at this point in the history 
Update nettiauto handler and broken tests
  • Loading branch information
@lepinkainen
lepinkainen committed Feb 23, 2016
2 parents 8a676c3 + 4023808 commit 53e60d2
Show file tree
Hide file tree
Showing 49 changed files with 21,955 additions and 7 deletions.
5 changes: 4 additions & 1 deletion pyfibot/modules/module_urltitle.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1217,7 +1217,10 @@ def fetch_nettiX(url, fields_to_fetch):

def _handle_nettiauto(url):
"""http*://*nettiauto.com/*/*/*"""
return fetch_nettiX(url, ['Vuosimalli', 'Mittarilukema', 'Moottori', 'Vaihteisto', 'Vetotapa'])
return fetch_nettiX(url, ['Vuosimallit', 'Mittarilukema', 'Moottori', 'Vaihteisto', 'Vetotapa'])


# TODO: Update other nettiX handlers


def _handle_nettivene(url):
Expand Down
2,416 changes: 2,416 additions & 0 deletions tests/cassettes/test_alko.yaml
View file

Large diffs are not rendered by default.

58 changes: 58 additions & 0 deletions tests/cassettes/test_apina.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,62 @@ interactions:
server: [nginx]
strict-transport-security: [max-age=15768000]
status: {code: 200, message: OK}
- request:
body: null
headers:
Accept: ['*/*']
Accept-Encoding: ['gzip, deflate']
Connection: [keep-alive]
User-Agent: ['Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0']
method: GET
uri: http://apina.biz/
response:
body:
string: !!binary |
H4sIAAAAAAAAAz2PQU/rMBCE7/0VixEUDiRpE2hDmyIkHogLQoILJ7Sxl2ZV145sB8i/f3ZBXCzv
7Gjm2zXvt+CdbETes/Q59mxwnvVmK8CHUVMjWusUuWsoVtCi3G2dHYy6kFbbKB4XRdR7VIrN9hpm
Rf+9EoA6NOL2+fHpVkC+Wbfu7528duzBcyCQ1gRk42GPgRyjhtBhiNMIxgbwAwcY7ZDBP9+TjHs9
An8kaeoIIgM5mC1hJHQerFbZT8/kziYPfLHvINhDDZuB0j90dOi++bWuETpHH40QEDikYx8s4Fck
iE6TCbBGapa7RqAmF86mbzFYokl8mvCTjqbnK7F5suscN5DDX15+w++OEnJjiNQ7h1Pc9ys3OG5O
5veHoftsystFXSyLRVvNsVXqihZlJYuyrq4WdVXVqpwV9bIkEps38qnjl/s5e8kgsaTweKHdMXkg
g60mlU3+A2St0kXVAQAA
headers:
cache-control: ['no-cache, must-revalidate']
connection: [keep-alive]
content-encoding: [gzip]
content-type: [text/html; charset=UTF-8]
date: ['Tue, 23 Feb 2016 18:21:07 GMT']
expires: ['Sat, 23 Oct 1999 00:55:20 GMT']
server: [nginx]
strict-transport-security: [max-age=15768000]
status: {code: 200, message: OK}
- request:
body: null
headers:
Accept: ['*/*']
Accept-Encoding: ['gzip, deflate']
Connection: [keep-alive]
User-Agent: ['Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0']
method: GET
uri: http://apina.biz/
response:
body:
string: !!binary |
H4sIAAAAAAAAAz2PS0/DMBCE7/0VixEUDuRVoC1tWiHxEBeEBBdO1cbZNKu6dmQ7hfx77IK4WN7Z
0cy3S95vwVlZirRj6VLsWGORdHorwPlBUSkqY2uyd5AtoEK521rT6/pKGmWCeJplQe+wrllv7yDP
uu+FAFS+FPdvL6/3AtLVsrL/7+ijZQeOPYE02iNrB3v0ZBkV+BZ9mAbQxoPr2cNg+gQeXUcy7NUA
3ERpbAkCA1nIZzAQWgdG1clvz+jBRA98sWvBm2MN657i37d07F7/WZcIraWmFAI8+3jsswH8CgTB
qRMBRkvFclcKVGT9xfgzBEvUkU8RHuhkfLkQq1ezTHEFKfznpWveWIrIpSaqN+zPcd8tbG+5PCue
jkN7KItmKpvsOqeJxCniTVHNJ/Mmw/zmFquaZDGRs7y4bsTqk1zs+ON+S94TiCwxPFxodkwOSGOl
qE5GP5NNZmLVAQAA
headers:
cache-control: ['no-cache, must-revalidate']
connection: [keep-alive]
content-encoding: [gzip]
content-type: [text/html; charset=UTF-8]
date: ['Tue, 23 Feb 2016 18:22:44 GMT']
expires: ['Sat, 23 Oct 1999 00:55:20 GMT']
server: [nginx]
strict-transport-security: [max-age=15768000]
status: {code: 200, message: OK}
version: 1
745 changes: 745 additions & 0 deletions tests/cassettes/test_areena_live.yaml
View file

Large diffs are not rendered by default.

2,008 changes: 2,008 additions & 0 deletions tests/cassettes/test_areena_radio.yaml
View file

Large diffs are not rendered by default.

2,048 changes: 2,048 additions & 0 deletions tests/cassettes/test_areena_series.yaml
View file

Large diffs are not rendered by default.

112 changes: 112 additions & 0 deletions tests/cassettes/test_areena_tv.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -590,4 +590,116 @@ interactions:
via: [1.1 varnish]
x-varnish: ['637772845']
status: {code: 200, message: OK}
- request:
body: null
headers:
Accept: ['*/*']
Accept-Encoding: ['gzip, deflate']
Connection: [keep-alive]
User-Agent: ['Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0']
method: GET
uri: http://areena-v3.yle.fi/tv/3210197.json
response:
body: {string: !!python/unicode '{ "prodNumber": null, "playCount": 94, "links":
[ ], "reportCategory": "uutiset", "naviCategory": "tv_uutiset", "pageTitle":
"Jouluruoka eri puolella Suomea", "pageDesc": "Miksi suomalaisessa joulup\u00f6yd\u00e4ss\u00e4
on kinkku ja laatikoita? Ent\u00e4 millaisia eroja l\u00f6ytyy juhlap\u00f6ydist\u00e4
id\u00e4ss\u00e4 ja l\u00e4nness\u00e4? Kysymyksiin vastasivat Ruokatiedon
viestint\u00e4p\u00e4\u00e4llikk\u00f6 Minna Asunmaa ja Maa- ja mets\u00e4talousnaisten
kotitalousopettaja Helena Velin.", "masterPrograms": [ ], "masterSerieses":
[ { "id": "2418425", "rootId": "2418425", "episodeCountViewable": 258, "clipCountViewable":
4799, "episodeCountTotal": 3057, "name": "Ylen aamu-tv", "shortDesc": null,
"headLine": null, "img": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_720.jpg",
"uiMegaImg": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_334_aamutv2015-sarja1024.jpg",
"images": { "orig": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_720.jpg",
"XS": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_65.jpg",
"S": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_160.jpg",
"M": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_220.jpg",
"L": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_620.jpg",
"XL": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_720.jpg"
} } ], "series": null, "availableClips": [ ], "media": { "id": "29-0_ng88dhvd",
"protection": 1, "contentId": null, "type": "video", "audioLang": "", "duration":
652, "subtitles": [ ], "downloadUrl": "" }, "downloadableMedias": [ ], "daysSincePublish":
62, "reportingTitle": "Jouluruoka eri puolella Suomea", "reportingAudio":
null, "images": { "orig": "http://cfvod.kaltura.com/p/1955031/sp/0/thumbnail/entry_id/0_ng88dhvd/width/1280",
"XS": "", "S": "", "M": "", "L": "", "XL": "" }, "directlyPlayable": true,
"published": "2015-12-23T08:55:00", "expires": null, "classifications": {
"main": [ { "children": [ { "key": "uutisohjelmat", "keyPath": "tv/uutiset/uutisohjelmat",
"name": "Uutisohjelmat", "id": "163" } ], "key": "uutiset", "keyPath": "tv/uutiset",
"name": "Uutiset", "id": "162" } ] }, "id": "3210197", "contentType": "CLIP",
"type": "video", "title": "Jouluruoka eri puolella Suomea", "desc": "Miksi
suomalaisessa joulup\u00f6yd\u00e4ss\u00e4 on kinkku ja laatikoita? Ent\u00e4
millaisia eroja l\u00f6ytyy juhlap\u00f6ydist\u00e4 id\u00e4ss\u00e4 ja l\u00e4nness\u00e4?
Kysymyksiin vastasivat Ruokatiedon viestint\u00e4p\u00e4\u00e4llikk\u00f6
Minna Asunmaa ja Maa- ja mets\u00e4talousnaisten kotitalousopettaja Helena
Velin.", "headLine": null, "headLineText": null, "duration": 10, "durationSec":
652, "ageLimit": "1", "restrictions": [ ], "international": true, "broadcasted":
null, "latestPublicationUiChannelName": "Yle TV1" } '}
headers:
accept-ranges: [bytes]
age: ['0']
connection: [keep-alive]
content-length: ['3038']
content-type: [application/json;charset=utf-8]
date: ['Tue, 23 Feb 2016 18:20:40 GMT']
server: [Apache-Coyote/1.1]
via: [1.1 varnish]
x-varnish: ['1120371666']
status: {code: 200, message: OK}
- request:
body: null
headers:
Accept: ['*/*']
Accept-Encoding: ['gzip, deflate']
Connection: [keep-alive]
User-Agent: ['Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0']
method: GET
uri: http://areena-v3.yle.fi/tv/3210197.json
response:
body: {string: !!python/unicode '{ "prodNumber": null, "playCount": 94, "links":
[ ], "reportCategory": "uutiset", "naviCategory": "tv_uutiset", "pageTitle":
"Jouluruoka eri puolella Suomea", "pageDesc": "Miksi suomalaisessa joulup\u00f6yd\u00e4ss\u00e4
on kinkku ja laatikoita? Ent\u00e4 millaisia eroja l\u00f6ytyy juhlap\u00f6ydist\u00e4
id\u00e4ss\u00e4 ja l\u00e4nness\u00e4? Kysymyksiin vastasivat Ruokatiedon
viestint\u00e4p\u00e4\u00e4llikk\u00f6 Minna Asunmaa ja Maa- ja mets\u00e4talousnaisten
kotitalousopettaja Helena Velin.", "masterPrograms": [ ], "masterSerieses":
[ { "id": "2418425", "rootId": "2418425", "episodeCountViewable": 258, "clipCountViewable":
4799, "episodeCountTotal": 3057, "name": "Ylen aamu-tv", "shortDesc": null,
"headLine": null, "img": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_720.jpg",
"uiMegaImg": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_334_aamutv2015-sarja1024.jpg",
"images": { "orig": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_720.jpg",
"XS": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_65.jpg",
"S": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_160.jpg",
"M": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_220.jpg",
"L": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_620.jpg",
"XL": "http://areena-v3.yle.fi/static/mk/images/areena/series/69267763_621_aamutv2015-sarja940_720.jpg"
} } ], "series": null, "availableClips": [ ], "media": { "id": "29-0_ng88dhvd",
"protection": 1, "contentId": null, "type": "video", "audioLang": "", "duration":
652, "subtitles": [ ], "downloadUrl": "" }, "downloadableMedias": [ ], "daysSincePublish":
62, "reportingTitle": "Jouluruoka eri puolella Suomea", "reportingAudio":
null, "images": { "orig": "http://cfvod.kaltura.com/p/1955031/sp/0/thumbnail/entry_id/0_ng88dhvd/width/1280",
"XS": "", "S": "", "M": "", "L": "", "XL": "" }, "directlyPlayable": true,
"published": "2015-12-23T08:55:00", "expires": null, "classifications": {
"main": [ { "children": [ { "key": "uutisohjelmat", "keyPath": "tv/uutiset/uutisohjelmat",
"name": "Uutisohjelmat", "id": "163" } ], "key": "uutiset", "keyPath": "tv/uutiset",
"name": "Uutiset", "id": "162" } ] }, "id": "3210197", "contentType": "CLIP",
"type": "video", "title": "Jouluruoka eri puolella Suomea", "desc": "Miksi
suomalaisessa joulup\u00f6yd\u00e4ss\u00e4 on kinkku ja laatikoita? Ent\u00e4
millaisia eroja l\u00f6ytyy juhlap\u00f6ydist\u00e4 id\u00e4ss\u00e4 ja l\u00e4nness\u00e4?
Kysymyksiin vastasivat Ruokatiedon viestint\u00e4p\u00e4\u00e4llikk\u00f6
Minna Asunmaa ja Maa- ja mets\u00e4talousnaisten kotitalousopettaja Helena
Velin.", "headLine": null, "headLineText": null, "duration": 10, "durationSec":
652, "ageLimit": "1", "restrictions": [ ], "international": true, "broadcasted":
null, "latestPublicationUiChannelName": "Yle TV1" } '}
headers:
accept-ranges: [bytes]
age: ['98']
connection: [keep-alive]
content-length: ['3038']
content-type: [application/json;charset=utf-8]
date: ['Tue, 23 Feb 2016 18:22:18 GMT']
server: [Apache-Coyote/1.1]
via: [1.1 varnish]
x-varnish: [1120371825 1120371666]
status: {code: 200, message: OK}
version: 1
92 changes: 92 additions & 0 deletions tests/cassettes/test_complex.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -137,4 +137,96 @@ interactions:
set-cookie: [WR_SID=217.30.184.161.1451380667278368; path=/; max-age=315360000;
domain=.wolframalpha.com]
status: {code: 200, message: OK}
- request:
body: null
headers:
Accept: ['*/*']
Accept-Encoding: ['gzip, deflate']
Connection: [keep-alive]
User-Agent: ['Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0']
method: GET
uri: http://api.wolframalpha.com/v2/query?input=answer%20to%20the%20life%20universe%20and%20everything&appid=3EYA3R-WVR6GJQWLH
response:
body: {string: !!python/unicode "<?xml version='1.0' encoding='UTF-8'?>\n<queryresult
success='true'\n error='false'\n numpods='2'\n datatypes=''\n timedout=''\n
\ timedoutpods=''\n timing='0.806'\n parsetiming='0.427'\n parsetimedout='false'\n
\ recalculate=''\n id='MSPa10022i8i7gge58c03ab000021eab2gddef5173h'\n
\ host='http://www5a.wolframalpha.com'\n server='16'\n related='http://www5a.wolframalpha.com/api/v2/relatedQueries.jsp?id=MSPa10122i8i7gge58c03ab00004ch4118c922a0439&amp;s=16'\n
\ version='2.6'\n profile='EnterDoQuery:0.,StartWrap:0.806218'>\n <pod
title='Input interpretation'\n scanner='Identity'\n id='Input'\n position='100'\n
\ error='false'\n numsubpods='1'>\n <subpod title=''>\n <img src='http://www5a.wolframalpha.com/Calculate/MSP/MSP10222i8i7gge58c03ab0000625730cbbafggh07?MSPStoreType=image/gif&amp;s=16'\n
\ alt='Answer to the Ultimate Question of Life, the Universe, and Everything'\n
\ title='Answer to the Ultimate Question of Life, the Universe, and Everything'\n
\ width='455'\n height='18' />\n <plaintext>Answer to the Ultimate
Question of Life, the Universe, and Everything</plaintext>\n </subpod>\n
</pod>\n <pod title='Result'\n scanner='Data'\n id='Result'\n position='200'\n
\ error='false'\n numsubpods='1'\n primary='true'>\n <subpod title=''>\n
\ <img src='http://www5a.wolframalpha.com/Calculate/MSP/MSP10322i8i7gge58c03ab0000294656fg7fgb7c9g?MSPStoreType=image/gif&amp;s=16'\n
\ alt='42\n(according to the book The Hitchhiker&apos;s Guide to the
Galaxy, by Douglas Adams)'\n title='42\n(according to the book The Hitchhiker&apos;s
Guide to the Galaxy, by Douglas Adams)'\n width='442'\n height='36'
/>\n <plaintext>42\n(according to the book The Hitchhiker&apos;s Guide to
the Galaxy, by Douglas Adams)</plaintext>\n </subpod>\n </pod>\n <assumptions
count='1'>\n <assumption type='SubCategory'\n word='answer to the life
universe and everything'\n template='Assuming ${desc1}. Use ${desc2}
instead'\n count='2'>\n <value name='AnswerToLifeUniverseEverything'\n
\ desc='The Ultimate Answer'\n input='*DPClash.MiscellaneousE.answer+to+the+life+universe+and+everything-_*AnswerToLifeUniverseEverything-'
/>\n <value name='MontyPythonsMeaningOfLife'\n desc='Monty Python&apos;s
Meaning of Life'\n input='*DPClash.MiscellaneousE.answer+to+the+life+universe+and+everything-_*MontyPythonsMeaningOfLife-'
/>\n </assumption>\n </assumptions>\n</queryresult>"}
headers:
connection: [close]
content-length: ['2468']
content-type: [text/xml;charset=utf-8]
date: ['Tue, 23 Feb 2016 18:21:16 GMT']
server: [Apache-Coyote/1.1]
set-cookie: [WR_SID=217.30.184.161.1456251676708056; path=/; max-age=315360000;
domain=.wolframalpha.com]
status: {code: 200, message: OK}
- request:
body: null
headers:
Accept: ['*/*']
Accept-Encoding: ['gzip, deflate']
Connection: [keep-alive]
User-Agent: ['Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0']
method: GET
uri: http://api.wolframalpha.com/v2/query?input=answer%20to%20the%20life%20universe%20and%20everything&appid=3EYA3R-WVR6GJQWLH
response:
body: {string: !!python/unicode "<?xml version='1.0' encoding='UTF-8'?>\n<queryresult
success='true'\n error='false'\n numpods='2'\n datatypes=''\n timedout=''\n
\ timedoutpods=''\n timing='0.839'\n parsetiming='0.436'\n parsetimedout='false'\n
\ recalculate=''\n id='MSPa25411hg9516fa6abf22d00002d4ba34d9ei2g34h'\n
\ host='http://www4c.wolframalpha.com'\n server='53'\n related='http://www4c.wolframalpha.com/api/v2/relatedQueries.jsp?id=MSPa25421hg9516fa6abf22d00003dbdc8db4agec53b&amp;s=53'\n
\ version='2.6'\n profile='EnterDoQuery:0.,StartWrap:0.839316'>\n <pod
title='Input interpretation'\n scanner='Identity'\n id='Input'\n position='100'\n
\ error='false'\n numsubpods='1'>\n <subpod title=''>\n <img src='http://www4c.wolframalpha.com/Calculate/MSP/MSP25431hg9516fa6abf22d000040ag5b58g06fc939?MSPStoreType=image/gif&amp;s=53'\n
\ alt='Answer to the Ultimate Question of Life, the Universe, and Everything'\n
\ title='Answer to the Ultimate Question of Life, the Universe, and Everything'\n
\ width='455'\n height='18' />\n <plaintext>Answer to the Ultimate
Question of Life, the Universe, and Everything</plaintext>\n </subpod>\n
</pod>\n <pod title='Result'\n scanner='Data'\n id='Result'\n position='200'\n
\ error='false'\n numsubpods='1'\n primary='true'>\n <subpod title=''>\n
\ <img src='http://www4c.wolframalpha.com/Calculate/MSP/MSP25441hg9516fa6abf22d00004ai0gfhf5ced7cia?MSPStoreType=image/gif&amp;s=53'\n
\ alt='42\n(according to the book The Hitchhiker&apos;s Guide to the
Galaxy, by Douglas Adams)'\n title='42\n(according to the book The Hitchhiker&apos;s
Guide to the Galaxy, by Douglas Adams)'\n width='442'\n height='36'
/>\n <plaintext>42\n(according to the book The Hitchhiker&apos;s Guide to
the Galaxy, by Douglas Adams)</plaintext>\n </subpod>\n </pod>\n <assumptions
count='1'>\n <assumption type='SubCategory'\n word='answer to the life
universe and everything'\n template='Assuming ${desc1}. Use ${desc2}
instead'\n count='2'>\n <value name='AnswerToLifeUniverseEverything'\n
\ desc='The Ultimate Answer'\n input='*DPClash.MiscellaneousE.answer+to+the+life+universe+and+everything-_*AnswerToLifeUniverseEverything-'
/>\n <value name='MontyPythonsMeaningOfLife'\n desc='Monty Python&apos;s
Meaning of Life'\n input='*DPClash.MiscellaneousE.answer+to+the+life+universe+and+everything-_*MontyPythonsMeaningOfLife-'
/>\n </assumption>\n </assumptions>\n</queryresult>"}
headers:
connection: [close]
content-length: ['2472']
content-type: [text/xml;charset=utf-8]
date: ['Tue, 23 Feb 2016 18:22:53 GMT']
server: [Apache-Coyote/1.1]
set-cookie: [WR_SID=217.30.184.161.1456251773593519; path=/; max-age=315360000;
domain=.wolframalpha.com]
status: {code: 200, message: OK}
version: 1

0 comments on commit 53e60d2

Please sign in to comment.