parse npl TTS data

[clkao]

basically what https://github.com/g0v/ly-crx does, but automated.

there's are some issues with the tts site making it hard for usual scrapper tools (mechanize, phantomjs) to work:

• sessions are encoded in URL
• form encoding is big5

[clkao]

for example:

http://lis.ly.gov.tw/lgcgi/ttswebpw?in_out/mempropg08in
select all sittings (08-01-01 - 08-03.-xxx), query
ly-crx will create "download all" for you

we then use twlyparser/parse-tts.ls to process the downloaded html to json

[audreyt]

有一個想法是 generic transcoding proxy，例如 Perlbal 強制吃五筒打八萬。

[gugod]

我之前寫了個簡繁轉換的放在 heroku，剛加上了 big5_to_utf8

curl http://hanconvert.herokuapp.com/big5_to_utf8/'?url=http://udn.com/NEWS/mainpage.shtml'

Repo: https://github.com/g0v/hanconvert

[audreyt]

@gugod 有可能接 http://search.cpan.org/dist/HTTP-Proxy/lib/HTTP/Proxy/BodyFilter/save.pm 然後多寫一個 Big5ToUTF8 的 BodyFilter 嗎?

[audreyt]

Test case 除了 apily 之外，還有 https://github.com/hcchien/doh-cancer/blob/master/tree.js 的最後一步 (其實是每一步) 要留下檔案。

[sweslo17]

來爬這個試試看

[clkao]

@sweslo17 有用上面的麻將 proxy 嗎？另外抓下來的 html 可以用 twlyparser/parse-tts 處理，一些該 normalize 的東西都在那邊做了

[clkao]

quick note: the tts site (lis.ly.gov.tw) fails if your request header contains 'Connection: closed'
you must use keep-alive or request without the connection header.

[clkao]

ok, now we can access lis site in utf-8 with lisproxy.psgi.
Usage:

• install https://github.com/clkao/Plack-App-Proxy/
• plackup lisproxy.psgi
• go to http://localhost:5000/ttscgi/ttsweb?@@4163B777639423C04145

[clkao]

ok this is now working with casperjs:

• install phantomjs and casperjs
• npm i -g bower
• bower install jquery
• casperjs scripts/tts.coffee
  this will generate output.html that is ready for parse-tts to use.

we now just need to refactor scripts/tts.coffee and make it more flexible.