Data-Exploration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processed English"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Life in the deep oceans',\n",
       " 'With vibrant video clips captured by submarines , David Gallo takes us to some of Earth &apos;s darkest , most violent , toxic and beautiful habitats , the valleys and volcanic ridges of the oceans &apos; depths , where life is bizarre , resilient and shockingly abundant .',\n",
       " 'This is Bill Lange . I &apos;m Dave Gallo .',\n",
       " 'And we &apos;re going to tell you some stories from the sea here in video .',\n",
       " 'We &apos;ve got some of the most incredible video of Titanic that &apos;s ever been seen , and we &apos;re not going to show you any of it .',\n",
       " 'The truth of the matter is that the Titanic -- even though it &apos;s breaking all sorts of box office records -- it &apos;s not the most exciting story from the sea .',\n",
       " 'And the problem , I think , is that we take the ocean for granted .',\n",
       " 'When you think about it , the oceans are 75 percent of the planet .',\n",
       " 'Most of the planet is ocean water .',\n",
       " 'The average depth is about two miles .']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('data/iwslt-zh-en/train.tok.en', encoding = 'utf-8').read().\\\n",
    "            strip().split('\\n')[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Life in the deep oceans',\n",
       " \"With vibrant video clips captured by submarines , David Gallo takes us to some of Earth's darkest , most violent , toxic and beautiful habitats , the valleys and volcanic ridges of the oceans' depths , where life is bizarre , resilient and shockingly abundant .\",\n",
       " \"This is Bill Lange . I'm Dave Gallo .\",\n",
       " \"And we're going to tell you some stories from the sea here in video .\",\n",
       " \"We've got some of the most incredible video of Titanic that's ever been seen , and we're not going to show you any of it .\",\n",
       " \"The truth of the matter is that the Titanic -- even though it's breaking all sorts of box office records -- it's not the most exciting story from the sea .\",\n",
       " 'And the problem , I think , is that we take the ocean for granted .',\n",
       " 'When you think about it , the oceans are 75 percent of the planet .',\n",
       " 'Most of the planet is ocean water .',\n",
       " 'The average depth is about two miles .']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### replace ' &apos;' to \"'\"\n",
    "open('data/iwslt-zh-en/train.tok.en', encoding = 'utf-8').read().\\\n",
    "            replace(' &apos;', \"'\").strip().split('\\n')[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "process_en = open('data/iwslt-zh-en/train.tok.en', encoding = 'utf-8').read().\\\n",
    "strip().split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total num of samples is: 213376\n",
      "total nums of samples whose length <= 100 is: 212956\n"
     ]
    }
   ],
   "source": [
    "print(\"total num of samples is:\",len([len(i.split(\" \")) for i in process_en]))\n",
    "print(\"total nums of samples whose length <= 100 is:\",len([i for i in [len(j.split(\" \")) for j in process_en] if i < 100]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processed Chinese"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['深海 海中 的 生命   大卫   盖罗 ',\n",
       " '大卫   盖罗 通过 潜水 潜水艇 拍下 的 影片 把 我们 带到 了 地球 最 黑暗   最 险恶 同时 也 最美 美丽 的 生物 栖息 栖息地   这里 是 海洋 深处 的 峡谷 和 火山 山脊   这里 怪诞   适应 适应力 应力 强 而且 数量 惊人 的 生命  ',\n",
       " '大卫   盖罗   这位 是 比尔   兰格    我 是 大卫   盖罗  ',\n",
       " '我们 将 用 一些 影片 来讲 讲述 一些 深海 海里 的 故事  ',\n",
       " '我们 这 有 不少 精彩 的 泰坦 泰坦尼克 坦尼 尼克 的 影片    可惜 您 今天 看不到 不到  ',\n",
       " '泰坦 泰坦尼克 泰坦尼克号 坦尼 尼克 号   是 拿 了 不少 票房 冠军   但 事实 事实上 它 并 不是 关于 于海洋 海洋 的 最 刺激 的 故事  ',\n",
       " '原因 在于 我们 一直 没 把 海洋 当回事 回事 回事儿 事儿  ',\n",
       " '大家 想想   海洋 占 了 地球 球面 面积 的 75  ',\n",
       " '地球 的 大部 大部分 部分 都 是 海水  ',\n",
       " '海洋 的 平均 深度 是 两英里 英里 ']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('data/iwslt-zh-en/train.tok.zh', encoding = 'utf-8').read().\\\n",
    "strip().split('\\n')[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "process_ch = open('data/iwslt-zh-en/train.tok.zh', encoding = 'utf-8').read().\\\n",
    "strip().split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total num of samples is: 213376\n",
      "total nums of samples whose length <= 100 is: 211873\n"
     ]
    }
   ],
   "source": [
    "print(\"total num of samples is:\",len([len(i.split(\" \")) for i in process_ch]))\n",
    "print(\"total nums of samples whose length <= 100 is:\",len([i for i in [len(j.split(\" \")) for j in process_ch] if i < 100]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unprocessed Chinese"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['深海中的生命－大卫.盖罗',\n",
       " '大卫.盖罗通过潜水艇拍下的影片把我们带到了地球最黑暗，最险恶同时也最美丽的生物栖息地。这里是海洋深处的峡谷和火山脊，这里怪诞，适应力强而且数量惊人的生命。',\n",
       " '大卫.盖罗：这位是比尔.兰格， 我是大卫.盖罗。',\n",
       " '我们将用一些影片来讲述一些深海里的故事。',\n",
       " '我们这有不少精彩的泰坦尼克的影片， 可惜您今天看不到。',\n",
       " '泰坦尼克号 是拿了不少票房冠军 但事实上它并不是关于海洋的最刺激的故事。',\n",
       " '原因在于我们一直没把海洋当回事儿。',\n",
       " '大家想想，海洋占了地球面积的75％。',\n",
       " '地球的大部分都是海水。',\n",
       " '海洋的平均深度是两英里']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('data/iwslt-zh-en-processed/train.zh', encoding = 'utf-8').read().strip().split('\\n')[0:10]\n",
    "\n",
    "# .\\\n",
    "# strip().split('\\n')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}