hdf5/HDF5_dataset_group_dataset.attrs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HDF5, 大量(海量?)数据存储的一种解决方案. HDF的全称是Hiearchical Data Format, 5是版本号(未考证过TODO). 一个HDF5文件操作起来就像一个独立的文件系统.  (TODO, I/O特性, 是事件驱动的吗?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import h5py\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 创建一个HDF5文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f = h5py.File(\"/home/dengdan/temp/no-use/hdftest.hdf5\", \"w\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "它可以存储两类数据对象:\n",
    "1. dataset, 类比于文件系统的文件, 可以用操作list/ndarray的方式来操作它\n",
    "2. group, 类比于文件系统的文件夹. , 可以用操作dict的方式来操作它"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(100, 100)\n"
     ]
    }
   ],
   "source": [
    "dset = f.create_dataset(name = \"/mydataset1\", shape = (100,100), dtype= np.uint8)\n",
    "print dset.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "uint8\n",
      "[[0 0 0 ..., 0 0 0]\n",
      " [0 0 0 ..., 0 0 0]\n",
      " [0 0 0 ..., 0 0 0]\n",
      " ..., \n",
      " [0 0 0 ..., 0 0 0]\n",
      " [0 0 0 ..., 0 0 0]\n",
      " [0 0 0 ..., 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "print dset.dtype\n",
    "print dset[:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "hdf5以POISX文件系统的风格存储数据对象, 每个对象都有自己的名字, 格式与linux文件路径相同"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/mydataset1\n"
     ]
    }
   ],
   "source": [
    "print dset.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grp = f.create_group(\"subgroup\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/subgroup\n"
     ]
    }
   ],
   "source": [
    "print grp.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/subgroup/another_ds\n"
     ]
    }
   ],
   "source": [
    "dset2 = grp.create_dataset(\"another_ds\", (50,), dtype='f')\n",
    "print dset2.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建dataset时若指定了上级group, 会自动创建"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/subgroup2/dataset_three\n"
     ]
    }
   ],
   "source": [
    "dset3 = f.create_dataset('subgroup2/dataset_three', (10,), dtype='i')\n",
    "print dset3.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "整个hdf文件就像一个大字典,读取dataset时可以根据它的name从中直接取出."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dset3_read = f['subgroup2/dataset_three']\n",
    "dset3 == dset3_read"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mydataset1\n",
      "subgroup\n",
      "subgroup2\n"
     ]
    }
   ],
   "source": [
    "for name in f:\n",
    "    print name\n",
    "    # 只会显示根目录下的对象."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mydataset1\n",
      "subgroup\n",
      "subgroup/another_ds\n",
      "subgroup2\n",
      "subgroup2/dataset_three\n"
     ]
    }
   ],
   "source": [
    "def visit_file(name):\n",
    "    print name\n",
    "f.visit(visit_file) \n",
    "    # 显示所有对象."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# dataset.attrs\n",
    "dataset对象可以有自己的属性, 但所有属性数据的长度加起来不能超过64K, 包括属性名字."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dset.attrs['length'] = 100\n",
    "dset.attrs['name'] = 'This is a dataset'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length : 100\n",
      "name : This is a dataset\n"
     ]
    }
   ],
   "source": [
    "for attr in dset.attrs:\n",
    "    print attr, \":\", dset.attrs[attr]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<hr>\n",
    "# Reference\n",
    "* http://docs.h5py.org/en/latest/quick.html\n",
    "\n",
    "markdown由jupyter notebook生成, [note book](https://github.com/dengdan/blog_posts/blob/master/hdf5/HDF5_dataset_group_dataset.attrs.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}