461 _build/html/dynamo.html
View file

Large diffs are not rendered by default.

232 _build/html/filesystems.html
View file
@@ -0,0 +1,232 @@


<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<title>Filesystems &mdash; PantsNotes 1 documentation</title>















<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />





<link rel="top" title="PantsNotes 1 documentation" href="index.html"/>
<link rel="prev" title="Welcome to PantsNotes’s documentation!" href="index.html"/>


<script src="_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav" role="document">

<div class="wy-grid-for-nav">


<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">



<a href="index.html" class="icon icon-home"> PantsNotes



</a>




<div class="version">
1
</div>




<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>


</div>

<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



<ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="">Filesystems</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#general">General</a></li>
<li class="toctree-l2"><a class="reference internal" href="#xfs">XFS</a></li>
<li class="toctree-l2"><a class="reference internal" href="#id1">NTFS</a></li>
</ul>
</li>
</ul>



</div>
</div>
</nav>

<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">PantsNotes</a>
</nav>



<div class="wy-nav-content">
<div class="rst-content">






<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> &raquo;</li>

<li>Filesystems</li>
<li class="wy-breadcrumbs-aside">


<a href="_sources/filesystems.txt" rel="nofollow"> View page source</a>


</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">

<div class="section" id="filesystems">
<h1>Filesystems<a class="headerlink" href="#filesystems" title="Permalink to this headline"></a></h1>
<div class="section" id="general">
<h2>General<a class="headerlink" href="#general" title="Permalink to this headline"></a></h2>
<p>A filesystem is something which defines how data is stored and retrieved.</p>
<p>A filesystem typically stores all the metadata associated with the file - including the file name, length, and location within the directory hierarchy - separate from the contents of the file.</p>
<p>Most filesystems store the names of all the files which are in one directory in one place - the directory table for that directory - which is often stored like any other file. Many filesystems put only some of the metadata for a file in the directory table, and the rest of the metadata for that file in a completely separate structure, such as the inode.</p>
<p>Most filesystems also store metadata not associated with any one particular file. Such metadata includes information about unused regions - free space bitmap, or a B-Tree structure - and information about bad sectors. Often such information about an allocation group is stored inside the allocation group itself. More on this information in the filesystem-specific sections.</p>
</div>
<div class="section" id="xfs">
<h2>XFS<a class="headerlink" href="#xfs" title="Permalink to this headline"></a></h2>
<p>XFS is a high performance <em>journaling</em> file system. It generally excels in the execution of parallel input/output operations, due to its <em>allocation group</em> based design. XFS is an <em>extent-based</em> filesystem.</p>
<p>XFS is internally partitioned into <strong>allocation groups</strong>, which are virtual storage regions of fixed size. Any files and directories that you create can span multiple allocation groups. Each allocation group manages its own journal, its own inodes, and its free space independently of other allocation groups. This is where it gains its advantage in both scalability and parallelism. If the file system spans many physical devices, allocation groups can optimize throughput by taking advantage of the underlying separation of channels to the storage components.</p>
<p>XFS implements <strong>journaling</strong> for metadata operations. XFS records file system updates asynchronously to a circular buffer before it can commit the actual data updates to disk. The journal can either be located internally in the data section of the file system, or externally on a separate device to reduce contention for disk access. If the system crashes or loses power, the journal is read when the filesystem is remounted, and replays any pending metadata operations. The speed of this recovery is not dependent on the size of the filesystem.</p>
<p>To reduce fragmentation and file scattering, each file&#8217;s blocks in XFS can have variable length <strong>extents</strong></p>
<p>Whilst many other filesystems manage space allocation with one or more block oriented bitmaps (see <span class="target" id="ntfs">NTFS</span>), in XFS these structures are replaced with an extent oriented structure consisting of a pair of B+ trees (see TODO for more information on B-trees) for each file system allocation group.</p>
<p>Ref:
<a class="reference external" href="http://linux-xfs.sgi.com/projects/xfs/papers/xfs_white/xfs_white_paper.html">http://linux-xfs.sgi.com/projects/xfs/papers/xfs_white/xfs_white_paper.html</a>
<a class="reference external" href="https://en.wikipedia.org/wiki/XFS">https://en.wikipedia.org/wiki/XFS</a></p>
</div>
<div class="section" id="id1">
<h2>NTFS<a class="headerlink" href="#id1" title="Permalink to this headline"></a></h2>
<p>We generally don&#8217;t care about NTFS, but we did mention &#8220;free space bitmap&#8221; in the general section. You may also remember having to &#8220;defrag&#8221; your hard drive and were wondering why. While ext* also use free space bitmaps, NTFS&#8217;s implementation gives the clearest explanation of this phenomenon.</p>
<p>NTFS uses a special $BitMap file to keep track of all the used and unused &#8220;clusters&#8221; (ntfs nomenclature for logical blocks: anywhere from 512b to 64KB) on an NTFS volume. It&#8217;s a single file (a type of Free Space Bitmap as mentioned above) which contains a mapping of the entire volume, where when a file takes up space on the NTFS volume, the corresponding location in the $BitMap file is marked as used (11111...). When data is removed, the corresponding map location is marked 0 as unused. When an OS is looking to write out a file, it scans this map file to see where the next available logical block is in which it can place this data, writes out a block, then continues on looking for the next free space available such that it can finish writing the file (if said file is larger than a single block). Even if the OS manages to write out the whole file in contiguous blocks, another file will be written immediately after it. If the original file grows at all, it becomes fragmented. We can see with this approach how quickly fragmentation can occur, and how unruly a $BitMap file can get on a large filesystem (64MB on a 2TB drive).</p>
<p>Here we can see the bitmap from an NTFS volume with a single 1MB file added:</p>
<img alt="NTFS $BitMap with 1MB file added" class="align-center" src="_images/NTFS-BitMap.jpg" />
<p>Note that ext3/4 also use bitmaps, but in regards to fragmentation they are a little bit smarter. Instead of placing multiple files near each other on the hard disk, ext3/4 will scatter different files all over the disk, leaving a large amount of free space between them. They will also choose free blocks around the (growing) file when writing out new data, rather than just the first free block they find. If a file does end up fragmenting, the file system will attempt to move the files around to reduce fragmentation (in ext4 at least). It does this automatically, without the need to run any utility. It can do this so long as the filesystem is not super full (80%+).</p>
<p>Ref:
<a class="reference external" href="https://whereismydata.wordpress.com/2009/06/01/forensics-what-is-the-bitmap/">https://whereismydata.wordpress.com/2009/06/01/forensics-what-is-the-bitmap/</a></p>
</div>
</div>


</div>
</div>
<footer>

<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">


<a href="index.html" class="btn btn-neutral" title="Welcome to PantsNotes’s documentation!" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>

</div>


<hr/>

<div role="contentinfo">
<p>
&copy; Copyright 2015, Pants.

</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

</footer>

</div>
</div>

</section>

</div>





<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'1',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>





<script type="text/javascript" src="_static/js/theme.js"></script>




<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>


</body>
</html>
196 _build/html/genindex.html
View file
@@ -0,0 +1,196 @@



<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<title>Index &mdash; PantsNotes 1 documentation</title>















<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />





<link rel="top" title="PantsNotes 1 documentation" href="index.html"/>


<script src="_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav" role="document">

<div class="wy-grid-for-nav">


<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">



<a href="index.html" class="icon icon-home"> PantsNotes



</a>




<div class="version">
1
</div>




<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>


</div>

<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



<ul>
<li class="toctree-l1"><a class="reference internal" href="TODO-interview.html">Interview Material</a></li>
<li class="toctree-l1"><a class="reference internal" href="cassandra.html">Cassandra</a></li>
<li class="toctree-l1"><a class="reference internal" href="dynamo.html">DYNAMO</a></li>
<li class="toctree-l1"><a class="reference internal" href="filesystems.html">Filesystems</a></li>
</ul>



</div>
</div>
</nav>

<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">PantsNotes</a>
</nav>



<div class="wy-nav-content">
<div class="rst-content">






<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> &raquo;</li>

<li></li>
<li class="wy-breadcrumbs-aside">



</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">


<h1 id="index">Index</h1>

<div class="genindex-jumpbox">

</div>


</div>
</div>
<footer>


<hr/>

<div role="contentinfo">
<p>
&copy; Copyright 2015, Pants.

</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

</footer>

</div>
</div>

</section>

</div>





<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'1',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>





<script type="text/javascript" src="_static/js/theme.js"></script>




<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>


</body>
</html>
245 _build/html/index.html
View file
@@ -0,0 +1,245 @@


<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<title>Welcome to PantsNotes’s documentation! &mdash; PantsNotes 1 documentation</title>















<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />





<link rel="top" title="PantsNotes 1 documentation" href="#"/>
<link rel="next" title="&lt;no title&gt;" href="TODO-bootprocess.html"/>


<script src="_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav" role="document">

<div class="wy-grid-for-nav">


<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">



<a href="#" class="icon icon-home"> PantsNotes



</a>




<div class="version">
1
</div>




<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>


</div>

<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



<ul>
<li class="toctree-l1"><a class="reference internal" href="TODO-interview.html">Interview Material</a></li>
<li class="toctree-l1"><a class="reference internal" href="cassandra.html">Cassandra</a></li>
<li class="toctree-l1"><a class="reference internal" href="dynamo.html">DYNAMO</a></li>
<li class="toctree-l1"><a class="reference internal" href="filesystems.html">Filesystems</a></li>
</ul>



</div>
</div>
</nav>

<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="#">PantsNotes</a>
</nav>



<div class="wy-nav-content">
<div class="rst-content">






<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="#">Docs</a> &raquo;</li>

<li>Welcome to PantsNotes&#8217;s documentation!</li>
<li class="wy-breadcrumbs-aside">


<a href="_sources/index.txt" rel="nofollow"> View page source</a>


</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">

<div class="section" id="welcome-to-pantsnotes-s-documentation">
<h1>Welcome to PantsNotes&#8217;s documentation!<a class="headerlink" href="#welcome-to-pantsnotes-s-documentation" title="Permalink to this headline"></a></h1>
<p>This is a spot where various sysadmin related notes can be read in one place.</p>
<p>Contents:</p>
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference internal" href="TODO-interview.html">Interview Material</a><ul>
<li class="toctree-l2"><a class="reference internal" href="TODO-interview.html#questions">Questions</a></li>
<li class="toctree-l2"><a class="reference internal" href="TODO-interview.html#study-topics">Study Topics</a></li>
<li class="toctree-l2"><a class="reference internal" href="TODO-interview.html#design">Design</a></li>
<li class="toctree-l2"><a class="reference internal" href="TODO-interview.html#coding-questions">Coding Questions</a></li>
<li class="toctree-l2"><a class="reference internal" href="TODO-interview.html#quickies">Quickies</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="cassandra.html">Cassandra</a><ul>
<li class="toctree-l2"><a class="reference internal" href="cassandra.html#intro">Intro</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="dynamo.html">DYNAMO</a><ul>
<li class="toctree-l2"><a class="reference internal" href="dynamo.html#general">General</a></li>
<li class="toctree-l2"><a class="reference internal" href="dynamo.html#sla">SLA</a></li>
<li class="toctree-l2"><a class="reference internal" href="dynamo.html#design-choices">Design Choices</a></li>
<li class="toctree-l2"><a class="reference internal" href="dynamo.html#architecture">Architecture</a></li>
<li class="toctree-l2"><a class="reference internal" href="dynamo.html#data-versioning">Data Versioning</a></li>
<li class="toctree-l2"><a class="reference internal" href="dynamo.html#failure-and-membership">Failure and Membership</a></li>
<li class="toctree-l2"><a class="reference internal" href="dynamo.html#implementation">Implementation</a></li>
<li class="toctree-l2"><a class="reference internal" href="dynamo.html#nrw-performance-vs-durability">NRW: Performance vs Durability</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="filesystems.html">Filesystems</a><ul>
<li class="toctree-l2"><a class="reference internal" href="filesystems.html#general">General</a></li>
<li class="toctree-l2"><a class="reference internal" href="filesystems.html#xfs">XFS</a></li>
<li class="toctree-l2"><a class="reference internal" href="filesystems.html#id1">NTFS</a></li>
</ul>
</li>
</ul>
</div>
</div>
<div class="section" id="indices-and-tables">
<h1>Indices and tables<a class="headerlink" href="#indices-and-tables" title="Permalink to this headline"></a></h1>
<ul class="simple">
<li><a class="reference internal" href="genindex.html"><span>Index</span></a></li>
<li><a class="reference internal" href="py-modindex.html"><span>Module Index</span></a></li>
<li><a class="reference internal" href="search.html"><span>Search Page</span></a></li>
</ul>
</div>


</div>
</div>
<footer>

<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">

<a href="TODO-bootprocess.html" class="btn btn-neutral float-right" title="&lt;no title&gt;" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>


</div>


<hr/>

<div role="contentinfo">
<p>
&copy; Copyright 2015, Pants.

</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

</footer>

</div>
</div>

</section>

</div>





<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'1',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>





<script type="text/javascript" src="_static/js/theme.js"></script>




<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>


</body>
</html>
238 _build/html/interview.html
View file
@@ -0,0 +1,238 @@


<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<title>Interview Questions &mdash; PantsNotes 1 documentation</title>















<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />





<link rel="top" title="PantsNotes 1 documentation" href="index.html"/>
<link rel="next" title="IP" href="ip.html"/>
<link rel="prev" title="Welcome to PantsNotes’s documentation!" href="index.html"/>


<script src="_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav" role="document">

<div class="wy-grid-for-nav">


<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">



<a href="index.html" class="icon icon-home"> PantsNotes



</a>




<div class="version">
1
</div>




<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>


</div>

<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



<ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="">Interview Questions</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#filesystems">Filesystems</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#general">General</a></li>
<li class="toctree-l3"><a class="reference internal" href="#xfs">XFS</a></li>
<li class="toctree-l3"><a class="reference internal" href="#id1">NTFS</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="ip.html">IP</a></li>
<li class="toctree-l1"><a class="reference internal" href="kernel.html">Kernel</a></li>
<li class="toctree-l1"><a class="reference internal" href="protocols.html">Protocols</a></li>
</ul>



</div>
</div>
</nav>

<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">PantsNotes</a>
</nav>



<div class="wy-nav-content">
<div class="rst-content">






<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> &raquo;</li>

<li>Interview Questions</li>
<li class="wy-breadcrumbs-aside">


<a href="_sources/interview.txt" rel="nofollow"> View page source</a>


</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">

<div class="section" id="interview-questions">
<h1>Interview Questions<a class="headerlink" href="#interview-questions" title="Permalink to this headline"></a></h1>
<div class="section" id="filesystems">
<h2>Filesystems<a class="headerlink" href="#filesystems" title="Permalink to this headline"></a></h2>
<div class="section" id="general">
<h3>General<a class="headerlink" href="#general" title="Permalink to this headline"></a></h3>
<p>A filesystem is something which defines how data is stored and retrieved.</p>
<p>A filesystem typically stores all the metadata associated with the file - including the file name, length, and location within the directory hierarchy - separate from the contents of the file.</p>
<p>Most filesystems store the names of all the files which are in one directory in one place - the directory table for that directory - which is often stored like any other file. Many filesystems put only some of the metadata for a file in the directory table, and the rest of the metadata for that file in a completely separate structure, such as the inode.</p>
<p>Most filesystems also store metadata not associated with any one particular file. Such metadata includes information about unused regions - free space bitmap, or a B-Tree structure - and information about bad sectors. Often such information about an allocation group is stored inside the allocation group itself. More on this information in the filesystem-specific sections.</p>
</div>
<div class="section" id="xfs">
<h3>XFS<a class="headerlink" href="#xfs" title="Permalink to this headline"></a></h3>
<p>Whilst many other filesystems manage space allocation with one or more block oriented bitmaps (see <span class="target" id="ntfs">NTFS</span>), in XFS these structures are replaced with an extent oriented structure consisting of a pair of B+ trees (see TODO for more information on B-trees) for each file system allocation group.</p>
<p>[<a class="reference external" href="http://linux-xfs.sgi.com/projects/xfs/papers/xfs_white/xfs_white_paper.html">http://linux-xfs.sgi.com/projects/xfs/papers/xfs_white/xfs_white_paper.html</a>]
[<a class="reference external" href="https://en.wikipedia.org/wiki/XFS">https://en.wikipedia.org/wiki/XFS</a>]</p>
</div>
<div class="section" id="id1">
<h3>NTFS<a class="headerlink" href="#id1" title="Permalink to this headline"></a></h3>
<p>We generally don&#8217;t care about NTFS, but we did mention &#8220;free space bitmap&#8221; in the general section. You may also remember having to &#8220;defrag&#8221; your hard drive and were wondering why. While ext* also use free space bitmaps, NTFS&#8217;s implementation gives the clearest explanation of this phenomenon.</p>
<p>NTFS uses a special $BitMap file to keep track of all the used and unused &#8220;clusters&#8221; (ntfs nomenclature for logical blocks: anywhere from 512b to 64KB) on an NTFS volume. It&#8217;s a single file (a type of Free Space Bitmap as mentioned above) which contains a mapping of the entire volume, where when a file takes up space on the NTFS volume, the corresponding location in the $BitMap file is marked as used (11111...). When data is removed, the corresponding map location is marked 0 as unused. When an OS is looking to write out a file, it scans this map file to see where the next available logical block is in which it can place this data, writes out a block, then continues on looking for the next free space available such that it can finish writing the file (if said file is larger than a single block). Even if the OS manages to write out the whole file in contiguous blocks, another file will be written immediately after it. If the original file grows at all, it becomes fragmented. We can see with this approach how quickly fragmentation can occur, and how unruly a $BitMap file can get on a large filesystem (64MB on a 2TB drive).</p>
<p>Here we can see the bitmap from an NTFS volume with a single 1MB file added:</p>
<img alt="NTFS $BitMap with 1MB file added" class="align-center" src="_images/NTFS-BitMap.jpg" />
<p>Note that ext3/4 also use bitmaps, but in regards to fragmentation they are a little bit smarter. Instead of placing multiple files near each other on the hard disk, ext3/4 will scatter different files all over the disk, leaving a large amount of free space between them. They will also choose free blocks around the (growing) file when writing out new data, rather than just the first free block they find. If a file does end up fragmenting, the file system will attempt to move the files around to reduce fragmentation (in ext4 at least). It does this automatically, without the need to run any utility. It can do this so long as the filesystem is not super full (80%+).</p>
<p>[<a class="reference external" href="https://whereismydata.wordpress.com/2009/06/01/forensics-what-is-the-bitmap/">https://whereismydata.wordpress.com/2009/06/01/forensics-what-is-the-bitmap/</a>]</p>
</div>
</div>
</div>


</div>
</div>
<footer>

<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">

<a href="ip.html" class="btn btn-neutral float-right" title="IP" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>


<a href="index.html" class="btn btn-neutral" title="Welcome to PantsNotes’s documentation!" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>

</div>


<hr/>

<div role="contentinfo">
<p>
&copy; Copyright 2015, Pants.

</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

</footer>

</div>
</div>

</section>

</div>





<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'1',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>





<script type="text/javascript" src="_static/js/theme.js"></script>




<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>


</body>
</html>
211 _build/html/ip.html
View file
@@ -0,0 +1,211 @@


<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<title>IP &mdash; PantsNotes 1 documentation</title>















<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />





<link rel="top" title="PantsNotes 1 documentation" href="index.html"/>
<link rel="next" title="Kernel" href="kernel.html"/>
<link rel="prev" title="Interview Questions" href="interview.html"/>


<script src="_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav" role="document">

<div class="wy-grid-for-nav">


<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">



<a href="index.html" class="icon icon-home"> PantsNotes



</a>




<div class="version">
1
</div>




<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>


</div>

<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="interview.html">Interview Questions</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="">IP</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#dash-header">Dash header</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="kernel.html">Kernel</a></li>
<li class="toctree-l1"><a class="reference internal" href="protocols.html">Protocols</a></li>
</ul>



</div>
</div>
</nav>

<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">PantsNotes</a>
</nav>



<div class="wy-nav-content">
<div class="rst-content">






<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> &raquo;</li>

<li>IP</li>
<li class="wy-breadcrumbs-aside">


<a href="_sources/ip.txt" rel="nofollow"> View page source</a>


</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">

<div class="section" id="ip">
<h1>IP<a class="headerlink" href="#ip" title="Permalink to this headline"></a></h1>
<div class="section" id="dash-header">
<h2>Dash header<a class="headerlink" href="#dash-header" title="Permalink to this headline"></a></h2>
</div>
</div>


</div>
</div>
<footer>

<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">

<a href="kernel.html" class="btn btn-neutral float-right" title="Kernel" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>


<a href="interview.html" class="btn btn-neutral" title="Interview Questions" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>

</div>


<hr/>

<div role="contentinfo">
<p>
&copy; Copyright 2015, Pants.

</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

</footer>

</div>
</div>

</section>

</div>





<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'1',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>





<script type="text/javascript" src="_static/js/theme.js"></script>




<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>


</body>
</html>
199 _build/html/ipsuite.html
View file
@@ -0,0 +1,199 @@


<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<title>IP SUITE &mdash; PantsNotes 1 documentation</title>















<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />





<link rel="top" title="PantsNotes 1 documentation" href="index.html"/>
<link rel="prev" title="Welcome to PantsNotes’s documentation!" href="index.html"/>


<script src="_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav" role="document">

<div class="wy-grid-for-nav">


<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">



<a href="index.html" class="icon icon-home"> PantsNotes



</a>




<div class="version">
1
</div>




<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>


</div>

<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



<ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="">IP SUITE</a></li>
</ul>



</div>
</div>
</nav>

<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">PantsNotes</a>
</nav>



<div class="wy-nav-content">
<div class="rst-content">






<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> &raquo;</li>

<li>IP SUITE</li>
<li class="wy-breadcrumbs-aside">


<a href="_sources/ipsuite.txt" rel="nofollow"> View page source</a>


</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">

<div class="section" id="ip-suite">
<h1>IP SUITE<a class="headerlink" href="#ip-suite" title="Permalink to this headline"></a></h1>
</div>


</div>
</div>
<footer>

<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">


<a href="index.html" class="btn btn-neutral" title="Welcome to PantsNotes’s documentation!" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>

</div>


<hr/>

<div role="contentinfo">
<p>
&copy; Copyright 2015, Pants.

</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

</footer>

</div>
</div>

</section>

</div>





<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'1',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>





<script type="text/javascript" src="_static/js/theme.js"></script>




<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>


</body>
</html>
205 _build/html/kernel.html
View file
@@ -0,0 +1,205 @@


<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<title>Kernel &mdash; PantsNotes 1 documentation</title>















<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />





<link rel="top" title="PantsNotes 1 documentation" href="index.html"/>
<link rel="next" title="Protocols" href="protocols.html"/>
<link rel="prev" title="IP" href="ip.html"/>


<script src="_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav" role="document">

<div class="wy-grid-for-nav">


<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">



<a href="index.html" class="icon icon-home"> PantsNotes



</a>




<div class="version">
1
</div>




<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>


</div>

<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="interview.html">Interview Questions</a></li>
<li class="toctree-l1"><a class="reference internal" href="ip.html">IP</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="">Kernel</a></li>
<li class="toctree-l1"><a class="reference internal" href="protocols.html">Protocols</a></li>
</ul>



</div>
</div>
</nav>

<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">PantsNotes</a>
</nav>



<div class="wy-nav-content">
<div class="rst-content">






<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> &raquo;</li>

<li>Kernel</li>
<li class="wy-breadcrumbs-aside">


<a href="_sources/kernel.txt" rel="nofollow"> View page source</a>


</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">

<div class="section" id="kernel">
<h1>Kernel<a class="headerlink" href="#kernel" title="Permalink to this headline"></a></h1>
</div>


</div>
</div>
<footer>

<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">

<a href="protocols.html" class="btn btn-neutral float-right" title="Protocols" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>


<a href="ip.html" class="btn btn-neutral" title="IP" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>

</div>


<hr/>

<div role="contentinfo">
<p>
&copy; Copyright 2015, Pants.

</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

</footer>

</div>
</div>

</section>

</div>





<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'1',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>





<script type="text/javascript" src="_static/js/theme.js"></script>




<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>


</body>
</html>
7 _build/html/objects.inv
View file
@@ -0,0 +1,7 @@
# Sphinx inventory version 2
# Project: PantsNotes
# Version: 1
# The remainder of this file is compressed using zlib.
xڍ�MO�0 @����Z$��M�@��6� ��K�6Z>�$��'k�v�E�{�c+�� ��L�� ���IqwO1��V��S<�"��XϋM�`h�Rw2���7\tZ^��֚S�ɚI��ܵ)w��>#[4�6Er�d9����� O=�E
�4��M��B�D��D��F�D��q�R�jBF_�9�d�E �ǡY)e�2���oK*4=&���qdс�l�*�Wp��f~�\1�:1,���3��3�˔�o��|����%H�~)մyL��
�N~���Bw����U]�1��7B�h�_ɕ�^r����1���7C�o}�-T�}��j�
202 _build/html/protocols.html
View file
@@ -0,0 +1,202 @@


<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<title>Protocols &mdash; PantsNotes 1 documentation</title>















<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />





<link rel="top" title="PantsNotes 1 documentation" href="index.html"/>
<link rel="prev" title="Kernel" href="kernel.html"/>


<script src="_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav" role="document">

<div class="wy-grid-for-nav">


<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">



<a href="index.html" class="icon icon-home"> PantsNotes



</a>




<div class="version">
1
</div>




<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>


</div>

<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="interview.html">Interview Questions</a></li>
<li class="toctree-l1"><a class="reference internal" href="ip.html">IP</a></li>
<li class="toctree-l1"><a class="reference internal" href="kernel.html">Kernel</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="">Protocols</a></li>
</ul>



</div>
</div>
</nav>

<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">PantsNotes</a>
</nav>



<div class="wy-nav-content">
<div class="rst-content">






<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> &raquo;</li>

<li>Protocols</li>
<li class="wy-breadcrumbs-aside">


<a href="_sources/protocols.txt" rel="nofollow"> View page source</a>


</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">

<div class="section" id="protocols">
<h1>Protocols<a class="headerlink" href="#protocols" title="Permalink to this headline"></a></h1>
</div>


</div>
</div>
<footer>

<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">


<a href="kernel.html" class="btn btn-neutral" title="Kernel" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>

</div>


<hr/>

<div role="contentinfo">
<p>
&copy; Copyright 2015, Pants.

</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

</footer>

</div>
</div>

</section>

</div>





<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'1',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>





<script type="text/javascript" src="_static/js/theme.js"></script>




<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>


</body>
</html>
207 _build/html/search.html
View file
@@ -0,0 +1,207 @@


<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<title>Search &mdash; PantsNotes 1 documentation</title>















<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />





<link rel="top" title="PantsNotes 1 documentation" href="index.html"/>


<script src="_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav" role="document">

<div class="wy-grid-for-nav">


<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">



<a href="index.html" class="icon icon-home"> PantsNotes



</a>




<div class="version">
1
</div>




<div role="search">
<form id="rtd-search-form" class="wy-form" action="#" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>


</div>

<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



<ul>
<li class="toctree-l1"><a class="reference internal" href="TODO-interview.html">Interview Material</a></li>
<li class="toctree-l1"><a class="reference internal" href="cassandra.html">Cassandra</a></li>
<li class="toctree-l1"><a class="reference internal" href="dynamo.html">DYNAMO</a></li>
<li class="toctree-l1"><a class="reference internal" href="filesystems.html">Filesystems</a></li>
</ul>



</div>
</div>
</nav>

<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">PantsNotes</a>
</nav>



<div class="wy-nav-content">
<div class="rst-content">






<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> &raquo;</li>

<li></li>
<li class="wy-breadcrumbs-aside">

</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">

<noscript>
<div id="fallback" class="admonition warning">
<p class="last">
Please activate JavaScript to enable the search
functionality.
</p>
</div>
</noscript>


<div id="search-results">

</div>

</div>
</div>
<footer>


<hr/>

<div role="contentinfo">
<p>
&copy; Copyright 2015, Pants.

</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

</footer>

</div>
</div>

</section>

</div>





<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'1',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/searchtools.js"></script>





<script type="text/javascript" src="_static/js/theme.js"></script>




<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>

<script type="text/javascript">
jQuery(function() { Search.loadIndex("searchindex.js"); });
</script>

<script type="text/javascript" id="searchindexloader"></script>



</body>
</html>
1 _build/html/searchindex.js
View file

Large diffs are not rendered by default.

357 conf.py
View file
@@ -0,0 +1,357 @@
# -*- coding: utf-8 -*-
#
# PantsNotes documentation build configuration file, created by
# sphinx-quickstart on Fri Nov 27 06:30:43 2015.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.

import sys
import os
import shlex
import sphinx_rtd_theme

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.'))

# -- General configuration ------------------------------------------------

# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.pngmath',
]

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'

# The encoding of source files.
#source_encoding = 'utf-8-sig'

# The master toctree document.
master_doc = 'index'

# General information about the project.
project = u'PantsNotes'
copyright = u'2015, Pants'
author = u'Pants'

# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '1'
# The full version, including alpha/beta/rc tags.
release = '1'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None

# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']

# The reST default role (used for this markup: `text`) to use for all
# documents.
#default_role = None

# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True

# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True

# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False

# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'

# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []

# If true, keep warnings as "system message" paragraphs in the built documents.
#keep_warnings = False

# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False


# -- Options for HTML output ----------------------------------------------

# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'sphinx_rtd_theme'

# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}

# Add any paths that contain custom themes here, relative to this directory.
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]

# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None

# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None

# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None

# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#html_extra_path = []

# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'

# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True

# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}

# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}

# If false, no module index is generated.
#html_domain_indices = True

# If false, no index is generated.
#html_use_index = True

# If true, the index is split into individual pages for each letter.
#html_split_index = False

# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True

# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True

# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True

# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''

# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None

# Language to be used for generating the HTML full-text search index.
# Sphinx supports the following languages:
# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
#html_search_language = 'en'

# A dictionary with options for the search language support, empty by default.
# Now only 'ja' uses this config value
#html_search_options = {'type': 'default'}

# The name of a javascript file (relative to the configuration directory) that
# implements a search results scorer. If empty, the default will be used.
#html_search_scorer = 'scorer.js'

# Output file base name for HTML help builder.
htmlhelp_basename = 'PantsNotesdoc'

# -- Options for LaTeX output ---------------------------------------------

latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#'papersize': 'letterpaper',

# The font size ('10pt', '11pt' or '12pt').
#'pointsize': '10pt',

# Additional stuff for the LaTeX preamble.
#'preamble': '',

# Latex figure (float) alignment
#'figure_align': 'htbp',
}

# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'PantsNotes.tex', u'PantsNotes Documentation',
u'Pants', 'manual'),
]

# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None

# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False

# If true, show page references after internal links.
#latex_show_pagerefs = False

# If true, show URL addresses after external links.
#latex_show_urls = False

# Documents to append as an appendix to all manuals.
#latex_appendices = []

# If false, no module index is generated.
#latex_domain_indices = True


# -- Options for manual page output ---------------------------------------

# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'pantsnotes', u'PantsNotes Documentation',
[author], 1)
]

# If true, show URL addresses after external links.
#man_show_urls = False


# -- Options for Texinfo output -------------------------------------------

# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'PantsNotes', u'PantsNotes Documentation',
author, 'PantsNotes', 'One line description of project.',
'Miscellaneous'),
]

# Documents to append as an appendix to all manuals.
#texinfo_appendices = []

# If false, no module index is generated.
#texinfo_domain_indices = True

# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'

# If true, do not generate a @detailmenu in the "Top" node's menu.
#texinfo_no_detailmenu = False


# -- Options for Epub output ----------------------------------------------

# Bibliographic Dublin Core info.
epub_title = project
epub_author = author
epub_publisher = author
epub_copyright = copyright

# The basename for the epub file. It defaults to the project name.
#epub_basename = project

# The HTML theme for the epub output. Since the default themes are not optimized
# for small screen space, using the same theme for HTML and epub output is
# usually not wise. This defaults to 'epub', a theme designed to save visual
# space.
#epub_theme = 'epub'

# The language of the text. It defaults to the language option
# or 'en' if the language is not set.
#epub_language = ''

# The scheme of the identifier. Typical schemes are ISBN or URL.
#epub_scheme = ''

# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#epub_identifier = ''

# A unique identification for the text.
#epub_uid = ''

# A tuple containing the cover image and cover page html template filenames.
#epub_cover = ()

# A sequence of (type, uri, title) tuples for the guide element of content.opf.
#epub_guide = ()

# HTML files that should be inserted before the pages created by sphinx.
# The format is a list of tuples containing the path and title.
#epub_pre_files = []

# HTML files shat should be inserted after the pages created by sphinx.
# The format is a list of tuples containing the path and title.
#epub_post_files = []

# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']

# The depth of the table of contents in toc.ncx.
#epub_tocdepth = 3

# Allow duplicate toc entries.
#epub_tocdup = True

# Choose between 'default' and 'includehidden'.
#epub_tocscope = 'default'

# Fix unsupported image types using the Pillow.
#epub_fix_images = False

# Scale large images.
#epub_max_image_width = 0

# How to display URL addresses: 'footnote', 'no', or 'inline'.
#epub_show_urls = 'inline'

# If false, no index is generated.
#epub_use_index = True
2 dynamo.rst
View file
@@ -1,6 +1,6 @@
.. _dynamo:

DYNAMO
Dynamo
======

A highly available key/value store. Provides a place for various services (shopping cart, best seller lists, customer preferences, session management, product catalog, etc) to store their data in a "primary key only," non-relational way.
52 filesystems.rst
View file
@@ -0,0 +1,52 @@
Filesystems
===========

General
-------

A filesystem is something which defines how data is stored and retrieved.

A filesystem typically stores all the metadata associated with the file - including the file name, length, and location within the directory hierarchy - separate from the contents of the file.

Most filesystems store the names of all the files which are in one directory in one place - the directory table for that directory - which is often stored like any other file. Many filesystems put only some of the metadata for a file in the directory table, and the rest of the metadata for that file in a completely separate structure, such as the inode.

Most filesystems also store metadata not associated with any one particular file. Such metadata includes information about unused regions - free space bitmap, or a B-Tree structure - and information about bad sectors. Often such information about an allocation group is stored inside the allocation group itself. More on this information in the filesystem-specific sections.

XFS
---

XFS is a high performance *journaling* file system. It generally excels in the execution of parallel input/output operations, due to its *allocation group* based design. XFS is an *extent-based* filesystem.

XFS is internally partitioned into **allocation groups**, which are virtual storage regions of fixed size. Any files and directories that you create can span multiple allocation groups. Each allocation group manages its own journal, its own inodes, and its free space independently of other allocation groups. This is where it gains its advantage in both scalability and parallelism. If the file system spans many physical devices, allocation groups can optimize throughput by taking advantage of the underlying separation of channels to the storage components.

XFS implements **journaling** for metadata operations. XFS records file system updates asynchronously to a circular buffer before it can commit the actual data updates to disk. The journal can either be located internally in the data section of the file system, or externally on a separate device to reduce contention for disk access. If the system crashes or loses power, the journal is read when the filesystem is remounted, and replays any pending metadata operations. The speed of this recovery is not dependent on the size of the filesystem.

To reduce fragmentation and file scattering, each file's blocks in XFS can have variable length **extents**


Whilst many other filesystems manage space allocation with one or more block oriented bitmaps (see _`NTFS`), in XFS these structures are replaced with an extent oriented structure consisting of a pair of B+ trees (see TODO for more information on B-trees) for each file system allocation group.


Ref:
http://linux-xfs.sgi.com/projects/xfs/papers/xfs_white/xfs_white_paper.html
https://en.wikipedia.org/wiki/XFS

NTFS
----

We generally don't care about NTFS, but we did mention "free space bitmap" in the general section. You may also remember having to "defrag" your hard drive and were wondering why. While ext* also use free space bitmaps, NTFS's implementation gives the clearest explanation of this phenomenon.

NTFS uses a special $BitMap file to keep track of all the used and unused "clusters" (ntfs nomenclature for logical blocks: anywhere from 512b to 64KB) on an NTFS volume. It's a single file (a type of Free Space Bitmap as mentioned above) which contains a mapping of the entire volume, where when a file takes up space on the NTFS volume, the corresponding location in the $BitMap file is marked as used (11111...). When data is removed, the corresponding map location is marked 0 as unused. When an OS is looking to write out a file, it scans this map file to see where the next available logical block is in which it can place this data, writes out a block, then continues on looking for the next free space available such that it can finish writing the file (if said file is larger than a single block). Even if the OS manages to write out the whole file in contiguous blocks, another file will be written immediately after it. If the original file grows at all, it becomes fragmented. We can see with this approach how quickly fragmentation can occur, and how unruly a $BitMap file can get on a large filesystem (64MB on a 2TB drive).

Here we can see the bitmap from an NTFS volume with a single 1MB file added:

.. image:: media/NTFS-BitMap.jpg
:alt: NTFS $BitMap with 1MB file added
:align: center

Note that ext3/4 also use bitmaps, but in regards to fragmentation they are a little bit smarter. Instead of placing multiple files near each other on the hard disk, ext3/4 will scatter different files all over the disk, leaving a large amount of free space between them. They will also choose free blocks around the (growing) file when writing out new data, rather than just the first free block they find. If a file does end up fragmenting, the file system will attempt to move the files around to reduce fragmentation (in ext4 at least). It does this automatically, without the need to run any utility. It can do this so long as the filesystem is not super full (80%+).


Ref:
https://whereismydata.wordpress.com/2009/06/01/forensics-what-is-the-bitmap/

26 index.rst
View file
@@ -0,0 +1,26 @@
.. PantsNotes documentation master file, created by
sphinx-quickstart on Fri Nov 27 06:30:43 2015.
Welcome to PantsNotes's documentation!
======================================

This is a spot where various sysadmin related notes can be read in one place.

Contents:

.. Add paths/to/file (.rst not needed) to the toc
OR you can just glob all files in current dir
.. toctree::
:maxdepth: 2
:glob:

*


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

BIN +279 KB media/NTFS-BitMap.jpg
View file
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.