application-suite.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V5.0//EN"
                 "http://www.docbook.org/xml/5.0/dtd/docbook.dtd" [
  <!ENTITY % local SYSTEM "local-entities.ent">
  <!ENTITY % custom SYSTEM "custom-entities.ent">
  %local;
  %custom;
]>
<chapter xml:id="chapter.application-suite" 
         xmlns="http://docbook.org/ns/docbook" version="5.0"
         xmlns:product="http://www.carrot2.org"
         xmlns:xi="http://www.w3.org/2001/XInclude">
  <title>Tools and APIs</title>
  <subtitle>&PROD; distribution suite</subtitle>

  <para>
    &PROD; comes with a suite of tools and APIs that you can use to
    quickly set up clustering on your own data, tune clustering results,
    call &PROD; clustering from your Java or C# code or
    access &PROD; clustering as a remote service. 
  </para>
  
  <para>
    &PROD; distribution contains the following elements:
  </para>
  
  <itemizedlist>
    <listitem>
      <formalpara>
        <title>&DCW;</title>
        <para>
          which is a standalone GUI application you can use to experiment with &PROD;
          clustering on data from common search engines or your own data,
        </para>
      </formalpara>
    </listitem>

    <listitem>
      <formalpara>
        <title>&JA;</title>
        <para>
          for calling &PROD; document clustering from your Java code,
        </para>
      </formalpara>
    </listitem>

    <listitem>
      <formalpara>
        <title>&CSA;</title>
        <para>
          for calling &PROD; document clustering from your C# or .NET code,
        </para>
      </formalpara>
    </listitem>

    <listitem>
      <formalpara>
        <title>&DCS;</title>
        <para>
          which exposes &PROD; clustering as a REST service,
        </para>
      </formalpara>
    </listitem>

    <listitem>
      <formalpara>
        <title>&CLI;</title>
        <para>
          applications which allow invoking &PROD; clustering from command line,
        </para>
      </formalpara>
    </listitem>

    <listitem role="webapp">
      <formalpara>
        <title>&WA;</title>
        <para>
          which exposes &PROD; clustering as a web application for end users.
        </para>
      </formalpara>
    </listitem>
  </itemizedlist>

  <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="application-suite-intro-suffix.xml" /> 

  <section xml:id="section.workbench">
    <title>&DCW;</title>

    <para>
      &DCW; is a standalone GUI application you can use to experiment with &PROD;
      clustering on data from common search engines or your own data. 
    </para>
    
    <para>
      You can use &DCW; to:
    </para>
    
    <itemizedlist>
      <listitem>
        <para>
          Quickly test &PROD; clustering with your own data. Please see <xref linkend="chapter.getting-started" />
          for instructions for the most common scenarios.
        </para>
      </listitem>
      
      <listitem>
        <para>
          Fine tune &PROD; clustering algorithms' settings to work best with your specific data.
          Please see <xref linkend="chapter.tuning" /> for more details.
        </para>
      </listitem>
      
      <listitem>
        <para>Run simple performance benchmarks using different settings to predict maximum clustering throughput
        on a single machine. Please see <xref linkend="section.workbench.benchmarking" /> for details.</para>
      </listitem>      
    </itemizedlist>

    <para>
      &DCW; features include:
    </para>

    <itemizedlist>
      <listitem>
        <formalpara>
          <title>Various document sources included.</title>
          <para>
            &DCW; can fetch and cluster documents from a number of sources, including major search
            engines, indexing engines (Lucene, Solr) as well as generic
            XML feeds and files.
          </para>
        </formalpara>
      </listitem>
      
      <listitem>
        <formalpara>
          <title>Live tuning of clustering algorithm attributes.</title>
          <para>
            &DCW; enables modifying clustering algorithm's attributes and observing the
            results in real time.
          </para>
        </formalpara>
      </listitem>

      <listitem>
        <formalpara>
          <title>Performance benchmarking.</title>
          <para>
            &DCW; can run simple performance benchmarks of &PROD; clustering algorithms.
          </para>
        </formalpara>
      </listitem>
      
      <listitem>
        <formalpara>
          <title>Attractive visualizations.</title>
          <para>
            &DCW; comes with two visualizations of the cluster structure, one developed
            within the &PROD; project and another one from Aduna Software.
          </para>
        </formalpara>
      </listitem>

      <listitem>
        <formalpara>
          <title>Modular architecture and extendability.</title>
          <para>
            &DCW; is based on Eclipse Rich Client Platform, which makes it easily
            extendable.
          </para>
        </formalpara>
      </listitem>
    </itemizedlist>

    <figure>
      <title>&DCW; screenshot</title>
      <mediaobject>
        <imageobject role="html">
          <imagedata format="PNG" fileref="img/workbench.png" />
        </imageobject>
      </mediaobject>  
    </figure>
    
    <section xml:id="section.workbench.installation" role="notoc">
      <title>Installation and running</title>
    
      <para>
        To run &DCW;:
      </para>
      
      <orderedlist>
        <listitem>
          <para>Download and install <link xlink:href="http://java.sun.com/javase/downloads/index.jsp">Java Runtime Environment</link> (version 1.7.0 or newer) if you have not done so.</para>
        </listitem>
      
        <listitem role="private">
          <para>
            Download &DCW; <product:workbench-download-link os="win32" wm="win32">Windows binaries</product:workbench-download-link>
            or <product:workbench-download-link os="linux" wm="gtk">Linux binaries</product:workbench-download-link>
            and extract the archive to some local disk location.
          </para>
        </listitem>
         
        <xi:include href="application-suite-dcw-download.xml">
          <xi:fallback />
        </xi:include>
        
        <listitem>
          <para>Run <command>&PRODID;-workbench.exe</command> (Windows) or <command>&PRODID;-workbench</command> (Linux).</para>
        </listitem>
      </orderedlist>
    </section>
  </section>
  
  <section xml:id="section.java-api">
    <title>&JA;</title>
    
    <para>
      The &JA; package contains &PROD; JAR files along with all dependencies, JavaDoc API reference
      and Java code examples. You can use this package to integrate &PROD; clustering into your Java software. Please
      see <xref linkend="section.integration.compiling-java-program-with-carrot2" /> and
      <xref linkend="section.integration.seting-up-c2-project-in-eclipse" /> for instructions.
    </para>
  </section>
  
  <section xml:id="section.csharp-api">
    <title>&CSA;</title>
    
    <para>
      The &CSA; package contains all DLL libraries required to run &PROD;, 
      C# API reference and code examples. You can use this package to integrate &PROD; 
      clustering into your C# / .NET software. Please see 
      <xref linkend="section.integration.compiling-csharp-program-with-carrot2" /> for instructions.
    </para>
  </section>
    
  <section xml:id="section.dcs">
    <title>&DCS;</title>
    
    <para>
      &DCS; (DCS) exposes &PROD; clustering as a REST service. 
      It can cluster documents from an external source (e.g. a search engine) or 
      documents provided directly as an XML stream and returns results in 
      <link linkend="section.architecture.input-xml">XML</link> or 
      <link linkend="section.architecture.output-json">JSON</link> formats. 
    </para>
    
    <para>
      You can use &DCS; to:
    </para>
    
    <itemizedlist>
      <listitem>
        <para>Integrate &PROD; with your non-Java software.</para>
      </listitem>
      
      <listitem>
        <para>
          Build a high-throughput document clustering system by setting up a number of
          load-balanced instances of the DCS.
        </para> 
      </listitem>
    </itemizedlist>
    
    <para>
      &DCS; features include:
    </para>

    <itemizedlist>
      <listitem>
        <formalpara>
          <title>XML and JSON response formats.</title>
          <para>
            &DCS; can return results both in XML and <link linkend="section.architecture.output-json">JSON</link> 
            formats. JSON-P (with callback) is also supported.
          </para>
        </formalpara>
      </listitem>
        
      <listitem>
        <formalpara>
          <title>Various document sources included.</title>
          <para>
            &DCS; can fetch and cluster documents from a large number of sources, 
            including major search engines and indexing engines (Lucene, Solr).
          </para>
        </formalpara>
      </listitem>
        
      <listitem>
        <formalpara>
          <title>Direct XML feed.</title>
          <para>
            &DCS; can cluster documents fed directly in a simple XML format.
          </para>
        </formalpara>
      </listitem>
        
      <listitem>
        <formalpara>
          <title>PHP and C# examples included.</title>
          <para>
            &DCS; ships with ready-to-use examples of calling &PROD; DCS services from PHP 
            (version 5), C#, Ruby, Java and <command>curl</command>.
          </para>
        </formalpara>
      </listitem>
        
      <listitem>
        <formalpara>
          <title>Quick start screen.</title>
          <para>
            A simple quick start screen will let you make your first DCS request straight
            from your browser.
          </para>
        </formalpara>
      </listitem>
    </itemizedlist>
    
    <figure>
      <title>&DCS; quick start screen</title>
      <mediaobject>
        <imageobject role="html">
          <imagedata format="PNG" fileref="img/dcs.png" />
        </imageobject>
      </mediaobject>  
    </figure>
    
    <section xml:id="section.dcs.installation" role="notoc">
      <title>Installation and running</title>
    
      <para>To run &DCS;:</para>
      
      <orderedlist>
        <listitem>
          <para>Download and install <link xlink:href="http://java.sun.com/javase/downloads/index.jsp">Java Runtime Environment</link> (version 1.7.0 or newer) if you have not done so.</para>
        </listitem>
      
        <listitem role="private">
          <para>
            Download <product:dcs-download-link>&DCS; binaries</product:dcs-download-link> and
            extract the archive to some local disk location.
          </para>
        </listitem>
         
        <xi:include href="application-suite-dcs-download.xml">
          <xi:fallback />
        </xi:include>
        
        <listitem>
          <para>Run <command>dcs.cmd</command> (Windows) or <command>dcs.sh</command> (Linux).</para>
        </listitem>
        
        <listitem>
          <para>Point your browser to <uri>http://localhost:8080</uri> for further instructions.</para>
        </listitem>
        
        <listitem>
          <para>
            See the <filename>examples/</filename> directory in the distribution archive
            for PHP, C#, Ruby and Java code examples. <phrase role="carrot2">You can also 
            <link linkend="section.advanced-topics.using-dcs-with-curl">invoke DCS clustering
            using the <command>curl</command> command</link>.</phrase>
          </para>
        </listitem>
      </orderedlist>
      
      <tip>
        <para>
          If you need to start the DCS at a port different than 8080, you can use the 
          <option>-port</option> option:
        </para>
        <programlisting><![CDATA[dcs -port 9090]]></programlisting>
      </tip>
      
      <tip>
        <para>
          To deploy the DCS in an external servlet container, such as Apache Tomcat, use
          the <filename>&PRODID;-dcs.war</filename> file from the <filename>war/</filename>
          folder of the DCS distribution.
        </para>
      </tip>
    </section>
  </section>

  <section xml:id="section.webapp" role="webapp">
    <title>&WA;</title>
    
    <para>
      &WA; exposes &PROD; clustering as a web application for end users. It allows users
      to browse clusters using a conventional tree view, but also in an attractive visualization.
    </para>
    
    <para>
      &DCS; features include:
    </para>

    <itemizedlist>
      <listitem>
        <formalpara>
          <title>Two cluster views.</title>
          <para>
            &WA; offers two views of the clusters generated by &PROD;: conventional tree
            view and spatial visualizations.
          </para>
        </formalpara>
      </listitem>

      <listitem>
        <formalpara>
          <title>All &PROD; document sources and algorithms included.</title>
          <para>
            &WA; contains a large number of document sources, including major search
            engines. Optionally, further document sources can be added, such as
            Lucene or Solr ones. It also contains all &PROD;'s clustering algorithms.
          </para>
        </formalpara>
      </listitem>

      <listitem>
        <formalpara>
          <title>XSLT and JavaScript-based presentation layer.</title>
          <para>
            Look &amp; feel of the &WA; can be easily changed by editing a number of XSLT 
            style sheets. All common style sheets and JavaScripts can be re-used when
            implementing a new look &amp; feel. 
          </para>
        </formalpara>
      </listitem>

      <listitem>
        <formalpara>
          <title>High-performance front-end.</title>
          <para>
            The front-end of the &WA; has been optimized for fast loading by using such techniques as
            JavaScript and CSS merging and minification, as well as using 
            <link xlink:href="http://smartsprites.osinski.name">CSS sprites</link>. 
          </para>
        </formalpara>
      </listitem>
    </itemizedlist>    
    
    <figure>
      <title>&WA; results screen</title>
      <mediaobject>
        <imageobject role="html">
          <imagedata format="PNG" fileref="img/webapp.png" />
        </imageobject>
      </mediaobject>  
    </figure>

    <section xml:id="section.webapp.installation" role="notoc">
      <title>Installation and running</title>
    
      <para>To run &WA;:</para>
      
      <orderedlist>
        <listitem>
          <para>Make sure you have access to a Servlet API 2.4 compliant container, such as Apache Tomcat.</para>
        </listitem>
      
        <listitem>
          <para>Download <product:webapp-download-link>&WA; WAR file</product:webapp-download-link>.</para>
        </listitem>
        
        <listitem>
          <para>Deploy the WAR file to your servlet container.</para>
        </listitem>
      </orderedlist>
    </section>      
  </section>
      
  <section xml:id="section.cli">
    <title>&CLI;</title>
    
    <para>
      &CLI; (CLI) is a set of applications that allow invoking &PROD; clustering
      from the command line. Currently, the only available CLI application is &CLIBP;, which
      performs &PROD; clustering on one or more files in the &C2; <link linkend="section.architecture.input-xml">XML format</link>
      and saves the results as <link linkend="section.architecture.output-xml">XML</link> 
      or <link linkend="section.architecture.output-json">JSON</link>. Apart from 
      clustering large number of documents sets at one time, you can use the &CLIBP; 
      to integrate &PROD; with your non-Java applications.
    </para>
    
    <section xml:id="section.cli.bp.installation" role="notoc">
      <title>Installation and running</title>
    
      <para>To run &CLIBP;:</para>
      
      <orderedlist>
        <listitem>
          <para>Download and install <link xlink:href="http://java.sun.com/javase/downloads/index.jsp">Java Runtime Environment</link> (version 1.7.0 or newer) if you have not done so.</para>
        </listitem>
      
        <listitem role="private">
          <para>
            Download <product:cli-download-link>&CLI; binaries</product:cli-download-link> and
            extract the archive to some local disk location.
          </para>
        </listitem>
        
        <xi:include href="application-suite-cli-download.xml">
          <xi:fallback />
        </xi:include>

        <listitem>
          <para>
            Run <command>batch.cmd</command> (Windows) or <command>batch.sh</command> 
            (Linux) for an overview of the syntax. The &CLIBP; ships with two example
            input data sets located in the <filename>input/</filename> directory.
            Below is a list of some common example invocations.
          </para>
          
          <itemizedlist>
            <listitem>
              <para>To cluster one or more input files, specify their paths:</para>
              
              <programlisting><![CDATA[batch input/data-mining.xml input/seattle.xml]]></programlisting>
              
              <para>
                Clustering will be performed using the default clustering algorithm
                and the results in the XML format will be saved to the <filename>output</filename>
                directory relative to the current working directory.
              </para>
            </listitem>
          
            <listitem>
              <para>You can also cluster files from one or more directories:</para>
              
              <programlisting><![CDATA[batch input/]]></programlisting>
              
              <para>
                Each directory will be processed recursively, i.e. including subdirectories.
                For each specified input directory, a corresponding directory with results
                will be created in the output directory.
              </para>
            </listitem>
          
            <listitem>
              <para>To save results in the non-default directory, use the <option>-o</option> option:</para>
              
              <programlisting><![CDATA[batch input/ -o results]]></programlisting>
            </listitem>
          
            <listitem>
              <para>To repeat the input documents on the output, use the <option>-d</option> option:</para>
              
              <programlisting><![CDATA[batch input/ -d]]></programlisting>
            </listitem>
          
            <listitem>
              <para>To save the results in <link linkend="section.architecture.output-json">JSON</link>, use the <option>-f JSON</option> option:</para>
              
              <programlisting><![CDATA[batch input/ -f JSON]]></programlisting>
            </listitem>
          
            <listitem>
              <para>To use a different clustering algorithm, use the <option>-a</option> 
                option followed by the identifier of the algorithm:</para>
              
              <programlisting><![CDATA[batch input/ -a url]]></programlisting>
              
              <para>To see the list of available algorithm identifiers, run the application without arguments.</para>
            </listitem>

          
            <listitem>
              <para>In case of processing errors, you can use the <option>-v</option>
                option to see detailed messages and stack traces.</para>
            </listitem>
          </itemizedlist>
        </listitem>
      </orderedlist>
    </section>
  </section>
  
  <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="application-suite-solr.xml" />
  
  <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="application-suite-es.xml" />
   
  <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="application-suite-sections-suffix.xml" /> 

</chapter>