diff --git a/demo/demo_config.py b/demo/demo_config.py index c88a46c..ecd53ec 100644 --- a/demo/demo_config.py +++ b/demo/demo_config.py @@ -45,5 +45,3 @@ } APPLICATIONS = [app_hash_cracker_1] -#APPLICATIONS = [app_mc_pi_1, ] -#APPLICATIONS = [app_mc_pi_1, app_hash_cracker_1] diff --git a/doc/_static/Makefile b/doc/_static/Makefile new file mode 100644 index 0000000..674a1b3 --- /dev/null +++ b/doc/_static/Makefile @@ -0,0 +1,29 @@ +PROJECT_NAME = monte_carlo_pi +LIB = $(PROJECT_NAME).js +# location of coffee file and to-be-compiled js files. +LIBDIR = js + +# js target +TARGETS = $(LIBDIR)/$(LIB) +CLEAN_TARGETS = + +# if make has been called recursively, add remote targets +ifeq ($(origin PJ_RES_DIR), environment) +TARGETS += remote +CLEAN_TARGETS += clean_remote +endif + +all: $(TARGETS) + +$(LIBDIR)/$(LIB): $(LIBDIR)/$(PROJECT_NAME).coffee + coffee --bare -c $(LIBDIR)/$(PROJECT_NAME).coffee + +remote: + mkdir -p $(PJ_RES_DIR)/$(PROJECT_NAME) + cp $(LIBDIR)/*.js $(PJ_RES_DIR)/$(PROJECT_NAME) + +clean: $(CLEAN_TARGETS) + rm -f $(LIBDIR)/$(PROJECT_NAME).js + +clean_remote: + rm -rf $(PJ_RES_DIR)/$(PROJECT_NAME) diff --git a/doc/_static/alea.js b/doc/_static/alea.js index 4f1b94f..2a45d72 100644 --- a/doc/_static/alea.js +++ b/doc/_static/alea.js @@ -1,4 +1,29 @@ -// From http://baagoe.com/en/RandomMusings/javascript/ +// Copyright (C) 2010 by Johannes Baagøe >> 0; + h -= n; + h *= n; + n = h >>> 0; + h -= n; + n += h * 0x100000000; // 2^32 + } + return (n >>> 0) * 2.3283064365386963e-10; // 2^-32 + }; + + mash.version = 'Mash 0.9'; + return mash; +} + + function Alea() { return (function(args) { // Johannes Baagøe , 2010 diff --git a/doc/auto_filters.rst b/doc/auto_filters.rst index cd43f2c..27829e8 100644 --- a/doc/auto_filters.rst +++ b/doc/auto_filters.rst @@ -70,7 +70,7 @@ class:: '__next__' : [depleted_guard, ], 'normalize' : [ignore_null_result, ], 'store_result' : [ignore_null_result, ], - } + } Here, :ref:`auto_filter ` is a binary-flag attribute which defines the auto-decorating process behaviour. @@ -85,7 +85,7 @@ the ``auto_filter`` attribute with desired value, e.g.:: auto_filter = CONFIG_FILTERS The :ref:`auto_filters ` attribute defines the filters -bound to the methods of the class. In the example above the -both :meth:`Project.normalize` and :meth:`Project.store_result` methods +bound to the methods of the class. In the example above both +:meth:`Project.normalize` and :meth:`Project.store_result` methods are decorated by the :func:`ignore_null_result ` filter. diff --git a/doc/clientapi.rst b/doc/clientapi.rst index 0907fc8..682e63d 100644 --- a/doc/clientapi.rst +++ b/doc/clientapi.rst @@ -308,6 +308,7 @@ AJAX :param error: callback invoked in of request failure. + .. [1] http://en.wikipedia.org/wiki/Web_worker .. [2] http://www.w3schools.com/html5/html5_webworkers.asp diff --git a/doc/tutorial/client_side.rst b/doc/tutorial/client_side.rst index 61dd4c2..6318329 100644 --- a/doc/tutorial/client_side.rst +++ b/doc/tutorial/client_side.rst @@ -6,10 +6,8 @@ Step 3: Client-Side Code As you may already know, the client-side of Kaylee is written in `CoffeeScript `_. But there is nothing that prevents you from writing the project in pure JavaScript! -Yet, for simplicity and clarity we are going to use CoffeeScript. -Mastering the basics of CoffeeScript takes less than 10 minutes -and even without that, the project code should be simple and clear to -everybody. +Mastering the basics of CoffeeScript takes less than 30 minutes +and that is the language used in the tutorial application. The client-side code of Kaylee projects basically consists of two callbacks in ``pj`` namespace: :js:func:`pj.init(kl_config, app_config) ` and @@ -46,8 +44,8 @@ Here, we implement the Monte-Carlo PI algorithm and return the result via the :js:func:`task_completed() ` function from :ref:`Kaylee Worker API `. -If you are comfortable with the code above, just copy-paste it to -``js/monte_carlo_pi.coffee``. +If you are comfortable with the code above, copy-paste it to +``monte_carlo_pi/js/monte_carlo_pi.coffee``. Continue with :ref:`tutorial-server-side`. diff --git a/doc/tutorial/compiling.rst b/doc/tutorial/compiling.rst new file mode 100644 index 0000000..7c86dc7 --- /dev/null +++ b/doc/tutorial/compiling.rst @@ -0,0 +1,63 @@ +.. _tutorial-compiling: + +Step 6: Compiling the Application +================================= + +Makefile +-------- + +The Makefile builds ``monte_carlo_pi.js`` locally and (if built by the +master demo Makefile) copies the fields to to the ``build`` directory: + +.. code-block:: makefile + + PROJECT_NAME = monte_carlo_pi + LIB = $(PROJECT_NAME).js + # location of coffee file and to-be-compiled js files. + LIBDIR = js + + # js target + TARGETS = $(LIBDIR)/$(LIB) + CLEAN_TARGETS = + + # if make has been called recursively, add remote targets + ifeq ($(origin PJ_RES_DIR), environment) + TARGETS += remote + CLEAN_TARGETS += clean_remote + endif + + all: $(TARGETS) + + $(LIBDIR)/$(LIB): $(LIBDIR)/$(PROJECT_NAME).coffee + coffee --bare -c $(LIBDIR)/$(PROJECT_NAME).coffee + + remote: + mkdir -p $(PJ_RES_DIR)/$(PROJECT_NAME) + cp $(LIBDIR)/*.js $(PJ_RES_DIR)/$(PROJECT_NAME) + + clean: $(CLEAN_TARGETS) + rm -f $(LIBDIR)/$(PROJECT_NAME).js + + clean_remote: + rm -rf $(PJ_RES_DIR)/$(PROJECT_NAME) + + +Do **NOT** copy-paste the code above to because Makefiles syntax is based +on tab characters. `Download <../_static/Makefile>`_ the Makefile +and save it to ``monte_carlo_pi/Makefile``. + +It is also necessary to modify the master demo Makefile in order for the +build process find the project: + +.. code-block:: makefile + + # demo/Makefile + + PROJECTS = monte_carlo_pi # you can comment out the rest of the applications + + +Run ``make`` in demo directory to build the projects and collect the files +to the ``build`` directory. + + +Continue with :ref:`tutorial-running`. diff --git a/doc/tutorial/configuration.rst b/doc/tutorial/configuration.rst index ec5c5af..600943b 100644 --- a/doc/tutorial/configuration.rst +++ b/doc/tutorial/configuration.rst @@ -6,10 +6,13 @@ Step 5: Configuring the Application Kaylee has numerous ways of loading the configuration. One of the convenient ways is to load it from a ``.py`` file. The ``demo/demo_config.py`` file used for demo configuration can be as well used -to configure our application. +to configure the tutorial application. The applications are defined in a Python dictionary-like or JSON-like manner. First of all, the application needs a unique name and a description: -:: + +.. code-block:: python + + # demo/demo_config.py app_mc_pi_1 = { 'name' : 'mc_pi.1', @@ -18,8 +21,10 @@ manner. First of all, the application needs a unique name and a description: ... } -Now, lets add the project configuration. Note, that it is still an entry in -``app_mc_pi_1`` dict:: +Now, let's write the project configuration. Note, that we are still filling +the ``app_mc_pi_1`` dict: + +.. code-block:: python 'project' : { 'name' : 'MonteCarloPiProject', @@ -28,28 +33,29 @@ Now, lets add the project configuration. Note, that it is still an entry in 'alea_script' : '/static/js/projects/monte_carlo_pi/alea.js', 'random_points' : 1000000, 'tasks_count' : 10 - }, + }, 'storage' : { 'name' : 'MemoryPermanentStorage', - } - }, + } + }, .. module:: kaylee -The ``name`` entry indicated the Python class (Kaylee Project subclass) used -in this application (``MonteCarloPiProject``). -The ``config`` contains the keyword arguments passed to -``Project.__init__()``. The ``storage`` entry defines the -:py:class:`permanent storage ` to which the results are -saved. The :py:class:`MemoryPermanentStorage -` -is a simple in-memory storage from :ref:`kaylee.contrib `. - -The final piece of the configuration is the controller and its temporal results -storage. To keep things simple, lets use :py:class:`SimpleController -` from ``kaylee.contrib``. -Note that this is also the part of the application configuration and recides -inside the ``app_mc_pi_1`` dictionary.:: +The ``name`` entry indicates the Python class (:py:class:`kaylee.Project` +subclass) used in the application. The ``config`` contains the keyword +arguments passed to ``Project.__init__()``. The ``storage`` entry defines +the :py:class:`permanent storage ` to which the results +are saved. Here, :py:class:`MemoryPermanentStorage +` is a simple in-memory storage from +:ref:`kaylee.contrib ` package. + +The final piece of the configuration is the controller. To keep things +simple, let's use :py:class:`SimpleController +` from ``kaylee.contrib``. Note that this +is also the part of the application configuration and recides +inside the ``app_mc_pi_1`` dictionary: + +.. code-block:: python 'controller' : { 'name' : 'SimpleController', @@ -63,8 +69,4 @@ picks up the first available application). APPLICATIONS = [app_mc_pi_1 ] -That's it! You are now ready to launch the application to compute PI via -distributed calculations. - - -Continue with :ref:`tutorial-running`. +Continue with :ref:`tutorial-compiling`. diff --git a/doc/tutorial/index.rst b/doc/tutorial/index.rst index a9138d3..5d92367 100644 --- a/doc/tutorial/index.rst +++ b/doc/tutorial/index.rst @@ -17,9 +17,10 @@ http://github.com/BasicWolf/kaylee-tutorial-app :maxdepth: 1 introduction - structure requirements + structure client_side server_side configuration + compiling running diff --git a/doc/tutorial/introduction.rst b/doc/tutorial/introduction.rst index d98137d..b9e3ffc 100644 --- a/doc/tutorial/introduction.rst +++ b/doc/tutorial/introduction.rst @@ -61,4 +61,4 @@ Of course that can be done on your own computer! But this is just a small problem enough to learn building Kaylee apps :) -Continue with :ref:`tutorial-project-structure`. +Continue with :ref:`tutorial-requirements`. diff --git a/doc/tutorial/requirements.rst b/doc/tutorial/requirements.rst index 8c297d6..4c548f3 100644 --- a/doc/tutorial/requirements.rst +++ b/doc/tutorial/requirements.rst @@ -1,13 +1,13 @@ .. _tutorial-requirements: -Step 2: Project Requirements +Step 1: Project Requirements ============================ Pseudo-Random Generator ----------------------- -As we discussed before, the random numbers are what makes Monte Carlo method -possible. Unfortunately it is not that easy to quickly generate truly random +As we discussed before, the Monte Carlo method is based on random numbers' +sequences. Unfortunately it is not that easy to quickly generate truly random numbers on a computer. Instead the ``pseudo-random`` generators are used for this purpose. The problem with the javascript's standard ``Math.random()`` is that there is no official way to start a random numbers sequence from a certain @@ -16,48 +16,56 @@ verify the results. However there are great javascript libraries for pseudo-random numbers generation. One of them is the `alea.js`_ library which we are going to use. -So before continuing, please `download <../_static/alea.js>`_ -and include ``alea.js`` in project's ``js`` directory. - - .. _tutorial-requirements-configuration: Configuration ------------- The client-side application configuration is a JSON object passed from the -server to the client when an application is initialized on the client side. -It contains shared and application-specific information required for the -project initialization. -What kind of configuration do we need to pass to the client? +server to the client during the application initialization process. +What kind of configuration does the tutorial app client requires? First of all, a Node should know the number of random points to be generated. -Second, the project requires the ``alea.js`` library and should be able to load -it. Luckily the standard ``importScripts()`` function is available in HTML5 -Web Workers in order to load javascript code. All we have to do is to pass the -URL of the library. Considering these requirements, the desired configuration -is similar to:: +Second, the project requires the ``alea.js`` library. Fortunately the standard +``importScripts()`` function is available in HTML5 Web Workers, which loads +javascript code on the fly. Considering these requirements, the desired +client-side configuration would be similar to:: { 'alea_script' : '/static/projects/monte_carlo_pi/alea.js', 'random_points' : 100000, } -On server-side, the project needs to know the amount of completed *tasks* -that would be enough for the overall calculations. +The server-side part of the application should be aware of the amount of +completed tasks that would be enough to announce the computing process +to be `completed`:: + + { + 'tasks_count' : 10 + } Tasks and Solutions Data ------------------------ -Every random numbers sequence needs a seed to start with. And such unique seed -already exists in every task: it is unique task's ``id`` provided by the project. -For our purpose, even a numerical incremental id is enough to server as a seed. -The returned solution is simpy the calculated value of PI. +Every random numbers sequence needs a seed to start with. Fortunately, +that kind of seed already exists in every task: it is unique task's ``id`` +provided by the project. Even a numerical auto-incremental id is enough +to serve as a seed:: + + { + 'id' : 1 + } + +The returned solution is simpy the calculated value of PI:: + + { + 'pi' : 3.14212 + } The Algorithm ------------- The algorithm of calculating PI is based on the theory explained in -:ref:`introduction `:: +:ref:`tutorial-introduction`:: let points_counter = 0 repeat random_points times: @@ -71,6 +79,6 @@ On server-side the results are collected and the mean value is calculated:: pi = sum(pi_1, pi_2, ... pi_amount_of_tasks) / amount_of_tasks -Continue with :ref:`tutorial-client-side`. +Continue with :ref:`tutorial-project-structure`. .. _alea.js: http://baagoe.org/en/w/index.php/Better_random_numbers_for_javascript diff --git a/doc/tutorial/running.rst b/doc/tutorial/running.rst index e2a81f3..0bccaf0 100644 --- a/doc/tutorial/running.rst +++ b/doc/tutorial/running.rst @@ -1,58 +1,17 @@ .. _tutorial-running: -Step 6: Running the Application +Step 7: Running the Application =============================== - -Makefile --------- - -The Makefile builds ``monte_carlo_pi.js`` locally and (if built by demo - Makefile) copies the fields to to the ``build`` directory.:: - - PROJECT_NAME = monte_carlo_pi - LIB = $(PROJECT_NAME).js - # location of coffee file and to-be-compiled js files. - LIBDIR = js - - # js target - TARGETS = $(LIBDIR)/$(LIB) - CLEAN_TARGETS = - - # if make has been called recursively, add remote targets - ifeq ($(origin PJ_RES_DIR), environment) - TARGETS += remote - CLEAN_TARGETS += clean_remote - endif - - all: $(TARGETS) - - $(LIBDIR)/$(LIB): $(LIBDIR)/$(PROJECT_NAME).coffee - coffee --bare -c $(LIBDIR)/$(PROJECT_NAME).coffee - - remote: - mkdir -p $(PJ_RES_DIR)/$(PROJECT_NAME) - cp $(LIBDIR)/*.js $(PJ_RES_DIR)/$(PROJECT_NAME) - - clean: $(CLEAN_TARGETS) - rm -f $(LIBDIR)/$(PROJECT_NAME).js - - clean_remote: - rm -rf $(PJ_RES_DIR)/$(PROJECT_NAME) - -Now you can simply run ``make`` in project's directory to build it locally or -run ``make`` in Kaylee demo directory to build the projects and collect the -files to the build directory. - - Running the Application ----------------------- -If everything was configured properly, you should be able to run the -MonteCarloPi application the same way as the demo: -run ``python demo/run.py`` and open http://127.0.0.1:5000. +If everything was configured and compiled properly, you should be able +to run the MonteCarloPi application the same way as the :ref:`demo`: + +Run ``python demo/run.py`` and open http://127.0.0.1:5000. -Do you see something as in the picture below? +Do you see something similar to the picture below? .. image:: ../_static/tutorial.png :align: center @@ -70,3 +29,6 @@ Well, they were, but we had only one node participating. Try increasing the amount of tasks and then launch the calculations in two browser tabs or even two different browsers. That's when the calculations will be truly distributed. + +P.S. Don't forget, the source code of the tutorial application is located +at the following repository: http://github.com/BasicWolf/kaylee-tutorial-app diff --git a/doc/tutorial/server_side.rst b/doc/tutorial/server_side.rst index 50af8e5..671ffd4 100644 --- a/doc/tutorial/server_side.rst +++ b/doc/tutorial/server_side.rst @@ -17,12 +17,12 @@ to recognize it:: class MonteCarloPiProject(Project): ... -Before we continue with the code, lets first think, what kind of configuration -is required in order to initialize the project? As it was discussed in the -:ref:`project requirements ` and -implemented on the :ref:`client side `, we need -the amount of random points and the URL of ``alea.js`` script to be passed to -the client and the amount of tasks to execute on the server:: +Before continuing with the code, lets first think, what kind of +configuration is required in order to initialize the project? As it was +discussed in :ref:`tutorial-requirements` and implemented in +:ref:`tutorial-client-side`, the configuration consists of the amount +of random points and the URL of ``alea.js`` script to be passed to the +client and the amount of tasks to execute on the server:: def __init__(self, *args, **kwargs): super(MonteCarloPiProject, self).__init__(*args, **kwargs) @@ -37,9 +37,9 @@ the client and the amount of tasks to execute on the server:: Here, the :py:attr:`Project.client_config` attribute is the configuration object sent to the client and ``self.tasks_count`` and ``self._tasks_counter`` -are the attributes related to amount of tasks to be executed. +are the attributes related to the amount of tasks to be executed. -Next, lets implement two basic abstract methods of Kaylee Project:: +Next, let's implement two basic abstract methods of Kaylee Project:: def __getitem__(self, task_id): return Task(task_id) @@ -55,10 +55,10 @@ As you can see, ``__next__()`` in conjunction with ``__getitem__()`` yields a :py:class:`task ` object with numerical id derived from ``self._tasks_counter``. -The next important part of every project is the :py:meth:`Project.normalize` -method. It is used to verify and normalize the results returned by the client. -In case of our project the calculated value of PI is extracted from the -parsed JSON object (dictionary):: +The next important part of every project is the :py:meth:`normalize +` method. It is used to verify and normalize the results +returned by the client. In this case the calculated value of PI is +extracted from the parsed JSON object (dictionary):: def normalize(self, task_id, data): try: @@ -67,7 +67,7 @@ parsed JSON object (dictionary):: raise InvalidResultError(data, '"pi" key was not found') -And finally, :py:meth:`Project.store_results` - the method which stores +And finally, :py:meth:`Project.store_result` - the method which stores the distributed computation results and determines whether all required data is collected and the application is completed:: @@ -77,24 +77,21 @@ data is collected and the application is completed:: self.completed = True self._announce_results() -Here, ``Project.store_result()`` is called, which stores the results to -the predefined storage (we'll speak about them a bit later) and then -checks whether the project is completed. + Ah, almost missed the part which announces the final results:: def _announce_results(self): - mid_pi = ( sum(res[0] for res in self.storage.values()) / - len(self.storage) ) + mid_pi = (sum(res[0] for res in self.storage.values()) / + len(self.storage)) print('The value of PI computed by the Monte-Carlo method is: {}' .format(mid_pi)) -That is the message you're going to see in Kaylee's front-end shell -(or the logs). +That is the message you're going to see in Kaylee's front-end shell or +logs. -The last step to do with the code: we still need to import the project in -``__init__.py`` for Kaylee to find it:: +The last step to do with the code: the project is still has to be imported +in ``__init__.py`` in order for Kaylee to be able to find it:: from .monte_carlo_pi import MonteCarloPiProject - Continue with :ref:`tutorial-configuration`. diff --git a/doc/tutorial/structure.rst b/doc/tutorial/structure.rst index aadab1e..28d50ab 100644 --- a/doc/tutorial/structure.rst +++ b/doc/tutorial/structure.rst @@ -1,27 +1,26 @@ .. _tutorial-project-structure: -Step 1: Project Structure +Step 2: Project Structure ========================= -Kaylee utilizes recursive Makefile techniques to automate Kaylee client, -projects and demo building process. The ``demo/projects`` directory contains -all the demo projects with ``*.py`` and ``*.coffee`` and/or ``*.js`` files. -So, first of all let's create `monte_carlo_pi` directory in ``demo/projects/``. -It will also serve as a Python package thus, we'll need the ``__init__.py`` -in it. Another good idea is to leave the ``__init__.py`` with as few lines of -code as possible and to write the code in a separate ``monte_carlo_pi.py`` -file. -Finally we will need a place to store the client-side code of the project, -e.g. ``js`` sub-directory. It will contain the ``monte_carlo_pi.coffee`` and -possibly 3d-party libraries required for the project. -The structure of the project directory should now look as following:: +The ``demo/projects`` directory contains all the demo projects with +their ``*.py``, ``*.coffee`` and ``*.js`` files. First of all create +the `monte_carlo_pi` directory in ``demo/projects/``. Add ``__init__.py`` +to indicate that it is a Python package and ``monte_carlo_pi.py`` which +will contain the project code. The ``js`` sub-directory contains the +client-side of the project: ``monte_carlo_pi.coffee`` and ``alea.js`` +library (`download alea.js <../_static/alea.js>`_). +Kaylee utilizes recursive make techniques to automate Kaylee projects +and demo building process. For this a ``Makefile`` file is required. - monte_carlo_pi/ +The structure of the project directory should look as following:: + + demo/projects/monte_carlo_pi/ | |--js/ | | | --monte_carlo_pi.coffee - | --(3d-party js libraries) + | --alea.js | |--__init__.py |--monte_carlo_pi.py @@ -29,4 +28,4 @@ The structure of the project directory should now look as following:: |--Makefile -Continue with :ref:`tutorial-requirements`. +Continue with :ref:`tutorial-client-side`. diff --git a/kaylee/core.py b/kaylee/core.py index 6b94468..3ee7e09 100644 --- a/kaylee/core.py +++ b/kaylee/core.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- """ kaylee.core - ~~~~~~~~~~~~~~~~~ + ~~~~~~~~~~~ This module implements Kaylee's lower level front-end which could be easily used with any web framework.