Permalink
Browse files

Improved sphinx doc, moved deployment guide and running unisubs to th…

…e new doc.
  • Loading branch information...
1 parent 4412c45 commit a1ad9f0a0678b3da92cfae78c9e9e7f39f79196d @arthur-debert arthur-debert committed Jul 30, 2011
Showing with 159 additions and 90 deletions.
  1. +5 −0 .gitignore
  2. +0 −27 css-compression/README.txt
  3. +0 −30 deploy/deploy_guide.txt
  4. +35 −0 docs/deployment-guide.rst
  5. +2 −0 docs/index.rst
  6. +76 −0 docs/running-unisubs.rst
  7. +41 −33 docs/static-files.rst
View
@@ -20,6 +20,10 @@ media/js/mirosubs-widgetizer-debug.js
media/js/mirosubs-extension.js
media/js/mirosubs-statwidget.js
media/js/mirosubs-api.js
+media/js/js-base-dependencies.js
+media/js/js-moderation-dashboard.js
+media/js/js-onsite-dialog.js
+media/js/js-testing-base.js
media/static-cache/
media/teams/*
media/videos/*
@@ -29,3 +33,4 @@ server_local_settings.py
deploy/build/*
celerybeat-schedule
/media/js/site_base_js.js
+/media/docs/*
View
@@ -1,27 +0,0 @@
-Unisubs is compiling all css into specific files in order to make few requests as possible, use far future expire on headers and be able to compress (stripe comments and whitespaces) them.
-
-1. How it works:
-a) settings.py has media bundles, identified by their name (e.g 'video_history'), and the files to be included.
-b) Add / remove the path to the css files (always from the root of MEDIA_ROOT) to the desired bundles
-c) if locally run:
-$ python manage.py compile_media --settings=dev_settings && python manage.py update_compiled_urls --settings=dev_settings
-(to update the css)
-d) You can disable css compression with the setting CSS_USE_COMPILED ( which defaults to the oposite of debug)
-
-Then at compilation time, we go trough each bundle, read from disk the css file, concatenate them. Run the contatenated file though the yahoo yui compressor (at css-compression/yuicompressor....jar) . We take the md5 checksum of that result and save that file on static-cache with the checksum as the filename.
-After all bundles are done, we save the mapping between available bundles and their md5 names static-cache/compresses.py.
-When a new version is deployed (through the update fab command) we download the mappings above and read them, saving them as a dict on the settings file. (similar to how the commit.py works). Then, at run time, the template tag will subtistitute a 'include_css_bundle' block with the url that it fetched on the mappings file. If that files, it simply inserts the <link> tags pointing to the files in the bundle.
-
-2, How to add css files:
-a) Add files to media/css
-b) Add to the bundles needed ( check on the templates used, which bundles are already in use ).
-c) If you need a new page new a new bundle, create one in settings.py and use it on the {% include_css_bundle "new_name" %} on your template.
-d) Watchout for URLS, an image url should be, for example: "../images/foo.jpg".
-
-3. How to add an external lib (such as a Jquery plugin).
-a) Move the plugin's css file to the media/css directory.
-b) Include it on the needed bundles on settings.py (see above)
-c) Move the 'images' directory to "images/plugin-name/", for example "images/jalert" for jalert.
-d) Replace the "url()" tags in the css from "images/" to "../images/plugin_name"
-
-
View
@@ -1,30 +0,0 @@
-* update unisubs-settings.py if necessary *
-
-fab <server> add_disabled
-fab <server> switch_branch:<branch_name>
-fab <server> update_environment
-fab <server> syncdb
-fab <server> migrate_fake:<app_name>
-fab <server> migrate
-fab <server> update
-fab <server> bounce_memcached
-fab <server> remove_disabled
-
-- open ssh session and run
-$manage.py/ build_solr_schema --settings=unisubs_settings > /etc/solr/conf/production/conf/schema.xml # 20 seconds
-$python manage.py update_index --settings=unisubs_settings --verbosity=2 # ~1 hours
-
-
-for staging:
-
-fab staging:<username> refresh_db
-fab staging:<username> switch_branch:<branch_name>
-fab staging:<username> clear_environment_permissions
-fab staging:<username> update_environment
-fab staging:<username> syncdb
-fab <server> migrate_fake:<app_name>
-fab <server> migrate
-fab <server> update
-
-
-for 092volunteer: migrate_fake messages and targetter
View
@@ -0,0 +1,35 @@
+================
+Deployment Guide
+================
+
+
+Update unisubs-settings.py if necessary *::
+
+ fab <server> add_disabled
+ fab <server> switch_branch:<branch_name>
+ fab <server> update_environment
+ fab <server> syncdb
+ fab <server> migrate_fake:<app_name>
+ fab <server> migrate
+ fab <server> update
+ fab <server> bounce_memcached
+ fab <server> remove_disabled
+
+open ssh session and run::
+
+ $manage.py/ build_solr_schema --settings=unisubs_settings > /etc/solr/conf/production/conf/schema.xml # 20 seconds
+ $python manage.py update_index --settings=unisubs_settings --verbosity=2 # ~1 hours
+
+
+for staging::
+
+ fab staging:<username> refresh_db
+ fab staging:<username> switch_branch:<branch_name>
+ fab staging:<username> clear_environment_permissions
+ fab staging:<username> update_environment
+ fab staging:<username> syncdb
+ fab <server> migrate_fake:<app_name>
+ fab <server> migrate
+ fab <server> update
+
+
View
@@ -11,9 +11,11 @@ Contents:
.. toctree::
:maxdepth: 1
+ running-unisubs
static-files
video-models
demos
+ deployment-guide
Indices and tables
View
@@ -0,0 +1,76 @@
+===========================
+Running Universal Subtitles
+===========================
+
+To run the development version:
+
+1. Git clone the repository::
+
+ git clone git://github.com/8planes/mirosubs.git mirosubs
+
+ Now the entire project will be in the mirosubs directory.
+
+2. Install virtualenv http://pypi.python.org/pypi/virtualenv
+
+3. (optional) download and download the virtualenv wrapper bash
+ functions http://www.doughellmann.com/projects/virtualenvwrapper/
+
+4. Create a virtual environment and activate it. Here is how to do it
+ *without* the virtualenv wrapper. Run these commands from the parent
+ of the mirosubs directory created in #1::
+
+ $ virtualenv mirosubs-env
+ $ source mirosubs-env/bin/activate
+
+ If you're using the virtualenv wrapper (run from any directory)::
+
+ $ mkvirtualenv mirosubs
+ $ workon mirosubs
+
+5. run::
+
+ $ easy_install -U setuptools
+ $ easy_install pip
+ $ cd deploy
+ # this is the mirosubs directory you cloned from git, not the parent you created the virtualenv in.
+ $ pip install -r requirements.txt
+ note: you'll need mercurial installed to make this last command work.
+ note2: If you do not have the MySQL bindings installed (MySQLdb) and wish to keep it that way, unisubs runs just fine on sqlite, just comment out the line "MySQL_python>=1.2.2" on deploy/requirements.txt before running this command.
+
+
+6. Check out google closure into directory of your choice: svn checkout
+ http://closure-library.googlecode.com/svn/trunk/ <directory>. Then
+ symlink media/js/closure-library to the checkout location. From the
+ mirosubs directory in step 1::
+
+ $ cd media/js
+ $ ln -s <google closure checkout directory> closure-library
+
+7. Add mirosubs.example.com to your hosts file, pointing at 127.0.0.1.
+ This is necessary for Twitter oauth to work correctly.
+
+8. From the mirosubs directory created in step 1, first create the
+ database with::
+
+ python manage.py syncdb
+
+ Then update the database with::
+
+ python manage.py migrate
+
+ SQLLite warnings are okay. Then run the site with::
+
+ ./dev-runserver.sh
+
+ You can access the site at http://mirosubs.example.com:8000.
+
+9. (optional) If you want to run video searches / watch page locally, you need to set up solr:
+
+ A. Download solr and unzip to ../buildout/parts/solr (relative to this directory).
+ B. Run ./manage.py run_solr in one terminal that is dedicated to running the solr process.
+ C. run ./manage.py rebuild_index to update the index.
+ D. That should be it but, in case you're interested, there's a
+ list of haystack commands at
+ http://docs.haystacksearch.org/dev/management_commands.html
+ * If you want to install SOLR as a daemon on your Mac, please see
+ https://github.com/8planes/mirosubs/wiki/Running-SOLR-as-a-daemon-on-Mac
View
@@ -3,22 +3,30 @@ Static Files
**********************************
-*********
Overview
-*********
+=========
The static files pipeline for unisubs has two goals:
- Combine & compress files that are used together (e.g. one html page linking to many css filed)
- Make sure every time media is processed we have each file with a unique name. This allows us to the set the expire header to the far future, meaning that it will be aggressively cached.
- Each new release won't be cached by clients, since the URLS for each release are unique
-**************************
-Gotchas for developers
-**************************
-1. Make sure you run create_commit_file after each commit or merger (git hooks are the best place to keep that)
-***************************
+
+Quick how to
+============
+
+1. When in development with DEBUG = True all media will not be compressed.
+2. You can add media to the any of the bundles on settings.py or create a new bundle if you need a specific collection of media (see bellow)
+3. If it's a new bundle add the include in the template (see bellow)
+
+When you want to test the compiled version
+1. Set DEBUG to false
+2. Run the compile_config, compile_statwidgetconfig, compile_media commands.
+3. Run deploy/create_commit_file
+4. Run the compile_media command
+
Compilation / Minification
-***************************
+===========================
settings.py has a MEDIA_BUNDLES dictionary. Each key sets an id (a unique name for the bundle), with the following properties:
- `type`: Can be `css` or `js` for now
@@ -34,9 +42,8 @@ JS will be compiled by the closure compiler at closure/compiler.jar .
CSS will be compressed with the YUI compressor at css-compression/yuicompressor-2.4.6.jar .For css we are only minifinng, but we are not changing anything that might break the site.
-**********************************
In Templates
-**********************************
+==================================
In templates, instead of liking to the the individual css or js files, you'd simply::
{% load media_compressor %}
@@ -45,16 +52,14 @@ In templates, instead of liking to the the individual css or js files, you'd sim
If `CSS_USE_COMPILED` is True the link to that bundle compressed file will be inserted. Else we'll add the original urls. If a CSS_USE_COMPILED is not set, it will default to the oposite of `settings.DEBUG`.
-***************************
Compiling
-***************************
+===========================
Compiling should be done with ::
- $manage.py compile_media --settings*[settings file]
+ $manage.py compile_media --settings=[settings file]
-***************************
Dir layout
-***************************
+===========================
Inside MEDIA_ROOT media will be compiled to `static-cache/[hash of latest git commit]/.
i.e. MEDIA_ROOT/static-cache/21a1bbcc/js/mirosubs-onsite-compiled.js
@@ -63,34 +68,30 @@ In this way we use one unique identifier for each revision, in fact, we relly on
Static cache is never in source control and should not be the storage path for anything. It's sole purpose is to be a disposable place where media can be compiled and moved to a unique URL.
-***************************
+Relevant Settings
+=================
+These need to be defined in settings.py:
+
MEDIA_URL
-***************************
+----------
Since the location of media is no longer static, the MEDIA_URL takes into consideration the new directory layout. Every time a new git version is running and deploy/create_commit_file.py is ran, the MEDIA_URL will change (and) therefore you need to restart the sever/reload app coder for that to take effect.
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Relevant Settings
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-These need to be defined in settings.py:
-""""""""""""""""""""
COMPRESS_YUI_BINARY
-""""""""""""""""""""
+-------------------
The path, relative to the mirosubs dir where the YUI compressor jar lives( "java -jar ./css-compression/yuicompressor-2.4.6.jar")
-"""""""""""""""""
-JS_USE_COMPILED
-"""""""""""""""""
-Should always be True on dev, staging and production. Developers can set it to false to make development/debugging easier.
+COMPRESS_MEDIA
+--------------
+When true media will be compressd / packed, Both CSS and JS. The default value for this is the oposite of DEBUG.
-""""""""""""""""""""
COMPRESS_OUTPUT_NAME
-""""""""""""""""""""
+--------------------
+
The directory that holds the root to the static cache, i.e. where all compiled and version specific media will be copyed to (see dir layout above). Defaults to 'static-cache'.
-********************
Serving Media
-********************
+=============
On the local development machine or the dev environment media is stored locally in the file disk. Staging and production with Amazon's s3, so in those environments media needs to be copied to s3.
This is achieved by calling::
@@ -105,5 +106,12 @@ That command requires the USE_AMAZON set (needs correct values for secret, id an
All files above 1kb will be served with gzip compression (smaller files tend to actually inflate ).
-
-
+TODOS
+====
+
+- Join all the compile_config... commands with compile media
+- Remove all lingering instances of `include _js_onsite... ` and `JS_USE_COMPRESSES`.
+- Make compilation fail on any error.
+- Find out how to fix warnings for jQuery and others
+- Put all binaries that deal with media compression (closure compiler, yui compressor) on the same place
+

0 comments on commit a1ad9f0

Please sign in to comment.