updating all rst-based documentation from laest website info

doc/source/_static/rtd.css

@@ -37,6 +37,10 @@ body {
     margin: 0;
 }
 
+dt {
+	font-weight: bold;
+}
+
 div.document {
     text-align: left;
     background-color: #e8ecef;

doc/source/acknowledgments.rst

@@ -0,0 +1,17 @@
+acknowledgments
+---------------
+
+.. _Guido van Rossum: http://www.python.org/~guido/
+.. _python: http://python.org/
+.. _Dave Coffin: http://www.cybercom.net/~dcoffin/
+.. _dcraw: http://www.cybercom.net/~dcoffin/dcraw/
+.. _Python Imaging Library: http://www.pythonware.com/products/pil/
+.. _Ali-Akber Saifee: http://www.indydevs.org/
+
+nefarious was made possible thanks to:
+
+- `Guido van Rossum`_: for making python_
+- `Dave Coffin`_: for making dcraw_, which helped me a lot to understand the nef structure
+- The `Python Imaging Library`_: which got me started with tiff parsing, many of the tag constants and tiff headers in nefarious comes straight from PIL.
+- `Ali-Akber Saifee`_: my python mentor, which became involved a lot more than he anticipated in making nefarious presentable :p
+

doc/source/faq.rst

@@ -1,27 +1,79 @@
 FAQ
 ---
+
+.. _tiff file format specifications: http://partners.adobe.com/asn/developer/PDFS/TN/TIFF6.pdf
+.. _dcraw: http://www.cybercom.net/~dcoffin/dcraw/
+.. _rawtherapee: http://www.rawtherapee.com/
+.. _nefarious license: https://raw.github.com/timotheeg/nefarious/master/license/license_nefarious.txt
+.. _PIL license: https://raw.github.com/timotheeg/nefarious/master/license/license_PIL.txt
+
 Wait... did you say you were manipulating raw files? That's the greatest sin of digital photography!!!! Haven't you read the memo? NO program should ever touch a raw file!! You should rot in hell, along with all your descendants for 10 generations.
-yes. yes. you are a moron.
+    yes. yes. thank you.
 
-The "Raw file", is just a tiff file, a bunch of tags to tell you how to read the file itself. What is really precious in that file is the sensor data. nefarious doesn't alter that in any way.
+    Raw NEF files follow the `tiff file format specifications`_, which means they're reasonnably easy to parse and understand. A NEF file basically consists of a bunch of tags and directories, themselves containing more tags and data. What is really precious in that NEF file is the sensor data, which sits in one of the directory. nefarious doesn't alter that in any way.
 
-Anything else is fair game: change the date, change/add gps location, update the preview jpeg, whatever. The aim of nefarious is to spit out nef files in such a way that no program, not even from nikon itself, would ever realize the nef was not straight out of a Nikon camera. and of course the ability to actually use the sensor data for editing is fully retained.
+    Anything else is fair game: change the date, change/add gps location, **update the preview jpeg**, whatever. The aim of nefarious is to spit out nef files in such a way that no nef reader (e.g. dcraw_), and hopefully not even a nef reader from Nikon itself, would realize the nef was not straight out of a Nikon camera. Of course the ability to actually use the sensor data for developpment in a raw editor like rawtherapee_ is fully retained.
 
-Just to be clear though, this program comes with no warranty whatsoever. Clearly, you are a zealot, so by all mean do NOT use my program, it might just give you an ulcer. Here is a better site for you.
+    Just to be clear though, this program comes with *no warranty whatsoever*. If you are religious about your raw nef files, do NOT use this program, it might just give you an ulcer. And please, do not send me any message either to tell me how horrible nefarious is... If you don't like it, just move on and forget about it.
 
-Now, if you DO try it, please be a good computer user and:
+    Now, if you DO try it, please be a good computer user and 1) try first on a few files before running this on your entire image collection. 2) do me and yourself a favour and have your data backed up.
 
-#. Try first on a few files before running this on your entire image collection. 
-#. Do me and yourself a favour and have your data backed up.
+Why was that the first question?
+    The first time I submitted the idea of a nef manipulation program to replace the dummy jpeg preview, the first few reactions I received were rather negative. So, the first FAQ question was relevant to:
 
-Why was that first question?
-The first time I submitted the idea of a nef manipulation program to replace the dummy jpeg preview, it was the first reaction I received. Some guy said editing raw files was "a crime". So, the question was relevant as the first of the FAQ because:
+    #. inform people nefarious is something they might not like, and they can stop right there if they feel offended
+    #. "nefarious" as the utility name was chosen because of such people :D
 
-it can tell buggers to f%@$ off right away
-nefarious is the name chosen because of them
 Why the name "nefarious"?
-Well,nefarious means nasty, and many people think touching raw files is just that. Additionally, I am manipulating nef files, and well, there is only one word in the oxford dictionary that starts with "nef", and that's nefarious :).
+    "nefarious" means nasty, and many people think touching raw files is just that. Additionally, we are manipulating nef files here, and well, there is only one word in the oxford dictionary that starts with "nef", and that's "nefarious" :).
+
+What can nefarious do?
+    nefarious is basically a python lib, that allows you to manipulate tiff tags and tiff structure, for example adding a new subdirectory to store additional images or data is possible, modifying the comments, adding your own custom fields in the file, or replacing the jpeg preview image.
+
+    nefarious also has a simple cli tool that can do 2 things: 1) print the structure of a given nef; and 2) replace the preview jpeg by another jpg (jpg should have been jpegtran-ed first).
+
+    nefarious understands tiff tags, so any tag that is stored as a native tiff type (string, shorts, longs, rational, etc.) can be modified easily.
+
+    nefarious can't let you modify things it doesn't understand, for example: the actual pixel data of the jpeg preview; the raw sensor data; the Nikon maker notes
+
+Will nefarious have the ability to edit the tags in the Nikon makernotes, like other tiff tags ?
+    Unlikely. I know the makernotes encryption has been cracked and they are now fully readable, but I have no need for editing this myself, and I had enough pain dealing with tiff reading/writing.
+
+Is nefarious cross platform?
+    Hopefully. I've used python standard size structs for all reading and writing, which match the tiff specifications, it should work with python on any platform, but I've only tested it on my gentoo amd64 box.
 
 Which nef format/version does nefarious support?
-So far nefarious is tested on my own D300S nef files. it knows how to parse a typical tiff file by being aware of which tiff tags represent tiff directories (only 3 in the D300S).
+    So far nefarious is tested on my own D300S nef files. It knows how to parse a typical tiff file by being aware of which tiff tags represent tiff directories (only 3 in the D300S).
+
+    nefarious has successfully read and rewritten nefs from my ex-D60 as well, but I haven't attempted to set the jpeg preview for D60 pics (yet).
+
+why are nefarious written nef files smaller than the original nefs?
+    Very observant! so I guess you've noticed in plain read-and-write-again, around 100 bytes are missing from the written files... Did nefarious erase something? Actually, I'm not sure myself. I assume this is because the nef file from Nikon must be storing a few extra bytes for padding here and there when writing. nefarious writes the absolute mininum with no consideration for padding. and so saves these few bytes.
+
+    I must admit this has bothered me, especially since I haven't managed to find these extra bytes in some samples nefs (it's not easy to find a 100 bytes in a 15MB file). I have somewhat put my worries to rest after:
+
+    - doing structure prints of the original nef and written nef reported the same data
+    - dcraw was able to extract the preview jpeg from the written nef with no complaint whatsoever
+    - I could open the written nef in rawtherapee, see all data, including makernotes, and proceed with editing as usual
+
+    That being said, I actually couldn't find whether the (supposedly) padding bytes are supposed to be there or not (i.e. should I write pading bytes when saving the nef?). I'd welcome any information on this.
+
+Will nefarious support more nef versions?
+    I have nefs from 2 cameras, my ex-D60 and my current D300S. nefarious is only fully tested on the D300S nefs, full test and support for the D60 will ceratinly come soon.
+
+    I haven't researched the nef produced by other Nikon DSLRs like D90, etc. But assuming Nikon sticked to tiff file format for all its cameras, many other might just work with nefarious out of the box. DON'T take my word for it! Backup your data and run a few tests yourself.
+
+    If some nef files from some Nikon cameras don't work with nefarious, contributors are welcome to add support.
+
+Will nefarious support other types of raw files (e.g. from Canon)?
+    Unlikely. I don't own any canon cameras. That being said, if raw files from Canon also follow the tiff specifications, perhaps nefarious could understand them with some minor updates. I won't work on that though, but contributors are welcome.
+
+Will nef-cli support more operations than just print-structure, and replace-preview-jpeg?
+    Maybe. If there are nef editing tasks than come up often in my workflow, I'll make the sure they exists in nef-cli for convenience.
+
+    For any hadhoc manipulation job, it's very easy to write a one-off custom python program to load a nef file, modify it as needed and save it again.
+
+What's the licence of nefarious?
+    nefarious is published under the MIT license, full description in github: `nefarious license`_.
 
+    nefarious uses copy/pasted tiff tags constants from the PIL library. The PIL license is also provided in githug: `PIL license`_.

doc/source/how.rst

@@ -0,0 +1,57 @@
+how?
+----
+
+.. _libjpeg: http://en.wikipedia.org/wiki/Libjpeg
+
+
+first, download nefarious from github and install it::
+
+    $ python setup.py build install
+
+Now... sorry... nefarious has no real documentation yet, but ali is teaching me some really funky tools and hopefully we'll have doc teom the code itself soonish.
+
+For now, basically for using nefarious as a lib, I recommend reading the code itself as well as the cli tool (main.py) . Aslo using the cli tool, you can print the nef structure, which will let you understand what you can modify
+
+For using the cli tool, below are some examples. Assume that
+
+-  **input.nef** is the input nef file you want to manipulate
+-  **processed.jpeg** is the developed and processed jpeg you want to inject i nyour nef to replace the dumb preview jpeg.
+
+Note: as a prerequisite for replacing the preview jpeg with another file, that file should have all metadata stripped. That can be achieved easily with the jpegtran utility from libjpeg_.
+
+Things you can do:
+
+::
+
+    $ nef-cli -i input.nef
+    
+Will print the nef structure
+
+::
+
+    $ nef-cli -i input.nef -o out.nef
+    
+Will parse the input nef, and just save it again with its own tiff save algorithm, with no data modification. The command basically lets you test if out.nef is still usable in your favorite nef reader (basically the data should be identical amd no nef reader should be able to tell the difference)
+Note that while the data and nef structure is identical, the file byte layout is NOT. nefarious does not necessarily save tags and directories in the same file location as the original nef, but that should be of no consequence for a nef reader.
+
+::
+
+    $ jpegtran -optimize -copy none -outfile processed.stripped.jpg processed.jpeg
+    $ nef-cli -i input.nef -p processed.stripped.jpg -o out.nef
+    
+The jpegtran command above prepares the jpeg file by stripping it of any unecessary metadata
+The second command uses the stripped jpeg to replace the preview jpeg of the input nef, and save the resulting nef in out.nef
+
+::
+
+    $ jpegtran -optimize -copy none -outfile processed.stripped.jpg processed.jpeg
+    $ nef-cli -i input.nef -p processed.stripped.jpg
+    
+Same as above, but will overwrite the input nef with new data. It will prompt you to be sure you are ok with overwriting
+
+::
+
+    $ jpegtran -optimize -copy none -outfile processed.stripped.jpg processed.jpeg
+    $ nef-cli -i input.nef -y -p processed.stripped.jpg
+    
+Same as above, but automatically answers 'yes' when prompted to overwrite (use with care!)

doc/source/index.rst

@@ -2,7 +2,7 @@
 
 Nefarious : Nikon NEF File Parser
 =================================
-Manipulates your Nikon NEF files to (amongst other things) change the embedded jpeg thumbnail
+nefarious let's you edit your nikon raw nef files to store your processed images inside the nef files themselves
 
 Source: `GitHub Repository`_
 
@@ -12,6 +12,7 @@ Source: `GitHub Repository`_
    what
    why
    who
+   where
+   how
    faq
-
-
+   acknowledgments

doc/source/what.rst

@@ -1,7 +1,9 @@
-What
-----
+what?
+-----
 nefarious is a python library and utility that lets you manipulate parts of a Nikon NEF file WITHOUT altering the sensor data.
+
 In particular, nefarious allows you to replace the embedded jpeg image preview (typically a full-size, but low-quality, jpeg), by any other replacement image of your choice.
+
 For photographers shooting in RAW+FINE, nefarious will allow the photographer to replace the low-quality jpeg by the HQ jpeg and keeping both resources in one file.
-If image processing is involved, nefarious allows the artists to embed the final processed image into the NEF itself.
 
+If image processing is involved, nefarious allows the artists to embed the final processed image into the NEF itself.

doc/source/where.rst

@@ -0,0 +1,4 @@
+where?
+------
+
+nefarious is available on gihub https://github.com/timotheeg/nefarious. 

doc/source/who.rst

@@ -1,5 +1,7 @@
-Who
----
-* nefarious is made by me, Timothée Groleau, for my own usage. I have no doubt that such an awesome, open source, utility will soon attract hords of eager developers ;)
-* nefarious is made in python as a way for me to learn python through a real program. I am in great debt to my friend and colleague Ali who made it somewhat presentable before you had a chance to see it.
-* I am normally quite opened minded and thick-skinned, but if you spot some wtf moments while looking at the code, please stay polite is letting me know, you wouldn't want to hurt my feelings now, would you?
+who?
+----
+nefarious is made by me, *Timothée Groleau*, for my own usage. I have no doubt that such an awesome, open source utility will soon attract hords of eager contributors ;)
+
+nefarious is made in python as a way for me to learn python through a real program. I am in great debt to my friend and colleague Ali-Akber Saifee who made nefarious a lot more presentable before you had a chance to look at it.
+
+I am normally pretty opened minded and thick-skinned, but if you spot some wtf moments while looking at the code, please stay polite is letting me know, you wouldn't want to hurt my feelings now, would you?

doc/source/why.rst

@@ -1,8 +1,8 @@
-Why
----
+why?
+----
 .. _KPhotoAlbum: http://www.kphotoalbum.org/
 
-I've looked at several sofftware for image management (and I do NOT mean image retouching). I've settled some years back on KPhotoalbum_. and I have invested a significant of time with it in filesystem arrangement and photo tagging.
+I've looked at several sofftware for image management (and I do NOT mean image retouching). I've settled some years back on KPhotoAlbum_ and I have invested a significant of time with it in filesystem arrangement and photo tagging.
 
 Still, I'm rather old school in trusting only myself as far as 1) backups and 2) knowledge of my pics are concerned. I don't want to rely on any software to know that a nef and a same-named jpeg in some other folder are the actual same image. In the end, there are 2 files (or more) on the file system. To view, I want to see the jpeg, processed version. To edit, I want to start from the raw sensor data. How can I manage that?
 
@@ -14,12 +14,11 @@ I considered a zip file, but that would kill the os ability to display the image
 
 Then I realized that NEF files are just TIFF files; and the TIFF file format is just a container, with a well-defined, a well-understood tag-based format, that can hold just about anything. In fact, a NEF file from my Nikon D300S contains:
 
-#. a small thumbnail of the pic
-#. a full size, low quality jpeg preview
-#. the raw sensor data
-#. various metadata in standard (exif), and non-standard format (nikon's "maker notes")
+   1. a small thumbnail of the pic
+   2. a full size, low quality jpeg preview
+   3. the raw sensor data
+   4. various metadata in standard (exif), and non-standard format (Nikon's "maker notes")
 
-It rapidly became obvious that with a suitable utility, I should be able to replace the low -quality jpeg with whatever I considered to be the final processed look of the raw data.
-
-Lots of procrastination later, nefarious was finaly born.
+It rapidly became obvious that with a suitable utility, I should be able to replace the low-quality jpeg with whatever I considered to be the final processed look of the raw data.
 
+Lots of procrastination, and some work later, nefarious was finaly born.