#35 json (#42)

* fixig make dev_env

* rename null to nullable

* adding key and required fields

* #35  adding json_schema export

* base properties done

* #41 strict flag added

* some documenation and adding a model set method

* completing model.update

* first attempt to singer emitter

* altering escape/replacement order

* updating changelog

* pylint shitsu

* fix issues

CHANGELOG.md

@@ -4,10 +4,15 @@ This is a simple data-migration package for python lovers. It is declarative lan
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 
 ## [0.5.1] - 2017-04-24
+### Changed
+- pep 263 make shebang virtualenv aware #36
+- edit schema's more inline with json_schema #35
+- simplify semver to final/dev #37
+
 ### Added
 - test metadata in circle-ci #25
 - pyup.ip added #31
-- simplified semver release to dev/final #37
+- add strict flag #41
 
 
 ## [0.5.0] - 2017-04-21

Makefile

@@ -30,6 +30,8 @@ dev_requirements:
 	@pip install -r py.requirements/build.txt
 
 dev_env:
+	@pip install -r py.requirements/docs.txt
+	@pip install -r py.requirements/build.txt
 	@pip install -r py.requirements/environment.txt
 
 tox:

README.rst

@@ -49,7 +49,7 @@ fast, readable and extendable
   class Result(models.Model):
     id   = models.IntField(pos=0) # keep id
     uuid = models.UUIDField()     # generate new uuid4 field
-    a    = models.StringField(pos=1, default='NO_NULL', max_length=5, null='NULL', replacement=lambda x:x.upper())
+    a    = models.StringField(pos=1, default='NO_NULL', max_length=5, nullable='NULL', replacement=lambda x:x.upper())
     b    = models.StringField(pos=2, name='my_b')
 
   if __name__ == "__main__":

docs/concepts/scan-emit.txt

@@ -12,7 +12,7 @@ transformer will read stdin and send every CSV row to the model for scanning. Ou
 the fields define a scan loop:
 
 #. **select** the specified column from the row.
-#. **null** test if not allowed and replace by default.
+#. **nullable** test if not allowed and replace by None.
 #. **validate** the input (if validator is provided).
 #. **parse** the input (if parser is provided).
 #. **store** as native python value (aka NULL=>None).

docs/example.txt

@@ -53,7 +53,7 @@ is helpful. That is why we came up with *data-migrator*. One could simply replac
     id   = models.IntField(pos=0) # keep id
     uuid = models.UUIDField()     # generate new uuid4 field
     # replace NULLs and trim
-    a    = models.StringField(pos=1, default='NO_NULL', max_length=5, null='NULL', replacement=lambda x:x.upper())
+    a    = models.StringField(pos=1, default='NO_NULL', max_length=5, nullable='NULL', replacement=lambda x:x.upper())
     # parse this field
     b    = models.StringField(pos=2, parse=parse_b, name='my_b')
 

docs/ref/emitter.txt

@@ -4,25 +4,7 @@ Emitter class reference
 
 .. currentmodule:: data_migrator.emitters
 
-This document covers features of the :class:`~data_migrator.emitter.BaseEmitter` class. Currently the
-system has two emitters: ``CSVEmitter`` and ``MySQLEmitter`` implemented, of which the last is the
-default emitter. An emitter provides the export format for the scanned and cleaned datasets. It also
-provides preambles in the output files, for example to clean the target table before loading it.
-
-The basic structure for emitting is a combination between ``manager`` and ``emitter``:
-
-.. code-block:: python
-
-  e = Emitter(manager=Model.objects)
-  print e.preamble(header=[..my header lines to add..])
-  for l in Model.objects.all():
-    print e.emit(l)  # emit is returning a list of strings!
-
-
-.. note::
-
-    At this moment *data-migrator* does not an actively take part in schema migrations of any
-    sort. It is purely about cleaning and transforming data (yet!).
+.. automodule::  data_migrator.emitters
 
 
 MySQLEmitter

docs/ref/fields.txt

@@ -7,15 +7,15 @@ Model field reference
 
 .. currentmodule: data_migrator.models
 
-This document contains all API references of BaseClass :class:`~.fields.BaseField`
+This document contains all API references of BaseClass :class:`~.BaseField`
 including the `field options`_ and `field types`_ data_migrator offers.
 
 .. note::
 
-    Technically, these models are defined in :mod:`data_migrator.models.fields`, but
-    for convenience they're imported into :mod:`data_migrator.models`, the standard
-    convention is to use ``from data_migrator import models`` and refer to fields as
-    ``models.<Foo>Field``
+    Technically, these models are defined in :mod:`data_migrator.models.fields`
+    , but for convenience they're imported into :mod:`data_migrator.models`,
+    the standard convention is to use ``from data_migrator import models`` and
+    refer to fields as ``models.<Foo>Field``
 
 Field options
 =============
@@ -53,10 +53,17 @@ value. StringField only has empty string as empty value. With this field it can
 changed to some other standard value. Consider a Country field as string and setting it
 to the home country by default.
 
-``null``
---------
+``key``
+-------
+
+.. attribute:: Field.key
+
+If set, this indicates the field is a key field for identification of the object.
+
+``nullable``
+------------
 
-.. attribute:: Field.null
+.. attribute:: Field.nullable
 
 If set it will match the source column value and consider this a ``None`` value. By
 default this attribute is set to ``None``. Note that for none Null fields ``None``
@@ -71,6 +78,13 @@ If set, this is a pre-emit replacement function. This could be used to insert dy
 replacement lookup select queries, adding more indirection into the data generation.
 Value could be either function or a string.
 
+``required``
+---------------
+
+.. attribute:: Field.required
+
+If set, this indicates the field is required to be set.
+
 ``parse``
 ---------
 

docs/ref/meta.txt

@@ -4,113 +4,31 @@ Meta class reference
 
 .. currentmodule:: data_migrator.models
 
-This document covers features of the :class:`~data_migrator.models.Meta` class. The meta class
-defines model specific settings.
+This document covers features of the *Meta* class.
+The meta class defines model specific settings and is used as an inner class
+in the model:
 
-.. note::
-
-    Technically, Meta is just a container and forwarded to :class:`~.Options`
-
-Field options
-=============
-
-The following arguments are available to all field types. All are optional.
+.. code-block:: python
 
-``drop_if_none``
-----------------
+  from data_migrator import models
 
-.. attribute:: Meta.drop_if_none
+  class SampleModel(models.Model):
+    a = models.IntField(pos=1)
 
-Is a list of field names as defined. If set *data-migrator* will check if fields are not None
-and drop if one of the columns is.
+    class Meta:
+      drop_if_none = True
 
-Any field listed in this attribute is checked after scanning and just before save-ing.
+Every model can have its own meta class to define model specific options.
 
 .. note::
 
-    Note that only NullXXXFields actually can be ``None`` after scanning and parsing. Non
-    Null fields are set to their default value.
-
-
-``drop_non_unique``
--------------------
-
-.. attribute:: Meta.drop_non_unique
-
-If ``True``, *data-migrator* will drop values if the column uniqueness check fails
-(after parsing). Default is ``False``.
-
-Any field can be defined as a unique column. Any field set so, is checked after
-scanning and just before save-ing.
-
-``emitter``
------------
-
-.. attribute:: Meta.emitter
-
-If set, *data-migrator* will use this emitter instead of the default emitter.
-
-``fail_non_unique``
--------------------
-
-.. attribute:: Meta.fail_non_unique
-
-If ``True``, *data-migrator* will fail as a whole if the column uniqueness check fails
-(after parsing). Default is ``False``.
-
-Any field can be defined as a unique column. Any field set so, is checked after
-scanning and just before save-ing.
-
-``fail_non_validated``
-----------------------
-
-.. attribute:: Meta.fail_non_validated
-
-If ``True``, *data-migrator* will fail as a whole if the column validation check fails
-(after parsing). Default is ``False``.
-
-Any field can have its own validator, this is a rough method to prevent bad data from
-being transformed and loaded.
-
-``file_name``
--------------
-
-.. attribute:: Meta.file_name
-
-If set, *data-migrator* will use this as file_name for the emitter instead of the default
-filename based on table_name.
-
-
-``table_name``
---------------
-
-.. attribute:: Meta.table_name
-
-If set, *data-migrator* will use this as table_name for the emitter instead of the default
-tablename based on model_name.
-
-
-``prefix``
-----------
-
-.. attribute:: Meta.prefix
-
-If set, *data-migrator* will use this list of statements as a preamble in the generation of
-the output statements. By default an emitter uses this to clear the old records.
-
-
-``remark``
-----------
-
-.. attribute:: Meta.remark
+    Technically, Meta is just a container and forwarded to :class:`~.Options`
 
-If set, *data-migrator* will use this as the remark attribute in the Model, default='remark'.
-Use this for example if you have a ``remark`` field in your model and need to free the keyword.
 
-``manager``
------------
+.. autoclass::  data_migrator.models.options.Options
+  :members:
 
-.. attribute:: Meta.manager
+.. note::
 
-If set, *data-migrator* will use this as the manager for this model. This is useful if
-the ``transform`` method needs to be overridden.
+    Note that only NullXXXFields actually can be ``None`` after scanning and
+    parsing. Non Null fields are set to their default value.

docs/ref/model.txt

@@ -5,9 +5,9 @@ Model class reference
 .. module:: data_migrator.models.base
    :synopsis: Base for model definitions
 
-.. currentmodule:: data_migrator.models
+.. currentmodule:: data_migrator.models.base
 
-This document covers features of the :class:`~data_migrator.models.Model` class.
+This document covers features of the :class:`~Model` class.
 
 
 Model

docs/tutorial/tutorial0.txt

@@ -70,7 +70,7 @@ build a simple transformer:
     id   = models.IntField(pos=0) # keep id
     uuid = models.UUIDField()     # generate new uuid4 field
     # replace NULLs and trim
-    a    = models.StringField(pos=1, default='NO_NULL', max_length=5, null='NULL', replace=lambda x:x.upper())
+    a    = models.StringField(pos=1, default='NO_NULL', max_length=5, nullable='NULL', replace=lambda x:x.upper())
     # parse this field
     b    = models.StringField(pos=2, parse=parse_b, name='my_b')
 

docs/tutorial/tutorial2.txt

@@ -18,7 +18,7 @@ can be set on a model basis in the Meta block
   class Result(models.Model):
     id = models.IntField(pos=0) # keep id
     a  = models.StringField(pos=1)
-    b  = models.StringField(pos=2, null=None)
+    b  = models.StringField(pos=2, nullable=None)
 
     class Meta:
       drop_if_none = ['b']

docs/tutorial/tutorial3.txt

@@ -29,7 +29,7 @@ many, this is easily achieved by adding a dedicated manager:
   class Result(models.Model):
     id = models.IntField(pos=0) # keep id
     a  = models.StringField(pos=1)
-    b  = models.StringField(pos=2, null=None)
+    b  = models.StringField(pos=2, nullable=None)
 
     class Meta:
       manager = ResultManager
@@ -68,7 +68,7 @@ example permissions or links to models)
   class Result(models.Model):
     id = models.IntField(pos=0) # keep id
     a  = models.StringField(pos=1)
-    b  = models.StringField(pos=2, null=None)
+    b  = models.StringField(pos=2, nullable=None)
 
 
   if __name__ == "__main__":

src/data_migrator/emitters/__init__.py

@@ -2,13 +2,37 @@
 # -*- coding: UTF-8 -*-
 """Emitters are used to export models to output format.
 
-This module contains all classes for emitters: base and actuals
+This module contains all classes for emitters: base and actuals. Currently
+the system has two emitters: :class:`~.CSVEmitter` and :class:`~.MySQLEmitter`
+implemented, of which the last is the default emitter. An emitter provides the
+export format for the scanned and cleaned datasets. It also provides preambles
+and postambles in the output files, for example to clean the target table
+before loading it.
 
-* :class:`BaseEmitter`
-* :class:`MySQLEmitter`
-* :class:`CSVEmitter`
+The following classes are defined in this module:
+
+* :class:`~.BaseEmitter`
+* :class:`~.MySQLEmitter`
+* :class:`~.CSVEmitter`
+
+The basic structure for emitting is a combination between
+:class:`~.BaseManager` and :class:`~.BaseEmitter`:
+
+.. code-block:: python
+
+  e = Emitter(manager=Model.objects)
+  print e.preamble(header=[..my header lines to add..])
+  for l in Model.objects.all():
+    print e.emit(l)  # emit is returning a list of strings!
+
+.. note::
+
+    At this moment *data-migrator* does not an actively take part in schema
+    migrations of any sort. It is purely about cleaning and transforming
+    data (yet!).
 """
 
 
 from .mysql import MySQLEmitter # noqa
 from .csv import CSVEmitter # noqa
+from .singer import SingerEmitter # noqa

src/data_migrator/emitters/base.py

@@ -36,8 +36,8 @@ def emit(self, o):
     def filename(self):
         '''generate filename for this emitter.
 
-        generates a filename bases on :attr:`BaseEmitter.extension` and either
-        :attr:`~.Meta.file_name` or :attr:`~.Meta.table_name`
+        generates a filename bases on :attr:`~.BaseEmitter.extension` and
+        either :attr:`~.Options.file_name` or :attr:`~.Options.table_name`
 
         Returns:
             str: filename
@@ -58,3 +58,11 @@ def preamble(self, headers):
             list: preamble lines
         '''
         raise NotImplementedError
+
+    def postamble(self):
+        '''generate a postamble for the file to emit.
+
+        Returns:
+            list: postamble lines
+        '''
+        return []

src/data_migrator/emitters/csv.py

@@ -14,7 +14,6 @@ class CSVEmitter(BaseEmitter):
     Attributes:
         base_template: base template to output the object
         extension (str): file extension for output file of this emitter
-
     '''
     extension = '.csv'
     base_template = '''%s'''