Merge pull request #47 from autotelik/feature/finish_data_schema

Generate and read/use configuration for Import/Export

CHANGELOG.md

@@ -0,0 +1,88 @@
+# Change Log
+
+## [Unreleased](https://github.com/autotelik/datashift/tree/HEAD)
+
+[Full Changelog](https://github.com/autotelik/datashift/compare/v0.16.0...HEAD)
+
+**Closed issues:**
+
+- Issues with image import? [\#42](https://github.com/autotelik/datashift/issues/42)
+- Error when custom delimeter assigned [\#40](https://github.com/autotelik/datashift/issues/40)
+- FR: Admin gui for import/export [\#39](https://github.com/autotelik/datashift/issues/39)
+- Not working with Spree 3.0.5 stable [\#36](https://github.com/autotelik/datashift/issues/36)
+- uninitialized constant Thor::Sandbox::Datashift::Generate::ModelMapper in datashift:generate:csv [\#33](https://github.com/autotelik/datashift/issues/33)
+- Create has\_many records on import [\#32](https://github.com/autotelik/datashift/issues/32)
+
+**Merged pull requests:**
+
+- \#40 - Fix for custom delimiter set for csv files [\#41](https://github.com/autotelik/datashift/pull/41) ([wilgoszpl](https://github.com/wilgoszpl))
+- Complete rewrite and Restructure API [\#38](https://github.com/autotelik/datashift/pull/38) ([autotelik](https://github.com/autotelik))
+
+## [v0.16.0](https://github.com/autotelik/datashift/tree/v0.16.0) (2016-03-10)
+[Full Changelog](https://github.com/autotelik/datashift/compare/v0.11.0...v0.16.0)
+
+**Fixed bugs:**
+
+- default available\_on in config file [\#4](https://github.com/autotelik/datashift/issues/4)
+
+**Closed issues:**
+
+- Uninitialized constant Logging \(NameError\) [\#29](https://github.com/autotelik/datashift/issues/29)
+- uninitialized constant DataShift::SpreeHelper::SpreeBaseLoader \(NameError\) [\#28](https://github.com/autotelik/datashift/issues/28)
+- Map records with ID [\#27](https://github.com/autotelik/datashift/issues/27)
+- More Complex Examples [\#26](https://github.com/autotelik/datashift/issues/26)
+- Can't link the github repo in the Gemfile when deploying on Heroku [\#23](https://github.com/autotelik/datashift/issues/23)
+- uninitialized constant DataShift::Logging [\#21](https://github.com/autotelik/datashift/issues/21)
+- All exports fail with "Please supply array of records to export" [\#20](https://github.com/autotelik/datashift/issues/20)
+- How can i add image with product when import csv with datashift? [\#19](https://github.com/autotelik/datashift/issues/19)
+- Setting specific values on variants [\#13](https://github.com/autotelik/datashift/issues/13)
+
+**Merged pull requests:**
+
+- Rails 4.2.0 changed 'type\_cast' name to type\_cast\_from\_database [\#31](https://github.com/autotelik/datashift/pull/31) ([redglory](https://github.com/redglory))
+- CSV support [\#25](https://github.com/autotelik/datashift/pull/25) ([pnomolos](https://github.com/pnomolos))
+- Travis Support [\#24](https://github.com/autotelik/datashift/pull/24) ([pnomolos](https://github.com/pnomolos))
+- Replace call to trim! with strip! [\#18](https://github.com/autotelik/datashift/pull/18) ([swrobel](https://github.com/swrobel))
+- Fixed file descriptors leak in paperclip\_loader [\#16](https://github.com/autotelik/datashift/pull/16) ([arr-ee](https://github.com/arr-ee))
+
+## [v0.11.0](https://github.com/autotelik/datashift/tree/v0.11.0) (2012-10-23)
+[Full Changelog](https://github.com/autotelik/datashift/compare/v0.10.0...v0.11.0)
+
+## [v0.10.0](https://github.com/autotelik/datashift/tree/v0.10.0) (2012-09-25)
+[Full Changelog](https://github.com/autotelik/datashift/compare/v0.9.0...v0.10.0)
+
+**Closed issues:**
+
+- spec/fixtures/spree not available [\#15](https://github.com/autotelik/datashift/issues/15)
+- Regression on image loading when using master [\#14](https://github.com/autotelik/datashift/issues/14)
+- Spree import does not allow sub categories of same name [\#12](https://github.com/autotelik/datashift/issues/12)
+- thor csv processing options are missing? [\#8](https://github.com/autotelik/datashift/issues/8)
+
+## [v0.9.0](https://github.com/autotelik/datashift/tree/v0.9.0) (2012-09-03)
+[Full Changelog](https://github.com/autotelik/datashift/compare/v0.6.0...v0.9.0)
+
+**Closed issues:**
+
+- Not recognizing Jruby [\#11](https://github.com/autotelik/datashift/issues/11)
+- doesn't scan directory when loading images [\#9](https://github.com/autotelik/datashift/issues/9)
+- `perform\_load': .rb files not supported [\#7](https://github.com/autotelik/datashift/issues/7)
+- How to attach Images to product? [\#6](https://github.com/autotelik/datashift/issues/6)
+- spree variants and multiple option\_types [\#5](https://github.com/autotelik/datashift/issues/5)
+- What is your environment? [\#3](https://github.com/autotelik/datashift/issues/3)
+
+## [v0.6.0](https://github.com/autotelik/datashift/tree/v0.6.0) (2012-05-18)
+[Full Changelog](https://github.com/autotelik/datashift/compare/v0.5.0...v0.6.0)
+
+## [v0.5.0](https://github.com/autotelik/datashift/tree/v0.5.0) (2012-05-08)
+[Full Changelog](https://github.com/autotelik/datashift/compare/v0.4.1...v0.5.0)
+
+## [v0.4.1](https://github.com/autotelik/datashift/tree/v0.4.1) (2012-03-19)
+[Full Changelog](https://github.com/autotelik/datashift/compare/v0.4.0...v0.4.1)
+
+## [v0.4.0](https://github.com/autotelik/datashift/tree/v0.4.0) (2012-03-12)
+[Full Changelog](https://github.com/autotelik/datashift/compare/v0.0.2...v0.4.0)
+
+## [v0.0.2](https://github.com/autotelik/datashift/tree/v0.0.2) (2012-02-12)
+
+
+\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*

Gemfile

@@ -7,7 +7,9 @@ gemspec
 
 group :test do
 
-  gem "rspec-rails", "~> 3.4"
+  gem 'byebug'
+
+  gem 'rspec-rails', '~> 3.4'
 
   platform :jruby do
     gem 'jruby-openssl'

datashift.gemspec

@@ -23,7 +23,7 @@ Gem::Specification.new do |s|
 
   s.require_paths = ['lib']
 
-  s.add_runtime_dependency "rails", ">= 4.2", "< 5.1"
+  s.add_runtime_dependency 'rails', '>= 4.2', '< 5.1'
 
   s.add_runtime_dependency 'paperclip', '~> 4'
   s.add_runtime_dependency 'spreadsheet', '~> 1.1'
@@ -39,5 +39,4 @@ Gem::Specification.new do |s|
   s.add_development_dependency 'factory_girl_rails', '~> 4.5'
   s.add_development_dependency 'database_cleaner', '~> 1.5'
 
-
 end

lib/datashift.rb

@@ -62,6 +62,7 @@ def self.require_libraries
 
     begin
       require_relative 'datashift/delimiters'
+      require_relative 'datashift/load_object'
       require_relative 'generators/generator_base'
       require_relative 'loaders/reporters/reporter'
       require_relative 'loaders/loader_base'
@@ -70,24 +71,24 @@ def self.require_libraries
       puts "Problem initializing gem #{x.inspect}"
     end
 
-    require_libs = %w{
+    require_libs = %w(
       datashift/core_ext
       datashift
       datashift/mapping
       datashift/model_methods
-      datashift/transformer
+      datashift/transformation
       datashift/inbound_data
       loaders
       loaders/reporters
       exporters
       generators
       helpers
       applications
-    }
+    )
 
     require_libs.each do |base|
       Dir[File.join(library_path, base, '*.rb')].each do |rb|
-        #puts rb
+        # puts rb
         begin
           require_relative rb unless File.directory?(rb)
         rescue => x

lib/datashift/binder.rb

@@ -64,11 +64,11 @@ def forced_inclusion_columns
     end
 
     def forced
-      @forced ||= forced_inclusion_columns.compact.collect { |f| f.to_s.downcase }
+      forced_inclusion_columns.compact.collect { |f| f.to_s.downcase }
     end
 
     def forced?(column_name)
-      forced.include?(column_name.downcase)
+      forced.include?(column_name.to_s.downcase)
     end
 
     def include_all?
@@ -126,18 +126,19 @@ def map_inbound_headers(klass, columns)
         model_method = model_method_mgr.search(raw_col_name)
 
         # if not found via raw name, try various alternatives
+
+        if( model_method.nil? && (include_all? || forced?(raw_col_name)) )
+          logger.debug("Operator #{raw_col_name} not found but forced inclusion set - adding as :method")
+          model_method = model_method_mgr.insert(raw_col_name, :method)
+        end
+
         unless model_method
           Binder.substitutions(raw_col_name).each do |n|
             model_method = model_method_mgr.search(n)
             break if model_method
           end
         end
 
-        if( model_method.nil? && (include_all? || forced?(raw_col_name)) )
-          logger.debug("Operator #{raw_col_name} not found but forced inclusion set - adding as :method")
-          model_method = model_method_mgr.insert(raw_col_name, :method)
-        end
-
         if(model_method)
 
           binding = MethodBinding.new(raw_col_name, col_index, model_method)

lib/datashift/configuration.rb

@@ -35,6 +35,13 @@ class Configuration
     #
     attr_accessor :remove_columns
 
+    # List of headers/columns that are Mandatory i.e must be present in the inbound data
+    #
+    # @param [Array<#call>] List of headers/columns that are Mandatory
+    # @return [Array<#call>]
+    #
+    attr_accessor :mandatory
+
     # @param [Boolean] Remove standard Rails cols like :id, created_at etc
     # Default is false - i.e id, created_at etc are included by default
     # @return [Boolean]
@@ -123,7 +130,7 @@ def initialize
       @verbose = false
       @dummy_run = false
       @force_inclusion_of_columns = []
-      @exclude_associations=[]
+      @exclude_associations = []
 
       @expand_associations = false
 
@@ -157,7 +164,6 @@ def self.configure
       yield call
     end
 
-
     # Prepare the operators types in scope based on number of configuration attributes
     # Default is assignment only
     #

lib/datashift/context_factory.rb

@@ -60,17 +60,28 @@ def self.configure(load_object_class, yaml_file)
     #                    DataShift::Populator.new
     #                  end
 
+    # Set a Populator to be used against an INBOUND operator
+
+    class << self
+      attr_writer :global_populator_class
+    end
+
+    def self.global_populator_class
+      @global_populator_class || DataShift::Populator
+    end
+
     def self.set_populator(method_binding, klass)
-      populators[method_binding.operator] = klass
+      operator = method_binding.is_a?(DataShift::MethodBinding) ? method_binding.operator : method_binding
+      populators[operator] = klass
     end
 
     def self.get_populator(method_binding)
 
-      unless method_binding.nil? || method_binding.invalid?
-        return populators[method_binding.operator].new if populators.key?(method_binding.operator)
-      end
+      if(populators.key?(method_binding.operator))
+        return populators[method_binding.operator].new
+      end unless method_binding.nil? || method_binding.invalid?
 
-      DataShift::Populator.new
+      global_populator_class.new
     end
 
   end

lib/datashift/core_ext/array.rb

@@ -1,6 +1,6 @@
 Array.class_eval do
 
-  ARRAY_FWDABLE_EXCLUDED_METHODS  = [
+  ARRAY_FWDABLE_EXCLUDED_METHODS = [
     :class, :singleton_class, :clone, :dup, :initialize_dup, :initialize_clone,
     :freeze, :methods, :singleton_methods, :protected_methods, :private_methods, :public_methods,
     :instance_variables, :instance_variable_get, :instance_variable_set, :instance_variable_defined?,

lib/datashift/doc_context.rb

@@ -28,7 +28,7 @@ class DocContext
     def initialize( klass )
       reset_klass(klass)
 
-      @headers  = DataShift::Headers.new(:na)
+      @headers = DataShift::Headers.new(:na)
 
       @progress_monitor = ProgressMonitor.new
 
@@ -45,7 +45,7 @@ def reset_klass( klass )
     def reset(object = nil)
       @node_context = DataShift::EmptyContext.new
 
-      @load_object = LoadObject.new(object || new_load_object)
+      @load_object =  DataShift::LoadObject.new(object || new_load_object)
     end
 
     def new_load_object
@@ -73,7 +73,6 @@ def failed_count
       progress_monitor.failed_count
     end
 
-
     def current_errors
       load_object.errors.full_messages
     end
@@ -97,7 +96,6 @@ def save_if_new
       raise DataShift::SaveError, "Cannot Save Invalid #{load_object.class} Record : #{current_errors}"
     end
 
-
     # Save the object and then report the outcome to ProgressMonitor, as either success or failure
     #
     def save_and_monitor_progress

lib/datashift/exceptions.rb

@@ -12,7 +12,7 @@ class DataShiftException < StandardError
 
     def initialize( msg )
       super
-      logger.error( msg)
+      logger.error(msg)
     end
 
     def self.generate(name)
@@ -26,22 +26,26 @@ def initialize( msg )
     end
   end
 
-  class BadRuby < StandardError; end
+  # Non self logging errors
 
-  class UnsupportedFileType < StandardError; end
+  class BadConfig < StandardError; end
   class BadFile < StandardError; end
+  class BadRuby < StandardError; end
+  class BadUri < StandardError; end
 
   class MappingDefinitionError < StandardError; end
-
   class MissingHeadersError < StandardError; end
   class MissingMandatoryError < StandardError; end
 
   class PathError < StandardError; end
 
-  class BadUri < StandardError; end
+  class RuntimeError < StandardError; end
 
+  class UnsupportedFileType < StandardError; end
 end
 
+# Self logging errors
+
 DataShift::DataShiftException.generate( 'BadOperatorType')
 DataShift::DataShiftException.generate( 'ConnectionError')
 DataShift::DataShiftException.generate( 'CouldNotAssignAssociation')

lib/datashift/header.rb

@@ -3,21 +3,21 @@
 # Date ::     Feb 2016
 # License::   MIT
 #
-# Details::   A header
-#             The source text
+# Details::   A column header
 #
 module DataShift
 
   class Header
 
-    attr_accessor :source
+    attr_accessor :source, :presentation
 
     def initialize(source:)
       @source = source
+      @presentation = source
     end
 
     def to_s
-      source
+      presentation
     end
 
   end

lib/datashift/headers.rb

@@ -37,7 +37,7 @@ def self.klass_to_headers(klass)
 
       headers.class_source_to_headers
 
-      DataShift::Transformer::Remove.new.unwanted_headers(headers)
+      DataShift::Transformation::Remove.new.unwanted_headers(headers)
 
       headers
     end
@@ -47,15 +47,15 @@ def self.klass_to_headers(klass)
     #
     def class_source_to_headers
 
-      raise SourceIsNotAClass, "Cannot parse source for headers - source must be a Class" unless source.is_a?(Class)
+      raise SourceIsNotAClass, 'Cannot parse source for headers - source must be a Class' unless source.is_a?(Class)
 
       # TODO: This collection can now be sorted
       collection = ModelMethods::Manager.catalog_class(source)
 
       configuration = DataShift::Configuration.call
 
       collection.each do |mm|
-        next if(DataShift::Transformer::Remove.new.association?(mm))
+        next if(DataShift::Transformation::Remove.new.association?(mm))
 
         if(mm.association_type?)
           association_to_headers(mm)