dffml: cli: Pass model specific config options #31

sudharsana-kjl · 2019-03-22T11:43:58Z

Implement a way to pass model specific config options

sudharsana-kjl · 2019-03-22T11:44:19Z

I have tried to fetch -modelParams value but i'm getting the following error:

(env) sudharsana@sudharsana-HP-15-Notebook-PC:~/dffml(master)$ dffml train   -model dnn   -sources csv=iris_training.csv   -classifications 0 1 2   -features     def:SepalLength:float:1     def:SepalWidth:float:1     def:PetalLength:float:1     def:PetalWidth:float:1   -num_epochs 3000   -steps 20000   -log debug -modelParams hidden_units=[12, 40, 15]
/home/sudharsana/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
usage: dffml [-h] [-log LOG]
             {import,list,accuracy,applicable,edit,evaluate,export,merge,predict,service,train,version}
             ...
dffml: error: unrecognized arguments: -modelParams hidden_units=[12, 40, 15]

I have made changes in dffml/util/cli.py Could you please tell me where I should add the -modelParams argument?

codecov-io · 2019-03-22T11:46:26Z

Codecov Report

Merging #31 into master will decrease coverage by 0.33%.
The diff coverage is 40.9%.

@@            Coverage Diff             @@
##           master      #31      +/-   ##
==========================================
- Coverage   88.97%   88.64%   -0.34%     
==========================================
  Files          46       47       +1     
  Lines        3230     3250      +20     
  Branches      336      340       +4     
==========================================
+ Hits         2874     2881       +7     
- Misses        306      318      +12     
- Partials       50       51       +1

Impacted Files	Coverage Δ
dffml/util/cli/base.py	`100% <ø> (ø)`	⬆️
dffml/util/cli/cmd.py	`96.29% <0%> (-3.71%)`	⬇️
dffml/util/cli/__init__.py	`100% <100%> (ø)`
dffml/util/cli/parser.py	`77.41% <20%> (-6.92%)`	⬇️
dffml/model/model.py	`78.12% <75%> (-1.88%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a20211f...f3f6f85. Read the comment docs.

sudharsana-kjl · 2019-03-22T13:21:34Z

Fixed the previous issue. It was due to an indentation error. I'm currently working on fetching the params inside dnn model.

sudharsana-kjl · 2019-03-22T13:24:12Z

Hey Yash! I haven't modified dffml/cli.py. I think this requires changes in dffml/util/cli.py mainly to fetch the value. Please let me know if you have any suggestions. Regards,

…

On Fri, Mar 22, 2019 at 5:57 PM Yash Lamba ***@***.***> wrote: Hey! @sudharsana-kjl <https://github.com/sudharsana-kjl>, did you modify dffml/cli.py? I maybe wrong but I don't see any modifications to that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#31 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJpIdVxmIbzwMLp4TOLYc4UdrHKJipztks5vZMwogaJpZM4cDZSk> .

yashlamba · 2019-03-22T14:21:14Z

Yeah, I saw that later. Is the error resolved now?

sudharsana-kjl · 2019-03-22T14:24:26Z

Hey Yash! Yes. The error has been resolved now. I'm working on implementing the feature. Thanks!

…

On Fri, 22 Mar, 2019, 7:51 PM Yash Lamba, ***@***.***> wrote: Yeah, I saw that later. Is the error resolved now? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#31 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJpIdfoZBfCBK8amDMAwy4cnpeg3eJr9ks5vZObagaJpZM4cDZSk> .

johnandersen777 · 2019-03-22T15:40:48Z

You're on the right track! Apologies for the lack of comments in that file ( and well, all of them ) I think you may need to add a sub parser or a group or something.

The utils/cli is a custom abstraction layer I wrote around argparse.

dffml/dffml/util/cli.py

Lines 168 to 169 in f2a42ca

    
           parser = subparsers.add_parser(name, help=None \ 
        
                   if method.__doc__ is None else method.__doc__.strip())

Also, you'll want to change the model action so that the second () is used to pass the arguments parsed in via the sub parser or whatever

dffml/dffml/util/cli.py

Lines 37 to 40 in f2a42ca

    
           class ParseModelAction(argparse.Action): 
        
               def __call__(self, parser, namespace, value, option_string=None): 
        
                   setattr(namespace, self.dest, Model.load(value)())

So:

Model.load(value)(**dict_of_args_for_model)

sudharsana-kjl · 2019-03-23T01:59:33Z

Hey @pdxjohnny! where should I add the sub parser? Every argument is parsed by Arg objects inside which we mention the parser action.

I have added the Arg object here:

dffml/dffml/util/cli.py

Lines 234 to 235 in 89df20f

    
           arg_modelParams = Arg('-modelParams',help='Model specific params',  
        
                   nargs='*', action=ParseModelParamsAction, default=None)

I have added the corresponding parser action here:

dffml/dffml/util/cli.py

Lines 42 to 45 in 89df20f

    
           class ParseModelParamsAction(argparse.Action): 
        
               def __call__(self, parser, namespace, value, option_string=None): 
        
                   setattr(namespace, self.dest, Model.load(value)())

I'm getting the error that -modelParams not found again.

sudharsana-kjl · 2019-03-24T09:29:40Z

After a long struggle, It finally works when -modelParams parameter is added in dffml train command.

dffml train \
  -model dnn \
  -sources csv=iris_training.csv \
  -classifications 0 1 2 \
  -features \
    def:SepalLength:float:1 \
    def:SepalWidth:float:1 \
    def:PetalLength:float:1 \
    def:PetalWidth:float:1 \
  -num_epochs 3000 \
  -steps 20000 \
  -log debug \
  -modelParams \
     hidden_units=[10,20,10]

I'm facing the following issue when I try to run dffml accuracy command:

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

tensor_name = dnn/hiddenlayer_0/bias; shape in shape_and_slice spec [12] does not match the shape stored in checkpoint: [10]
	 [[node save/RestoreV2_1 (defined at /home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py:1537) ]]

The same error persists when I'm switching from training the model without -modelParams to training it with -modelParams

When I googled the error, I found out that clearing the model_dir resolves this. So, when I switched to training the model with -modelParams, I clear out the model_dir.

But in this case, clearing out the model_dir will delete the training model and there wont be a trained model for running further commands.

@pdxjohnny Could you please give your suggestions on this?

sudharsana-kjl · 2019-03-24T09:35:17Z

Complete log for your reference:

(env) sudharsana@sudharsana-HP-15-Notebook-PC:~/GSoC/dffml(master)$ dffml accuracy   -model dnn   -sources csv=iris_test.csv   -classifications 0 1 2   -features     def:SepalLength:float:1     def:SepalWidth:float:1     def:PetalLength:float:1     def:PetalWidth:float:1   -log critical2019-03-24 15:02:48.858281: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-03-24 15:02:48.862656: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2394360000 Hz
2019-03-24 15:02:48.862864: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4b05800 executing computations on platform Host. Devices:
2019-03-24 15:02:48.862891: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-03-24 15:02:48.895619: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Invalid argument: tensor_name = dnn/hiddenlayer_0/bias; shape in shape_and_slice spec [12] does not match the shape stored in checkpoint: [10]
Traceback (most recent call last):
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: tensor_name = dnn/hiddenlayer_0/bias; shape in shape_and_slice spec [12] does not match the shape stored in checkpoint: [10]
	 [[{{node save/RestoreV2_1}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 1276, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: tensor_name = dnn/hiddenlayer_0/bias; shape in shape_and_slice spec [12] does not match the shape stored in checkpoint: [10]
	 [[node save/RestoreV2_1 (defined at /home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py:1537) ]]

Caused by op 'save/RestoreV2_1', defined at:
  File "/home/sudharsana/GSoC/dffml/env/bin/dffml", line 11, in <module>
    load_entry_point('dffml==0.1.1', 'console_scripts', 'dffml')()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml-0.1.1-py3.6.egg/dffml/util/cli.py", line 154, in main
    result = loop.run_until_complete(cls.cli(*argv[1:]))
  File "/usr/lib/python3.6/asyncio/base_events.py", line 471, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 438, in run_forever
    self._run_once()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 1451, in _run_once
    handle._run()
  File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml-0.1.1-py3.6.egg/dffml/util/cli.py", line 135, in cli
    return await cmd.run()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml-0.1.1-py3.6.egg/dffml/cli.py", line 213, in run
    self.classifications))
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml_model_tensorflow-0.1.2-py3.6.egg/dffml_model_tensorflow/model/dnn.py", line 231, in accuracy
    .evaluate(input_fn=input_fn)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 469, in evaluate
    name=name)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 511, in _actual_eval
    return _evaluate()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 500, in _evaluate
    output_dir=self.eval_dir(name))
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 1537, in _evaluate_run
    config=self._session_config)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/evaluation.py", line 271, in _evaluate_once
    session_creator=session_creator, hooks=hooks) as session:
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 934, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 648, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 1122, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 1127, in _create_session
    return self._sess_creator.create_session()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 805, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 562, in create_session
    self._scaffold.finalize()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 217, in finalize
    self._saver = training_saver._get_saver_or_default()  # pylint: disable=protected-access
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 604, in _get_saver_or_default
    saver = Saver(sharded=True, allow_empty=True)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 507, in _build_internal
    restore_sequentially, reshape)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 385, in _AddShardedRestoreOps
    name="restore_shard"))
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): tensor_name = dnn/hiddenlayer_0/bias; shape in shape_and_slice spec [12] does not match the shape stored in checkpoint: [10]
	 [[node save/RestoreV2_1 (defined at /home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py:1537) ]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sudharsana/GSoC/dffml/env/bin/dffml", line 11, in <module>
    load_entry_point('dffml==0.1.1', 'console_scripts', 'dffml')()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml-0.1.1-py3.6.egg/dffml/util/cli.py", line 154, in main
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml-0.1.1-py3.6.egg/dffml/util/cli.py", line 135, in cli
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml-0.1.1-py3.6.egg/dffml/cli.py", line 213, in run
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml_model_tensorflow-0.1.2-py3.6.egg/dffml_model_tensorflow/model/dnn.py", line 231, in accuracy
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 469, in evaluate
    name=name)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 511, in _actual_eval
    return _evaluate()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 500, in _evaluate
    output_dir=self.eval_dir(name))
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 1537, in _evaluate_run
    config=self._session_config)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/evaluation.py", line 271, in _evaluate_once
    session_creator=session_creator, hooks=hooks) as session:
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 934, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 648, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 1122, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 1127, in _create_session
    return self._sess_creator.create_session()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 805, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 571, in create_session
    init_fn=self._scaffold.init_fn)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/session_manager.py", line 281, in prepare_session
    config=config)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/session_manager.py", line 195, in _restore_checkpoint
    saver.restore(sess, checkpoint_filename_with_path)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 1312, in restore
    err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

tensor_name = dnn/hiddenlayer_0/bias; shape in shape_and_slice spec [12] does not match the shape stored in checkpoint: [10]
	 [[node save/RestoreV2_1 (defined at /home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py:1537) ]]

Caused by op 'save/RestoreV2_1', defined at:
  File "/home/sudharsana/GSoC/dffml/env/bin/dffml", line 11, in <module>
    load_entry_point('dffml==0.1.1', 'console_scripts', 'dffml')()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml-0.1.1-py3.6.egg/dffml/util/cli.py", line 154, in main
    result = loop.run_until_complete(cls.cli(*argv[1:]))
  File "/usr/lib/python3.6/asyncio/base_events.py", line 471, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 438, in run_forever
    self._run_once()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 1451, in _run_once
    handle._run()
  File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml-0.1.1-py3.6.egg/dffml/util/cli.py", line 135, in cli
    return await cmd.run()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml-0.1.1-py3.6.egg/dffml/cli.py", line 213, in run
    self.classifications))
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/dffml_model_tensorflow-0.1.2-py3.6.egg/dffml_model_tensorflow/model/dnn.py", line 231, in accuracy
    .evaluate(input_fn=input_fn)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 469, in evaluate
    name=name)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 511, in _actual_eval
    return _evaluate()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 500, in _evaluate
    output_dir=self.eval_dir(name))
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 1537, in _evaluate_run
    config=self._session_config)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/evaluation.py", line 271, in _evaluate_once
    session_creator=session_creator, hooks=hooks) as session:
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 934, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 648, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 1122, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 1127, in _create_session
    return self._sess_creator.create_session()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 805, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 562, in create_session
    self._scaffold.finalize()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 217, in finalize
    self._saver = training_saver._get_saver_or_default()  # pylint: disable=protected-access
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 604, in _get_saver_or_default
    saver = Saver(sharded=True, allow_empty=True)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 507, in _build_internal
    restore_sequentially, reshape)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 385, in _AddShardedRestoreOps
    name="restore_shard"))
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow-1.13.1-py3.6-linux-x86_64.egg/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

tensor_name = dnn/hiddenlayer_0/bias; shape in shape_and_slice spec [12] does not match the shape stored in checkpoint: [10]
	 [[node save/RestoreV2_1 (defined at /home/sudharsana/GSoC/dffml/env/lib/python3.6/site-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py:1537) ]]

model/tensorflow/dffml_model_tensorflow/model/dnn.py

johnandersen777

Very nice work! Darn, tensorflow is a constant source of odd behavior. I'll look into this and get back to you about what I find this week.

You've got all the code that needs to be involved here. But I've left a few comments as to how we should re-structure things a bit.

Thank you!

dffml/util/cli.py

model/tensorflow/dffml_model_tensorflow/model/dnn.py

johnandersen777 · 2019-03-31T01:24:20Z

Tensorflow seems to be working for me now.

With regards to what I said about __dict__, we now want to put the ast parsing in the Parse...Action

Similar to

dffml/dffml/util/cli/parser.py

Lines 95 to 103 in 062946b

    
           class ParseOutputSpecsAction(argparse.Action): 
        
               def __call__(self, parser, namespace, values, option_string=None): 
        
                   if not isinstance(values, list): 
        
                       values = [values] 
        
                   ouput_specs = [(ast.literal_eval(value.split('=', maxsplit=2)[0]), 
        
                                   value.split('=', maxsplit=2)[1],) \ 
        
                                  for value in values] 
        
                   setattr(namespace, self.dest, ouput_specs)

Then have the CMD do something similar to the FeaturesCMD

dffml/dffml/util/cli/cmd.py

Lines 91 to 98 in 062946b

    
           arg_features = Arg('-features', nargs='+', required=True, 
        
                   default=Features(), action=ParseFeaturesAction) 
        
           arg_timeout = Arg('-timeout', help='Feature evaluation timeout', 
        
                   required=False, default=Features.TIMEOUT, type=int) 
        
           def __init__(self, *args, **kwargs): 
        
               super().__init__(*args, **kwargs) 
        
               self.features.timeout = self.timeout

Sorry if that's not clear, let me know if it's not.

sudharsana-kjl · 2019-04-04T00:30:33Z

@pdxjohnny Instead of storing the model params as dict, would you like to load it as how it is done in output_specs?

Could you give me the expected format in which you plan to get the params from the user and use it in the model?

Also, Should I create a separate ModelParamsCMD class or can I add this Arg to the existing ModelCMD class?

johnandersen777 · 2019-04-05T00:02:21Z

Hi @sudharsana-kjl. I'm going to suggest that we come at this from a different approach. (Sorry for changing things so much, you may want to follow HACKING.md/git and create a new PR, so as to not deal with merge conflicts)

When you opened this pull request it was before the Data Flow Facilitator has been merged.

Now that it is, I think there's a better way to pass options (not perfect, but hopefully better)

Looking at DataFlowFacilitatorCMD:

dffml/dffml/util/cli/cmd.py

Lines 176 to 193 in bf3493e

    
               # Load all entrypoints which may possibly be selected. Then have them add 
        
               # their arguments to the DataFlowFacilitator-tots command. 
        
               @classmethod 
        
               def add_bases(cls): 
        
                   class LoadedDataFlowFacilitator(cls): 
        
                       pass 
        
                   for base in [BaseInputNetwork, 
        
                                BaseOperationNetwork, 
        
                                BaseLockNetwork, 
        
                                BaseRedundancyChecker, 
        
                                BaseOperationImplementationNetwork, 
        
                                BaseOrchestrator]: 
        
                       for loaded in base.load(): 
        
                           for arg_name, arg in loaded.args().items(): 
        
                               setattr(LoadedDataFlowFacilitator, arg_name, arg) 
        
                   return LoadedDataFlowFacilitator 
        
           DataFlowFacilitatorCMD = BaseDataFlowFacilitatorCMD.add_bases()

It calls load() on all the df base classes to load any classes that have been registered with setuptools as an entrypoint for that class.

For instance, tensorlfow DNN is registered in it's setup.py file:

dffml/model/tensorflow/setup.py

Lines 54 to 58 in bf3493e

    
           entry_points={ 
        
               'dffml.model': [ 
        
                   'dnn = dffml_model_tensorflow.model.dnn:DNN', 
        
               ], 
        
           },

You'd want to copy the df behavior for models. Which means making anything deriving from Model define args and config methods.

dffml/dffml/df/base.py

Lines 17 to 25 in bf3493e

    
           @classmethod 
        
           @abc.abstractmethod 
        
           def args(cls) -> Dict[str, Arg]: 
        
               pass 
        
           @classmethod 
        
           @abc.abstractmethod 
        
           def config(cls, cmd: CMD): 
        
               pass

dffml/dffml/df/memory.py

Lines 658 to 678 in bf3493e

    
           @classmethod 
        
           def args(cls) -> Dict[str, Arg]: 
        
               # Enable the user to specify operation implementations to be loaded via 
        
               # the entrypoint system (by ParseOperationImplementationAction) 
        
               args = { 
        
                       'arg_opimpn_memory_opimps': Arg('-opimpn-memory-opimps', 
        
                           nargs='+', 
        
                           action=ParseOperationImplementationAction) 
        
                       } 
        
               # TODO similar to MemoryRedundancyChecker, load all Operation 
        
               # Implemenations and have them add args if they need it 
        
               return args 
        
           @classmethod 
        
           def config(cls, cmd: CMD) -> BaseConfig: 
        
               return MemoryOperationImplementationNetworkConfig( 
        
                       operations={imp.op.name: imp \ 
        
                                   for imp in \ 
        
                                   [Imp(BaseConfig()) \ 
        
                                   for Imp in cmd.opimpn_memory_opimps]} 
        
                   )

We will eventually need to do this for sources too.

sudharsana-kjl · 2019-04-14T13:58:31Z

@pdxjohnny Please review and let me know your suggestions on this. Also, could you tell me how I should approach to fix the build fail? I see that the problem is in test_model but its inside "/home/travis/build/intel/dffml/model/travis_test_model/tests/test_model.py"

sudharsana-kjl · 2019-05-02T05:38:46Z

@pdxjohnny fixed the build error, it was a simple code change. Sorry, I didn't look at it properly last time. Please review the code changes and let me know if this is alright.

johnandersen777 · 2019-05-16T14:37:54Z

Hi @sudharsana-kjl, thanks for your work on this. However, as I've been going through #46 to do the prep for your sources proposal I found that I had made some incorrect assumptions about Parse...Actons so don't worry about this PR for now and focus on labels and versions.

sudharsana-kjl · 2019-05-17T01:11:59Z

Thanks @pdxjohnny !

johnandersen777 · 2019-05-21T16:03:19Z

Closed via #71

sudharsana-kjl added 12 commits March 10, 2019 08:20

remove _close in csvfile.py

7bd4110

source: csv: Add update funcationality

4a2bf4e

source: csv: modify load_fd to include extra headers if exists

05a0084

tests: test_csv: add test for dump_fd

19c970a

feature: git: add logger for binary inpath error in cloc.py

1f870c2

fix merge conflicts

dbda7f7

fix: add new line at eof

85a6a55

modify GitClocFeature to print error

f82ed8f

modify CHANGELOG.md

8f165f1

modify CHANGELOG

17e0a53

Merge remote-tracking branch 'upstream/master'

40e787b

add functions to fetch -modelParams

98223d6

sudharsana-kjl added 3 commits March 22, 2019 18:06

add print statements to see if modelParams is fetched

8e1adb3

Try to get params inside dnn

77b0cb8

fix merge conflicts

89df20f

sudharsana-kjl added 2 commits March 24, 2019 10:57

Merge remote-tracking branch 'upstream/master'

e49b2df

add -modelParams for model train

7375838

johnandersen777 reviewed Mar 24, 2019

View reviewed changes

model/tensorflow/dffml_model_tensorflow/model/dnn.py Outdated Show resolved Hide resolved

johnandersen777 suggested changes Mar 24, 2019

View reviewed changes

dffml/util/cli.py Outdated Show resolved Hide resolved

dffml/util/cli.py Outdated Show resolved Hide resolved

model/tensorflow/dffml_model_tensorflow/model/dnn.py Outdated Show resolved Hide resolved

Make requested changes

86fc14e

sudharsana-kjl added 4 commits March 24, 2019 22:54

remove tab space

2aea82b

fix build issue

d3a04e9

modify changelog

08f7290

fix merge conflict

02aa86a

johnandersen777 approved these changes Mar 29, 2019

View reviewed changes

sudharsana-kjl changed the title ~~WIP: dffml: cli: Pass model specific config options~~ dffml: cli: Pass model specific config options Mar 30, 2019

fix merge conflict

9c73f94

sudharsana-kjl added 4 commits April 14, 2019 06:35

Merge remote-tracking branch 'upstream/master'

7d4baf7

add -modelParams feature in new dataflow facilitator architecture

f186dd7

remove trailing whitespace and modify test_cli

2e3a3fe

fix trailing white space in model.py

8c599bd

add args and config in Misc to fix build error

f3f6f85

sudharsana-kjl added 2 commits May 16, 2019 12:34

Merge remote-tracking branch 'upstream/master'

aa46178

fix build error

6c28080

johnandersen777 closed this May 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dffml: cli: Pass model specific config options #31

dffml: cli: Pass model specific config options #31

sudharsana-kjl commented Mar 22, 2019

sudharsana-kjl commented Mar 22, 2019

codecov-io commented Mar 22, 2019 •

edited

Loading

sudharsana-kjl commented Mar 22, 2019

sudharsana-kjl commented Mar 22, 2019 via email

yashlamba commented Mar 22, 2019

sudharsana-kjl commented Mar 22, 2019 via email

johnandersen777 commented Mar 22, 2019 •

edited

Loading

sudharsana-kjl commented Mar 23, 2019 •

edited

Loading

sudharsana-kjl commented Mar 24, 2019

sudharsana-kjl commented Mar 24, 2019

johnandersen777 left a comment

johnandersen777 commented Mar 31, 2019 •

edited

Loading

sudharsana-kjl commented Apr 4, 2019

johnandersen777 commented Apr 5, 2019

sudharsana-kjl commented Apr 14, 2019

sudharsana-kjl commented May 2, 2019

johnandersen777 commented May 16, 2019

sudharsana-kjl commented May 17, 2019

johnandersen777 commented May 21, 2019

dffml: cli: Pass model specific config options #31

dffml: cli: Pass model specific config options #31

Conversation

sudharsana-kjl commented Mar 22, 2019

sudharsana-kjl commented Mar 22, 2019

codecov-io commented Mar 22, 2019 • edited Loading

Codecov Report

sudharsana-kjl commented Mar 22, 2019

sudharsana-kjl commented Mar 22, 2019 via email

yashlamba commented Mar 22, 2019

sudharsana-kjl commented Mar 22, 2019 via email

johnandersen777 commented Mar 22, 2019 • edited Loading

sudharsana-kjl commented Mar 23, 2019 • edited Loading

sudharsana-kjl commented Mar 24, 2019

sudharsana-kjl commented Mar 24, 2019

johnandersen777 left a comment

Choose a reason for hiding this comment

johnandersen777 commented Mar 31, 2019 • edited Loading

sudharsana-kjl commented Apr 4, 2019

johnandersen777 commented Apr 5, 2019

sudharsana-kjl commented Apr 14, 2019

sudharsana-kjl commented May 2, 2019

johnandersen777 commented May 16, 2019

sudharsana-kjl commented May 17, 2019

johnandersen777 commented May 21, 2019

codecov-io commented Mar 22, 2019 •

edited

Loading

johnandersen777 commented Mar 22, 2019 •

edited

Loading

sudharsana-kjl commented Mar 23, 2019 •

edited

Loading

johnandersen777 commented Mar 31, 2019 •

edited

Loading