python setup.py install doesn't copy over central metadata

[JackKelly]

Oli emailed this bug report:

---

Hi Jack,

I'm trying to write the metadata for wikienergy at the moment, but have hit a problem when I try to read in the building1.yaml metadata for my dataset. When I don't include building1.yaml in the metadata, things seem to work just fine. However, when I do include it, it generates this stacktrace:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
/usr/lib/python2.7/dist-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
    176             else:
    177                 filename = fname
--> 178             __builtin__.execfile(filename, *where)

/home/oli/nilmtk/nilmtk/dataset_converters/wikienergy/test_dataset_wikienergy.py in <module>()
     10 print 'testing queries over single table'
     11 houses = {26:('2014-05-02','2014-05-03')}
---> 12 download_wikienergy(database_username, database_password, output_directory, periods_to_load=houses)
     13 print('')
     14 '''

/home/oli/nilmtk/nilmtk/dataset_converters/wikienergy/download_wikienergy.py in download_wikienergy(database_username, database_password, hdf_filename, periods_to_load)
    206
    207     convert_yaml_to_hdf5(join(_get_module_directory(), 'metadata'),
--> 208                          hdf_filename)
    209
    210 def _wikienergy_dataframe_to_hdf(wikienergy_dataframe, store, nilmtk_building_id):

/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/convert_yaml_to_hdf5.pyc in convert_yaml_to_hdf5(yaml_dir, hdf_filename)
     53         _set_data_location(elec_meters, building)
     54         _sanity_check_meters(elec_meters, meter_devices)
---> 55         _sanity_check_appliances(building_metadata)
     56         group._f_setattr('metadata', building_metadata)
     57

/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/convert_yaml_to_hdf5.pyc in _sanity_check_appliances(building_metadata)
    105     """
    106     appliances = building_metadata['appliances']
--> 107     appliance_types = get_appliance_types()
    108     building_instance = building_metadata['instance']
    109

/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/object_concatenation.pyc in get_appliance_types()
     12     recursively resolved.
     13     """
---> 14     appliance_types_from_disk = get_appliance_types_from_disk()
     15     appliance_types = _concatenate_all_appliance_types(appliance_types_from_disk)
     16     return appliance_types

/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/file_management.pyc in get_appliance_types_from_disk()
      8
      9 def get_appliance_types_from_disk():
---> 10     obj_filenames = _find_all_appliance_type_files()
     11     obj_cache = {}
     12     for filename in obj_filenames:

/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/file_management.pyc in _find_all_appliance_type_files()
     20 def _find_all_appliance_type_files():
     21     filenames = _find_all_files_with_suffix('.yaml',
---> 22 _get_appliance_types_directory())
     23     return filenames
     24

/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/file_management.pyc in _get_appliance_types_directory()
     25
     26 def _get_appliance_types_directory():
---> 27     return _path_to_directory('..', 'central_metadata', 'appliance_types')
     28
     29

/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/file_management.pyc in _path_to_directory(*args)
     43 def _path_to_directory(*args):
     44     path_to_directory = join(_get_module_directory(), *args)
---> 45     assert isdir(path_to_directory)
     46     return path_to_directory
     47

AssertionError:
> /usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/file_management.py(45)_path_to_directory()
     44     path_to_directory = join(_get_module_directory(), *args)
---> 45     assert isdir(path_to_directory)
     46     return path_to_directory

so it seems like nilm-metadata is breaking while asserting that the provided path is in fact a directory. Debugging a little further:

ipdb> path_to_directory
'/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/../central_metadata/appliance_types'

and it turns out that the central_metadata directory doesn't exist beside the nilm_metadata directory. I downloaded the central_metadata from github which seems to have solved the problem. Is this something that should have copied over automatically when I did 'sudo python setup.py install' for nilm-metadata?

Sorry for the long email. Feel free to call me if it's easier!
Oli

[JackKelly]

Ah, thanks loads for digging into this. This is a bug. I think the problem is that I did all my developing by installing NILM Metadata using python setup.py develop instead of python setup.py install. The difference is that develop doesn't copy the package to /usr/local/lib/python2.7/dist-packages/. I imagine there's fairly easy fix by getting setup.py to copy the *.yaml files over... but, because I'm supposed to be focussing on my IBM work right now, this should fix the issue (I'll update the README)

sudo python setup.py develop

[oliparson]

Great. I've just updated https://github.com/nilmtk/nilm_metadata/blob/master/README.md to include this line in the installation instructions rather than a duplicate of sudo python setup.py install

[JackKelly]

Ah, cool, I think we both did pretty much exactly the same thing at the same time! I think I've merged both our changes!

[paperbackraita]

Hey @JackKelly.
cc: @nipunreddevil

For making a working conda recipe for nilm_metadata, python setup.py install option was necessary. Tried fixing the missing central_metadata with the package_data option in setuptools. However, problems arose as setuptools takes a specific package structure as specified in this example under the topic "Setting up your Directory Structure".

For now, the central_metadata folder is at a position relative to the inner nilm_metadata folder. But this would create problems with multiple platforms and packages. What do you suggest?

[JackKelly]

Thanks loads for looking at this.

sorry, I won't have time to look into this today.

What would you suggest? Moving the central_metadata folder?

[paperbackraita]

Yes. The setuptools directory structure demands that all files required by the package have to reside in the inner folder named after the package taken into account. We could move central_metadata and have the setup.py file modified to make this work.
I'll fork nilm_metadata,try out the modifications and let you know if this works or not.

PS: The way in which nilm_metadata and central_metadata are being used have to be altered as well. Just basic path changes, but changes nevertheless.

[JackKelly]

cool. Yes, if you make the changes you feel necessary in your fork then
I'll take a look. Thanks for looking into this!

On 9 January 2015 at 09:46, Mridul Malpotra notifications@github.com
wrote:

Yes. The setuptools directory structure demands that all files required by
the package have to reside in the inner folder named after the package
taken into account. We could move central_metadata and have the setup.py
file modified to make this work.
I'll fork nilm_metadata,try out the modifications and let you know if
this works or not.

PS: The way in which nilm_metadata and central_metadata are being used
have to be altered as well. Just basic path changes, but changes
nevertheless.

—
Reply to this email directly or view it on GitHub
#6 (comment).

[nipunbatra]

conda-forge/staged-recipes#2667