New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

recipy.open issues - function signature #128

Closed
robintw opened this Issue Sep 9, 2016 · 7 comments

Comments

Projects
None yet
2 participants
@robintw
Contributor

robintw commented Sep 9, 2016

Using recipy.open (with recipy.open('file-open.txt', 'w') as f:) gives a gives an ValueError: I/O operation on closed file, with Python 2:

$ python check-recipy-open.py
recipy run inserted, with ID 817359aa-69ef-4d33-a062-026f343231c4
/usr/local/lib/python2.7/dist-packages/recipy-0.2.3-py2.7.egg/recipyCommon/libraryversions.py:30: UserWarning: requesting version of a module that has not been imported (recipy.open)
  'imported ({})'.format(modulename))
Traceback (most recent call last):
  File "check-recipy-open.py", line 2, in <module>
    with recipy.open('file-open.txt', 'w') as f:
  File "/usr/local/lib/python2.7/dist-packages/recipy-0.2.3-py2.7.egg/recipy/utils.py", line 35, in open
    log_output(args[0], 'recipy.open')
  File "/usr/local/lib/python2.7/dist-packages/recipy-0.2.3-py2.7.egg/recipy/log.py", line 177, in log_output
    db.update(append("libraries", get_version(source), no_duplicates=True), eids=[RUN_ID])
  File "/usr/local/lib/python2.7/dist-packages/tinydb/database.py", line 377, in update
    cond, eids
  File "/usr/local/lib/python2.7/dist-packages/tinydb/database.py", line 230, in process_elements
    data = self._read()
  File "/usr/local/lib/python2.7/dist-packages/tinydb/database.py", line 277, in _read
    return self._storage.read()
  File "/usr/local/lib/python2.7/dist-packages/tinydb/database.py", line 31, in read
    raw_data = (self._storage.read() or {})[self._table_name]
  File "/usr/local/lib/python2.7/dist-packages/tinydb_serialization/__init__.py", line 139, in read
    data = self.storage.read()
  File "/usr/local/lib/python2.7/dist-packages/tinydb/storages.py", line 93, in read
    self._handle.seek(0, 2)
ValueError: I/O operation on closed file

Using recipy.open (as with recipy.open('file-open.txt', 'w') as f:) gives a KeyError: 'mode' error, with Python 3:

$ python check-recipy-open.py
recipy run inserted, with ID 02c4d038-6934-4057-9c85-41d717ab6052
Traceback (most recent call last):
  File "check-recipy-open.py", line 2, in <module>
    with recipy.open('file-open.txt', 'w') as f:
  File "C:\Users\mjj\Anaconda3\lib\site-packages\recipy-0.2.3-py3.5.egg\recipy\utils.py", line 20, in open
    mode = kwargs['mode']
KeyError: 'mode'

Editing check-recipy-open.py and changing:

with recipy.open('file-open.txt', 'w') as f:

to

with recipy.open('file-open.txt', mode='w') as f:

so that the mode parameter is named, then re-running, gives:

$ python check-recipy-open.py
recipy run inserted, with ID c306c23b-9f24-4781-83e3-a4c9b19ea309
C:\Users\mjj\Anaconda3\lib\site-packages\recipy-0.2.3-py3.5.egg\recipyCommon\libraryversions.py:30: UserWarning: requesting version of a module that has not been imported (recipy.open)
  'imported ({})'.format(modulename))
Traceback (most recent call last):
  File "check-recipy-open.py", line 2, in <module>
    with recipy.open('file-open.txt', mode='w') as f:
  File "C:\Users\mjj\Anaconda3\lib\site-packages\recipy-0.2.3-py3.5.egg\recipy\utils.py", line 35, in open
    log_output(args[0], 'recipy.open')
  File "C:\Users\mjj\Anaconda3\lib\site-packages\recipy-0.2.3-py3.5.egg\recipy\log.py", line 177, in log_output
    db.update(append("libraries", get_version(source), no_duplicates=True), eids=[RUN_ID])
  File "C:\Users\mjj\Anaconda3\lib\site-packages\tinydb\database.py", line 377,
in update
    cond, eids
  File "C:\Users\mjj\Anaconda3\lib\site-packages\tinydb\database.py", line 230,
in process_elements
    data = self._read()
  File "C:\Users\mjj\Anaconda3\lib\site-packages\tinydb\database.py", line 277,
in _read
    return self._storage.read()
  File "C:\Users\mjj\Anaconda3\lib\site-packages\tinydb\database.py", line 31, in read
    raw_data = (self._storage.read() or {})[self._table_name]
  File "C:\Users\mjj\Anaconda3\lib\site-packages\tinydb_serialization\__init__.py", line 139, in read
    data = self.storage.read()
  File "C:\Users\mjj\Anaconda3\lib\site-packages\tinydb\storages.py", line 93, in read
    self._handle.seek(0, 2)
ValueError: I/O operation on closed file.

The above implies that recipy.open does not seem to support the same signature as Python open. I would expect it would support the same signature.

via @mikej888

@robintw robintw added hard bug labels Sep 9, 2016

@robintw

This comment has been minimized.

Show comment
Hide comment
@robintw

robintw Sep 19, 2016

Contributor

This issue was investigated by @BevelledEdge at the PyConUK 2016 sprint day.

There seem to be two errors:

  1. The code checks for a mode kwarg but that isn't set if kwargs aren't used (ie it is a positional arg).

  2. If you set mode manually (ie. recipy.open('blah.txt', mode='w')) then you get an error: ValueError: I/O operation on closed file.

This error is actually caused by TinyDB doing something weird with an error/warning that may have been raised by get_version - if you replace the call to get_version at https://github.com/recipy/recipy/blob/master/recipy/log.py#L153 with 'blah' then it works fine.

Potential avenues for exploration:

  1. What errors/warning does get_version return if it can't find a version?

  2. get_version needs updating to work with module.submod.func style packages anyway

  3. How does TinyDB deal with the errors or warnings returned by get_version, and why on earth does this mean it closes the DB?!

Contributor

robintw commented Sep 19, 2016

This issue was investigated by @BevelledEdge at the PyConUK 2016 sprint day.

There seem to be two errors:

  1. The code checks for a mode kwarg but that isn't set if kwargs aren't used (ie it is a positional arg).

  2. If you set mode manually (ie. recipy.open('blah.txt', mode='w')) then you get an error: ValueError: I/O operation on closed file.

This error is actually caused by TinyDB doing something weird with an error/warning that may have been raised by get_version - if you replace the call to get_version at https://github.com/recipy/recipy/blob/master/recipy/log.py#L153 with 'blah' then it works fine.

Potential avenues for exploration:

  1. What errors/warning does get_version return if it can't find a version?

  2. get_version needs updating to work with module.submod.func style packages anyway

  3. How does TinyDB deal with the errors or warnings returned by get_version, and why on earth does this mean it closes the DB?!

@robintw

This comment has been minimized.

Show comment
Hide comment
@robintw

robintw Sep 19, 2016

Contributor

As an update to this, the exception catching in get_version is very broad, for example:

try:
            version = sys.modules[modulename].__version__
        except:
            pass

Could this be causing some of the problems we've been having with TinyDB?

Contributor

robintw commented Sep 19, 2016

As an update to this, the exception catching in get_version is very broad, for example:

try:
            version = sys.modules[modulename].__version__
        except:
            pass

Could this be causing some of the problems we've been having with TinyDB?

@chiffa

This comment has been minimized.

Show comment
Hide comment
@chiffa

chiffa Feb 6, 2017

Contributor

This is not a behavior proper to the recipy.open but to the recipy.log_input function.

import recipy

recipy.log_input('test_input.txt', 'recipy.open')

leads to the failure with the same code:

C:\Anaconda2\lib\site-packages\recipyCommon\libraryversions.py:30: UserWarning: requesting version of a module that has not been imported (recipy.open)
  'imported ({})'.format(modulename))
Traceback (most recent call last):
  File [.....], line 3, in <module>
    recipy.log_input('test_input.txt', 'recipy.open')
  File "C:\Anaconda2\lib\site-packages\recipy\log.py", line 123, in log_input
    db.update(append("libraries", get_version(source), no_duplicates=True), eids=[RUN_ID])
  File "C:\Anaconda2\lib\site-packages\tinydb\database.py", line 377, in update
    cond, eids
  File "C:\Anaconda2\lib\site-packages\tinydb\database.py", line 230, in process_elements
    data = self._read()
  File "C:\Anaconda2\lib\site-packages\tinydb\database.py", line 277, in _read
    return self._storage.read()
  File "C:\Anaconda2\lib\site-packages\tinydb\database.py", line 31, in read
    raw_data = (self._storage.read() or {})[self._table_name]
  File "C:\Anaconda2\lib\site-packages\tinydb_serialization\__init__.py", line 139, in read
    data = self.storage.read()
  File "C:\Anaconda2\lib\site-packages\tinydb\storages.py", line 97, in read
    self._handle.seek(0, os.SEEK_END)
ValueError: I/O operation on closed file
Contributor

chiffa commented Feb 6, 2017

This is not a behavior proper to the recipy.open but to the recipy.log_input function.

import recipy

recipy.log_input('test_input.txt', 'recipy.open')

leads to the failure with the same code:

C:\Anaconda2\lib\site-packages\recipyCommon\libraryversions.py:30: UserWarning: requesting version of a module that has not been imported (recipy.open)
  'imported ({})'.format(modulename))
Traceback (most recent call last):
  File [.....], line 3, in <module>
    recipy.log_input('test_input.txt', 'recipy.open')
  File "C:\Anaconda2\lib\site-packages\recipy\log.py", line 123, in log_input
    db.update(append("libraries", get_version(source), no_duplicates=True), eids=[RUN_ID])
  File "C:\Anaconda2\lib\site-packages\tinydb\database.py", line 377, in update
    cond, eids
  File "C:\Anaconda2\lib\site-packages\tinydb\database.py", line 230, in process_elements
    data = self._read()
  File "C:\Anaconda2\lib\site-packages\tinydb\database.py", line 277, in _read
    return self._storage.read()
  File "C:\Anaconda2\lib\site-packages\tinydb\database.py", line 31, in read
    raw_data = (self._storage.read() or {})[self._table_name]
  File "C:\Anaconda2\lib\site-packages\tinydb_serialization\__init__.py", line 139, in read
    data = self.storage.read()
  File "C:\Anaconda2\lib\site-packages\tinydb\storages.py", line 97, in read
    self._handle.seek(0, os.SEEK_END)
ValueError: I/O operation on closed file
@robintw

This comment has been minimized.

Show comment
Hide comment
@robintw

robintw Feb 6, 2017

Contributor

That's really interesting - thanks @chiffa.

I haven't had chance to look at this recently due to health issues, but it looks like the fix might be easier than I thought - I'll try and have a look soon.

Contributor

robintw commented Feb 6, 2017

That's really interesting - thanks @chiffa.

I haven't had chance to look at this recently due to health issues, but it looks like the fix might be easier than I thought - I'll try and have a look soon.

@chiffa

This comment has been minimized.

Show comment
Hide comment
@chiffa

chiffa Feb 6, 2017

Contributor

Modifying version recovery to this removes the error altogether:

def get_version(modulename):
    "Return a string containing the module name and the library version."
    version = '?'
    if modulename in sys.modules:
        try:
            version = sys.modules[modulename].__version__
        except:
            pass

        try:
            version = sys.modules[modulename].version
        except:
            pass

        try:
            version = sys.modules[modulename].version.version
        except:
            pass

        try:
            version = sys.modules[modulename].VERSION
        except:
            pass
    else:
        print 'version unknown'
        # warnings.warn('requesting version of a module that has not been '
        #               'imported ({})'.format(modulename))

    return '{} v{}'.format(modulename, version)

I guess the problem is really with tinyDB treating warning calls in a weird fashion.

Contributor

chiffa commented Feb 6, 2017

Modifying version recovery to this removes the error altogether:

def get_version(modulename):
    "Return a string containing the module name and the library version."
    version = '?'
    if modulename in sys.modules:
        try:
            version = sys.modules[modulename].__version__
        except:
            pass

        try:
            version = sys.modules[modulename].version
        except:
            pass

        try:
            version = sys.modules[modulename].version.version
        except:
            pass

        try:
            version = sys.modules[modulename].VERSION
        except:
            pass
    else:
        print 'version unknown'
        # warnings.warn('requesting version of a module that has not been '
        #               'imported ({})'.format(modulename))

    return '{} v{}'.format(modulename, version)

I guess the problem is really with tinyDB treating warning calls in a weird fashion.

@chiffa

This comment has been minimized.

Show comment
Hide comment
@chiffa

chiffa Feb 6, 2017

Contributor

But not factoring removing the segment raising the error out from the database update call. This still casuses an error:

db = open_or_create_db()
    db.update(append("inputs", record, no_duplicates=True), eids=[RUN_ID])
    version = get_version(source)
    db.update(append("libraries", version, no_duplicates=True), eids=[RUN_ID])
    db.close()

However moving warnings out of scope where DB is open resores normal working:

#Update object in DB
    version = get_version(source)
    db = open_or_create_db()
    db.update(append("inputs", record, no_duplicates=True), eids=[RUN_ID])
    db.update(append("libraries", version, no_duplicates=True), eids=[RUN_ID])
    db.close()

I will patch and push back to main.

Contributor

chiffa commented Feb 6, 2017

But not factoring removing the segment raising the error out from the database update call. This still casuses an error:

db = open_or_create_db()
    db.update(append("inputs", record, no_duplicates=True), eids=[RUN_ID])
    version = get_version(source)
    db.update(append("libraries", version, no_duplicates=True), eids=[RUN_ID])
    db.close()

However moving warnings out of scope where DB is open resores normal working:

#Update object in DB
    version = get_version(source)
    db = open_or_create_db()
    db.update(append("inputs", record, no_duplicates=True), eids=[RUN_ID])
    db.update(append("libraries", version, no_duplicates=True), eids=[RUN_ID])
    db.close()

I will patch and push back to main.

chiffa added a commit to ank-forked/recipy that referenced this issue Feb 6, 2017

Moved version detection. Closes recipy/recipy recipy#128
The bug was due to interference between the warnings and tinyDB inner workings - if TinyDB had a warning while the session was open, the TinyDB source file was closed. This is workaround a more general problem with TinyDB interaction with warnings.
@chiffa

This comment has been minimized.

Show comment
Hide comment
@chiffa

chiffa Feb 6, 2017

Contributor

#172 should solve the issue now.

Contributor

chiffa commented Feb 6, 2017

#172 should solve the issue now.

robintw added a commit that referenced this issue Feb 7, 2017

Merge pull request #173 from chiffa/tinyDB_correction
Moved version detection. Closes recipy/recipy #128

@robintw robintw closed this Oct 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment