Dogpile.cache incompatible with Windows? #255

Closed
Choruptian opened this Issue Nov 4, 2013 · 12 comments

Comments

Projects
None yet
4 participants

ImportError: No module named fcntl

This module is Unix specific and I've yet to find any viable solution to make this work with or without this module.

I don't know the intention/goal of the project but I've run subliminal before on windows without issue and I was hoping this could be resolved. If this problem persists as is I presume the entire library "dogpile" will have to be cast out or rewritten and thus halt the project some?

"""
subliminal.exe -l en -- Z:\Media\Series
Traceback (most recent call last):
File "C:\Python27\Scripts\subliminal-script.py", line 8, in
load_entry_point('subliminal==0.7.0', 'console_scripts', 'subliminal')()
File "build\bdist.win-amd64\egg\subliminal\cli.py", line 67, in subliminal
File "C:\Python27\lib\site-packages\dogpile.cache-0.5.1-py2.7.egg\dogpile\cache\region.py", line 230, in configure
backend_cls = _backend_loader.load(backend)
File "C:\Python27\lib\site-packages\dogpile.cache-0.5.1-py2.7.egg\dogpile\cache\util.py", line 34, in load
return self.implsname
File "C:\Python27\lib\site-packages\dogpile.cache-0.5.1-py2.7.egg\dogpile\cache\util.py", line 51, in load
mod = import(modulepath)
File "C:\Python27\lib\site-packages\dogpile.cache-0.5.1-py2.7.egg\dogpile\cache\backends\file.py", line 15, in
import fcntl
ImportError: No module named fcntl
"""

Diaoul was assigned Nov 4, 2013

Owner

Diaoul commented Nov 4, 2013

I opened an issue for dogpile.cache: https://bitbucket.org/zzzeek/dogpile.cache/issue/44/cross-platform-file-caching
dogpile.cache supports different cache backends so, if needed, I might end up implementing my own: http://dogpilecache.readthedocs.org/en/latest/usage.html#creating-backends

Diaoul referenced this issue Nov 4, 2013

Closed

Windows #257

oscy commented Nov 4, 2013

Very sorry for opening the same case.. I was reading through the closed cases.. and up on first gaze I didn't see this among the open ones with same problem... Glad it's active though! Keep us updated!

Owner

Diaoul commented Nov 5, 2013

So it looks like, zzzeek is looking for someone to do the job and make a pull req. I don't use Windows myself with subliminal so if one of you is willing to solve this please do so.
https://bitbucket.org/zzzeek/dogpile.cache/issue/44/cross-platform-file-caching

Also hoping for this to get solved soon!

oscy commented Nov 7, 2013

I would love to help but I don't have the skills for it unfortunately :/ Would love to have this solved though otherwise Subliminal is useless for everyone with windows

tarzasai referenced this issue in Flexget/Flexget Nov 11, 2013

Closed

download subtitles with periscope #108

oscy commented Nov 12, 2013

Dutchreaper123 posted the following at zzeeks thread... workable solution?

I'm running subliminal and made some changes to overcome the problems in windows. I do not know if it solves all the problems or if it even works, but it fixed my errors :D 1. install https://github.com/WoLpH/portalocker (windows replacement fcntl) 2. install pywin32 for required libraries. 3. change dogpile.cache-0.5.1-py2.7.egg\dogpile\cache\backends\file.py to:

"""

File Backends

Provides backends that deal with local filesystem access.

"""

from future import with_statement
from dogpile.cache.api import CacheBackend, NO_VALUE
from contextlib import contextmanager
from dogpile.cache import compat
from dogpile.cache import util
import os

import portalocker
"""import fcntl"""

all = 'DBMBackend', 'FileLock'

class DBMBackend(CacheBackend):
"""A file-backend using a dbm file to store keys.

Basic usage::

    from dogpile.cache import make_region

    region = make_region().configure(
        'dogpile.cache.dbm',
        expiration_time = 3600,
        arguments = {
            "filename":"/path/to/cachefile.dbm"
        }
    )

DBM access is provided using the Python ``anydbm`` module,
which selects a platform-specific dbm module to use.
This may be made to be more configurable in a future
release.

Note that different dbm modules have different behaviors.
Some dbm implementations handle their own locking, while
others don't.  The :class:`.DBMBackend` uses a read/write
lockfile by default, which is compatible even with those
DBM implementations for which this is unnecessary,
though the behavior can be disabled.

The DBM backend by default makes use of two lockfiles.
One is in order to protect the DBM file itself from
concurrent writes, the other is to coordinate
value creation (i.e. the dogpile lock).  By default,
these lockfiles use the ``flock()`` system call
for locking; this is only available on Unix
platforms.

Currently, the dogpile lock is against the entire
DBM file, not per key.   This means there can
only be one "creator" job running at a time
per dbm file.

A future improvement might be to have the dogpile lock
using a filename that's based on a modulus of the key.
Locking on a filename that uniquely corresponds to the
key is problematic, since it's not generally safe to
delete lockfiles as the application runs, implying an
unlimited number of key-based files would need to be
created and never deleted.

Parameters to the ``arguments`` dictionary are
below.

:param filename: path of the filename in which to
 create the DBM file.  Note that some dbm backends
 will change this name to have additional suffixes.
:param rw_lockfile: the name of the file to use for
 read/write locking.  If omitted, a default name
 is used by appending the suffix ".rw.lock" to the
 DBM filename.  If False, then no lock is used.
:param dogpile_lockfile: the name of the file to use
 for value creation, i.e. the dogpile lock.  If
 omitted, a default name is used by appending the
 suffix ".dogpile.lock" to the DBM filename. If
 False, then dogpile.cache uses the default dogpile
 lock, a plain thread-based mutex.


"""
def __init__(self, arguments):
    self.filename = os.path.abspath(
                        os.path.normpath(arguments['filename'])
                    )
    dir_, filename = os.path.split(self.filename)

    self._rw_lock = self._init_lock(
                            arguments.get('rw_lockfile'),
                            ".rw.lock", dir_, filename)
    self._dogpile_lock = self._init_lock(
                            arguments.get('dogpile_lockfile'),
                            ".dogpile.lock",
                            dir_, filename,
                            util.KeyReentrantMutex.factory)

    # TODO: make this configurable
    if compat.py3k:
        import dbm
    else:
        import anydbm as dbm
    self.dbmmodule = dbm
    self._init_dbm_file()

def _init_lock(self, argument, suffix, basedir, basefile, wrapper=None):
    if argument is None:
        lock = FileLock(os.path.join(basedir, basefile + suffix))
    elif argument is not False:
        lock = FileLock(
                    os.path.abspath(
                        os.path.normpath(argument)
                    ))
    else:
        return None
    if wrapper:
        lock = wrapper(lock)
    return lock

def _init_dbm_file(self):
    exists = os.access(self.filename, os.F_OK)
    if not exists:
        for ext in ('db', 'dat', 'pag', 'dir'):
            if os.access(self.filename + os.extsep + ext, os.F_OK):
                exists = True
                break
    if not exists:
        fh = self.dbmmodule.open(self.filename, 'c')
        fh.close()

def get_mutex(self, key):
    # using one dogpile for the whole file.   Other ways
    # to do this might be using a set of files keyed to a
    # hash/modulus of the key.   the issue is it's never
    # really safe to delete a lockfile as this can
    # break other processes trying to get at the file
    # at the same time - so handling unlimited keys
    # can't imply unlimited filenames
    if self._dogpile_lock:
        return self._dogpile_lock(key)
    else:
        return None

@contextmanager
def _use_rw_lock(self, write):
    if self._rw_lock is None:
        yield
    elif write:
        with self._rw_lock.write():
            yield
    else:
        with self._rw_lock.read():
            yield

@contextmanager
def _dbm_file(self, write):
    with self._use_rw_lock(write):
        dbm = self.dbmmodule.open(self.filename,
                            "w" if write else "r")
        yield dbm
        dbm.close()

def get(self, key):
    with self._dbm_file(False) as dbm:
        if hasattr(dbm, 'get'):
            value = dbm.get(key, NO_VALUE)
        else:
            # gdbm objects lack a .get method
            try:
                value = dbm[key]
            except KeyError:
                value = NO_VALUE
        if value is not NO_VALUE:
            value = compat.pickle.loads(value)
        return value

def get_multi(self, keys):
    return [self.get(key) for key in keys]

def set(self, key, value):
    with self._dbm_file(True) as dbm:
        dbm[key] = compat.pickle.dumps(value)

def set_multi(self, mapping):
    with self._dbm_file(True) as dbm:
        for key,value in mapping.items():
            dbm[key] = compat.pickle.dumps(value)

def delete(self, key):
    with self._dbm_file(True) as dbm:
        try:
            del dbm[key]
        except KeyError:
            pass

def delete_multi(self, keys):
    with self._dbm_file(True) as dbm:
        for key in keys:
            try:
                del dbm[key]
            except KeyError:
                pass

class FileLock(object):
"""Use lockfiles to coordinate read/write access to a file.

Only works on Unix systems, using
`fcntl.flock() <http://docs.python.org/library/fcntl.html>`_.

"""

def __init__(self, filename):
    self._filedescriptor = compat.threading.local()
    self.filename = filename

def acquire(self, wait=True):
    return self.acquire_write_lock(wait)

def release(self):
    self.release_write_lock()

@property
def is_open(self):
    return hasattr(self._filedescriptor, 'fileno')

@contextmanager
def read(self):
    self.acquire_read_lock(True)
    try:
        yield
    finally:
        self.release_read_lock()

@contextmanager
def write(self):
    self.acquire_write_lock(True)
    try:
        yield
    finally:
        self.release_write_lock()

def acquire_read_lock(self, wait):
    return self._acquire(wait, os.O_RDONLY, portalocker.LOCK_SH)

def acquire_write_lock(self, wait):
    return self._acquire(wait, os.O_WRONLY, portalocker.LOCK_EX)

def release_read_lock(self):
    self._release()

def release_write_lock(self):
    self._release()

def _acquire(self, wait, wrflag, lockflag):
    wrflag |= os.O_CREAT
    fileno = os.open(self.filename, wrflag)
    try:
        if not wait:
            lockflag |= portalocker.LOCK_NB
        portalocker.lock(fileno, lockflag)
    except IOError:
        os.close(fileno)
        if not wait:
            # this is typically
            # "[Errno 35] Resource temporarily unavailable",
            # because of LOCK_NB
            return False
        else:
            raise
    else:
        self._filedescriptor.fileno = fileno
        return True

def _release(self):
    try:
        fileno = self._filedescriptor.fileno
    except AttributeError:
        return
    else:
        portalocker.unlock(fileno)
        os.close(fileno)
        del self._filedescriptor.fileno

PS This is my first time installing/editing python, entering a possible bugfix. sorry for the inconvenience of not testing it and not providing a .patch.

Owner

Diaoul commented Nov 14, 2013

Do you think that locking on a process basis is enough? The cache file can get corrupted if if someone runs more than one instance of the CLI at the same time.
I think I'll use that and add an clear message explaining parallel execution of subliminal with the same cache file is prohibited in the --help section.

Owner

Diaoul commented Nov 15, 2013

The issue is resolved with the latest version of dogpile.cache, I will update.

oscy commented Nov 15, 2013

Your comment above about cache file getting corrupt will this still be a problem.. I'll be running it on a server which is always on with some .bat script to automaticly run it once a day (during night time).

Owner

Diaoul commented Nov 16, 2013

As I stated, this will not be an issue unless you run the command line in parrallel and with the same cache file

oscy commented Nov 16, 2013

My server won't be logged on.. I'll have a scheduled task running the script file.. to be honest I don't know if it closes down command line inbetween, use the same or just open up a parallel one.. I've just configured so it runs.. I'm the type of guy who knows very little myself but just follow guides...

Owner

Diaoul commented Nov 20, 2013

Fixed with 3736d92

Diaoul closed this Nov 20, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment