Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache output of time-consuming routines #159

Closed
pascalchevrel opened this issue Dec 18, 2013 · 4 comments
Closed

Cache output of time-consuming routines #159

pascalchevrel opened this issue Dec 18, 2013 · 4 comments
Assignees

Comments

@pascalchevrel
Copy link
Member

We should cache requests, especially for the json apis that may be called by bots or by our other projects importing strings from Transvision, so as to speed up requests and not put too much load on the server. We can empty the static cache every time we run glossaire.sh.

@flodolo
Copy link
Collaborator

flodolo commented Dec 30, 2013

Does this approach make any sense?
https://github.com/flodolo/transvision/commits/cached_views

It's only for HTML views with template, but I'm wondering if I'm overlooking something (e.g. using l10n-init.php for all views).

Note that you'll need php5-sqlite for testing it, and web/cache/views must be writable

@pascalchevrel
Copy link
Member Author

I tried it and it works, it's a bit complicated for my taste and it adds a dependency on sqlite, but that's generally ok. there is one method which is Chinese to me: interpolateQuery() :)
In the past, l10n-init was not used on all views because it created problems with the json service as it was forcing a fallback to the locale of the server doing the request, I am not sure it is still the case.

My main worry would be that this is a all or nothing approach, there is no way to deactivate caching, that means that it will make development harder as we will see cached content constantly. Also, the json sources are not cached while those are more likely to cause server load as they are automated by tools.

I am not sure why you are storing in the db the name of the view and the locale. It seems to me that the DB should only contain the hashed request as a unique ID so as to load the right cached file and maybe fields about the creation time and duration of the cache per file which would be the added benefit of using a database instead of using a simpler file caching system or am I missing something?

@flodolo
Copy link
Collaborator

flodolo commented Dec 31, 2013

interpolateQuery: is basically a var_dump for prepared SQL statements, not currently used. It could be removed after development is done.

Deactivating cache: right, it shouldn't be hard to implement.

Storing name of the view and locale: that's necessary because locale is often determined at run-time, so I have no other ways to understand if a query is for "it" or for "fr", for example. Same thing for view's name: for example, based only on the query parameters, I can't know if I'm viewing "accesskey" or "repo status".

But on second thoughts I think I can just use 2 columns instead of 4.

@flodolo
Copy link
Collaborator

flodolo commented Dec 31, 2013

Opened pull request #162. Compared to the first version:

  • table has only 2 columns
  • possibility to serve non cached content with ?nocache
  • manage also json/jsonp views

@pascalchevrel pascalchevrel changed the title Add a static cache for searches Cache output of time-consuming routines Mar 21, 2014
pascalchevrel added a commit that referenced this issue Mar 21, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - isActivated()
 - flush()
 - isValid()
 - deleteKey()
 - getKeyPath()
 - getCachePath()
 - isObsolete()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

rename a class
pascalchevrel added a commit that referenced this issue Mar 21, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - isActivated()
 - flush()
 - isValid()
 - deleteKey()
 - getKeyPath()
 - getCachePath()
 - isObsolete()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}
pascalchevrel added a commit that referenced this issue Mar 21, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - isActivated()
 - flush()
 - isValid()
 - deleteKey()
 - getKeyPath()
 - getCachePath()
 - isObsolete()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 21, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - isActivated()
 - flush()
 - isValid()
 - deleteKey()
 - getKeyPath()
 - getCachePath()
 - isObsolete()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods

remove a debug line
pascalchevrel added a commit that referenced this issue Mar 21, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - isActivated()
 - flush()
 - isValid()
 - deleteKey()
 - getKeyPath()
 - getCachePath()
 - isObsolete()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 21, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - isActivated()
 - flush()
 - isValid()
 - deleteKey()
 - getKeyPath()
 - getCachePath()
 - isObsolete()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 22, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - isActivated()
 - flush()
 - isValid()
 - deleteKey()
 - getKeyPath()
 - getCachePath()
 - isObsolete()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 22, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - isActivated()
 - flush()
 - isValid()
 - deleteKey()
 - getKeyPath()
 - getCachePath()
 - isObsolete()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 22, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - isActivated()
 - flush()
 - isValid()
 - deleteKey()
 - getKeyPath()
 - getCachePath()
 - isObsolete()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 22, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - isActivated()
 - flush()
 - isValid()
 - deleteKey()
 - getKeyPath()
 - getCachePath()
 - isObsolete()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 22, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - deleteKey()
 - isValidKey()
 - isObsoleteKey()
 - getKeyPath()
 - getCachePath()
 - isActivated()
 - flush()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods

nits
pascalchevrel added a commit that referenced this issue Mar 22, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - deleteKey()
 - isValidKey()
 - isObsoleteKey()
 - getKeyPath()
 - getCachePath()
 - isActivated()
 - flush()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 22, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - deleteKey()
 - isValidKey()
 - isObsoleteKey()
 - getKeyPath()
 - getCachePath()
 - isActivated()
 - flush()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 22, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - deleteKey()
 - isValidKey()
 - isObsoleteKey()
 - getKeyPath()
 - getCachePath()
 - isActivated()
 - flush()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 22, 2014
* New Cache class with these methods:
 - setKey()
 - getKey()
 - deleteKey()
 - isValidKey()
 - isObsoleteKey()
 - getKeyPath()
 - getCachePath()
 - isActivated()
 - flush()

* New constants:
 - CACHE constant activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - can't be older than generated data (uses a timestamp file in glossaire.sh)
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 27, 2014
* New Cache class with these methods:
Public:
 - setKey()
 - getKey()
 - flush()

Private:
 - deleteKey()
 - isValidKey()
 - isObsoleteKey()
 - getKeyPath()
 - getCachePath()
 - isActivated()

* New constants:
 - CACHE_ENABLED activates/deactivates the cache
 - CACHE_TIME is the default duration of the cache
 - CACHE_PATH is the path to the cache folder

There are also fallback constants in the class to use it outside of Transvision as a drop-in class.

* GET['nocache'] to force a page not to use caching

* Cache duration
 - age of the last generation of data (uses a timestamp file in glossaire.sh), defaults to 5h20 if the timestamp file is missing
 - can be defined in seconds when using getKey(), ex getKey($id, 300)

* Onestring model makes a use of the class, ex:

if (! $data = Cache::getKey($cache_id)) {
    /* create $data */
    /* Now cache data */
    Cache::setKey($cache_id, $data);
}

* Tests for all methods
pascalchevrel added a commit that referenced this issue Mar 27, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants