New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resolver: check if update is needed #2249

Open
mpranj opened this Issue Sep 24, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@mpranj
Contributor

mpranj commented Sep 24, 2018

The resolver checks the file modification time to determine whether the storage plugin needs to read the file again. Other resolvers do it similarly (using timestamps, git commit-id, ... ). This only works within one KDB instance, once the resolver(s) have populated their data (and timestamps).

For the global cache it is necessary to store the change indicators (e.g. timestamps) within the cache. We then need a way to let the resolvers check whether the timestamp of the cached data still corresponds to the timestamp of the original config file. In other words, we need a check whether the cached data is up-to-date.

In a private email @markus2330 suggested that it might be necessary to change the plugin interface and export an additional function for this purpose or to change the kdbGet() logic. He also noted, that @tom-wa might have some ideas because of his work on the multifile resolver.
According to markus, the multifile resolver solves a similar problem, but I fail to see the similarity.

The issue is open to find and document a solution, so all suggestions are welcome.

@mpranj

This comment has been minimized.

Show comment
Hide comment
@mpranj

mpranj Sep 24, 2018

Contributor

My initial idea for this was not to export and additional plugin function. It should also be possible to inject such a timestamp into the resolver by adding metadata to the parentKey. That still implies that the resolvers need to implement the desired functionality.

Contributor

mpranj commented Sep 24, 2018

My initial idea for this was not to export and additional plugin function. It should also be possible to inject such a timestamp into the resolver by adding metadata to the parentKey. That still implies that the resolvers need to implement the desired functionality.

@markus2330

This comment has been minimized.

Show comment
Hide comment
@markus2330

markus2330 Sep 30, 2018

Contributor

According to markus, the multifile resolver solves a similar problem, but I fail to see the similarity.

What I mean here is that multifile uses functions (in particular the function "filename") and enums of resolver. It only uses it to resolve filenames but not to get the timestamp. Something similar could be done to get the timestamps.

Alternatively, we could change the resolvers so that they check for the mmap file during their regular execution. The string of the parentKey could contain the timestamp to compare with for that purpose. And resolvers which are already up-to-date with the cache could return 0. (They already do this now but never during the first run of kdbGet()).

Contributor

markus2330 commented Sep 30, 2018

According to markus, the multifile resolver solves a similar problem, but I fail to see the similarity.

What I mean here is that multifile uses functions (in particular the function "filename") and enums of resolver. It only uses it to resolve filenames but not to get the timestamp. Something similar could be done to get the timestamps.

Alternatively, we could change the resolvers so that they check for the mmap file during their regular execution. The string of the parentKey could contain the timestamp to compare with for that purpose. And resolvers which are already up-to-date with the cache could return 0. (They already do this now but never during the first run of kdbGet()).

@mpranj

This comment has been minimized.

Show comment
Hide comment
@mpranj

mpranj Sep 30, 2018

Contributor

Now I see what you meant. I only saw the alternative approach, but both seem fine for me. If there are no restrictions I will just see what fits better.

Contributor

mpranj commented Sep 30, 2018

Now I see what you meant. I only saw the alternative approach, but both seem fine for me. If there are no restrictions I will just see what fits better.

@mpranj mpranj referenced this issue Oct 15, 2018

Open

Global Cache #2270

0 of 6 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment