New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unify behavior of server/nb extensions enabling, user vs sys.prefix #1508
Comments
Do you propose that both follow:
|
Yup, I think that makes good sense as a goal. Where we have some trouble is that the traitlets config system doesn't perform any merging of the values of individual traits, it only merges at the configurable-object level (i.e. it merges user_config.Config with sys_config.Config, but doesn't merge the contents of user_config.Config.value with sys_config.Config.value). While this isn't desirable for extension loading, it is most of the time. In some ways, this points to the traitlets config system simply being a bad fit for extensions, as we rapidly expand the scope of what configuration we are talking about enabling. @SylvainCorlay discussed at the meeting a conf.d trait-type that would implement the directory-style loading without requiring changes to how traitlets.config works. Alternately, we could move the serverextensions piece to use the exact same code as nbextensions, which would ensure the behavior of the two is totally consistent, rather than using traitlets.config for one and frontend-config for the other. Right now, I'm leaning a bit toward this option. I think the nbextensions case is just a bug where the load order is backwards. That should be readily fixable. |
@minrk but if both used the frontend extensions stuff, wouldn't that be an On Wed, Jun 8, 2016 at 1:59 AM, Min RK notifications@github.com wrote:
Brian E. Granger |
@ellisonbg yes, though it could be shimmed (the CLI API doesn't need to change). Any fix involving changes to existing traitlets behavior would be a more significant backward-incompatible change. |
Perhaps having: def merge(self, target, deep=False) would allow this to work without breaking behavior. If the conf.d implementation (still working on a poc) did live in Then the distinction between types of extensions would fall away... though On 04:17, Thu, Jun 9, 2016 Min RK notifications@github.com wrote:
|
A couple of questions for that proposal:
We do have some of this functionality, just not in json config. Python config files do support modifications for container traits, which json config files do not. You can do: c.Class.dict_trait.update({'a': 5}) and it will add values, rather than overriding the whole trait. We could try to define a scheme for expressing such operations in JSON config, though I am wary of defining operations in such a format, rather than values. Re: nb_conda_kernels, I actually really don't want installing a package like that to be able to change the kernel manager. I really do think installing a package and modifying configuration should be separate steps in the vast majority of cases. What happens when I've installed two custom kernel managers, and they have registered themselves at install time, and assume that |
@minrk I think I have a very simple proposal for a conf.d mechanism. Imports: from traitlets import List, Dict, default
from traitlets.config import LoggingConfigurable
from jupyter_core.paths import jupyter_config_path
import itertools
from glob import glob
import json
import os A first simple version that does not take into account precedence: class NotebookApp0(LoggingConfigurable):
nbextensions = List(trait=Dict()).tag(config=True)
@default('nbextensions')
def _default_nbextensions(self):
files = itertools.chain.from_iterable(glob(d + '/nbextensions.d/*.json')
for d in jupyter_config_path())
return list(self.json_load(file) for file in files)
def json_load(self, name):
with open(name) as file:
return json.load(file) A version that takes into account directory precedence. class NotebookApp1(LoggingConfigurable):
nbextensions = List(trait=Dict()).tag(config=True)
@default('nbextensions')
def _default_nbextensions(self):
dicts = reversed([{
os.path.basename(v) : v for v in glob(d + '/nbextensions.d/*.json')
} for d in jupyter_config_path()])
return list({
k: self.json_load(v)
for d in dicts for k, v in d.items() }.values())
def json_load(self, name):
with open(name) as file:
return json.load(file) Now if I have a file n = NotebookApp1()
n.nbextensions Returns (the content of bqplot.json is) {
"src": "static",
"dest": "bqplot",
"require": "bqplot/extension"
} |
This is much simpler than trying to have the whole configuration be based on configuration directories. Besides, we can enforce a schema for nbextension declarations via the |
I am sure @minrk will find a clever way to make that dict comprehension even shorter. |
My question more pertains to the fact that the frontend and traditional
To address Nick's point - our answer to this point is that installation and On Thu, Jun 9, 2016 at 7:11 AM, Min RK notifications@github.com wrote:
Brian E. Granger |
Right, for developers, but we had discussed that package managers need to have the ability to do this sensibly. I guess my downstream needs are now really driven by what it takes to deploy a (or hundreds) of running notebook servers each with bespoke configuration, and every extra step needed is a step that can break.
Turns out there is a standard for doing just such a thing, as we have discussed over on |
What do you guys think of my implementation proposal above (which is simpler than what was proposed earlier IMO). The registration of an nbextension would correspond to adding the corresponding file. Enabling would correspond to set an |
I think @SylvainCorlay's solution is a good one if we want to go the route of supporting this principle for extensions specifically, rather than for configuration as a whole. If we want to do this more generally for configuration that is not this list-of-extensions structure in particular, then I'm guessing we probably shouldn't do both. @ellisonbg I think we can take any of these approaches in a way that's compatible with 4.2, by updating the new-in-4.2 APIs to load into this structure. I don't think anyone is proposing the removal of the currently defined places to look for config, but a strict addition. Since we already look for config in:
The simplest implementation of conf.d in my mind is to add
where all .py/.json (or we could limit it to .json) in JUPYTER_CONFIG_PATH/config_name.d/ are loaded in lexicographical order, as is the tradition of conf.d. That would be quite a small change to traitlets.config. I implemented a simple version of this in ipython/traitlets#242. The trickier bit is the merge behavior that's actually the topic of this issue. I fixed the nbconfig side in #1526, since that was just a bug in iteration direction. The traitlets side would require doing something like @bollwyvl suggested with |
Thanks @minrk, @sylvancorley, looks like good stuff. I am still in/am the camp of "let's configure it all," but appreciate I think the list merging behavior is a fine thing to take off the table, as @ellisonbg the ring 0 configurator/relauncher UI direction sounds good as On 04:43, Fri, Jun 10, 2016 Min RK notifications@github.com wrote:
|
Yeah, and any list-like configurable that wants merge can always switch from a list to a dict of bools, like we did with server extensions. |
list is in descending priority, so load should iterate back to front, to ensure user config wins the easy part of #1508
list is in descending priority, so load should iterate back to front, to ensure user config wins the easy part of #1508
list is in descending priority, so load should iterate back to front, to ensure user config wins the easy part of jupyter#1508
list is in descending priority, so load should iterate back to front, to ensure user config wins the easy part of jupyter#1508
list is in descending priority, so load should iterate back to front, to ensure user config wins the easy part of jupyter#1508
As part of this, perhaps we add a
to create a Additionally, more generally, I'd love a high-level
or take a JSON file, I suppose. This would remove the need for extension maintainers/packagers having to maintain things like Additionally, |
Can we bump this from 4.3 to a later release? I'm aware that there are various fixes it would be good to get out to users, and this sounds like a bit of a bigger task. |
@takluyver bumped |
@takluyver that's my understanding. |
@minrk can you summarize how all this will work as of 5.0? I want to make sure this stuff is sufficiently resolved that we don't have to change it again before moving to the conf.d approach. |
@ellisonbg the only change for 5.0 is that |
Closing here, since the titular issue is fixed: user wins, merge happens, nbextensions and serverextensions should behave the same now. |
Users are encountering exciting conditions with the extension enabling behavior.
We encountered that here where we have basically had to say
rm -rf ~/.jupyter
.The behavior of both the nbextensions and the serverextensions are surprising.
nbextensions:
user
andsys-prefix
merge, butsys-prefix
winsGood
Bad
potential fixes
serverextensions:
~/.jupyter
conquers allGood
Bad
potential fixes
The text was updated successfully, but these errors were encountered: