New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for storing/using metadata about hosts #4

Open
bitprophet opened this Issue Aug 19, 2011 · 8 comments

Comments

Projects
None yet
5 participants
@bitprophet
Member

bitprophet commented Aug 19, 2011

2018 update: this is an old, old ticket and it is primed to actually be implemented now that 2.0 is available to build features on top of.

The tl;dr is that users need ways to store rich data about their target hosts (and, usually, groups or "roles"), first, and then need a way of addressing that information when doing API or CLI level things, second.

Fabric 2 is built on Invoke which has a powerful configuration system, which is then exposed to tasks via a 'context' object. It seems likely that we will build this feature on those, something like (but not necessarily limited to):

  • Standardize on some relatively generic config-style format for representing hosts; basically Connection's parameters (user, hostname, port, connect_kwargs, timeout, etc etc) and probably with a bit more on top
    • or at least the opportunity for users to put arbitrary data on top and have that show up in the objects exposed to the user at runtime
  • Update Context so it has a lightweight link to some other class or classes which turn the 'raw' config into usable API objects, based on some query or lookup
    • Open question is whether these show up as actual Connections or if there's an intermediate representational class like Host

For example (again: just an off the cuff example!) perhaps we'll set it up so host data gets its own config-style file that lives alongside the regular config files - say, $PROJECT/hosts.(yml|json|py|etc):

web1:
  host: web1
  # Implicit local user, as with Connection
web2:
  host: web2
  user: admin2
  port: 2223
db:
  # Implicit dict-key-is-the-host-value, i.e. implicit "host: db"
  user: dbadmin

Then perhaps there's something like this (using pure Invoke style tasks for now, though certainly this would want the ability to use -H or decorators to select target hosts to 'wrap' the task, as in v1):

@task
def deploy_webs(c):
  # Assuming auto-creation of Connection objects...
  for cxn in [c.find_host('web1'), c.find_host('web2')]:
    cxn.run("hostname")

There are a whole lot of different ways we could slice and dice this, and a lot of directions that it could be extended in; the emphasis should be on giving users as much power and control as reasonably possible and then getting out of their way. Ideally all we'll do is standardize on some very basic way of shoveling data into Connection objects, and add support to the core CLI exec framework to arrive at something resembling Fabric 1 re: selecting execution targets.

All while exposing these mechanisms publicly so advanced users can take matters into their own hands - again I expect anybody beyond the most basic use cases to be highly likely to fall back on a "regular Invoke style tasks + making the needful API calls within those task bodies" approach.


Original description

Right now a "host" is solely limited to user/hostname/port. Would be nice, even just for user fabfiles, to store additional information such as operating system, per-host settings like env.shell, and so forth.

Note that this may (or may not) be a good time to reconsider changing the default value of shell to /bin/sh.


Originally submitted by Jeff Forcier (bitprophet) on 2009-07-20 at 05:02pm EDT

Relations

  • Duplicated by #43: /bin/bash is not always available – per host shell configuration?
  • Related to #97: In some situations, pressing Enter does not reuse the previous password
  • Related to #138: env.port not honored if host string lacks port specification
  • Related to #3: Make use of ssh_config where possible
  • Related to #76: Use decorator to define tasks

@ghost ghost assigned bitprophet Aug 19, 2011

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

Silas Sewell (silas) posted:


Regarding the default shell, even if you don't change to /bin/sh, it might be good to use /usr/bin/env bash to fix situations where the user installed bash.

Example patch: http://github.com/silas/fabric/commit/6f7d33c1a3180fbba21c89447bb9a32b84e839ba

P.S. I posted here because "Support #43" was marked as a duplicate.


on 2009-11-09 at 11:13pm EST

Member

bitprophet commented Aug 19, 2011

Silas Sewell (silas) posted:


Regarding the default shell, even if you don't change to /bin/sh, it might be good to use /usr/bin/env bash to fix situations where the user installed bash.

Example patch: http://github.com/silas/fabric/commit/6f7d33c1a3180fbba21c89447bb9a32b84e839ba

P.S. I posted here because "Support #43" was marked as a duplicate.


on 2009-11-09 at 11:13pm EST

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

Erich Heine (sophacles) posted:


I came up with a way of doing this host metadata that works pretty well for the simple cases. The relevant code bits are here http://paste.pocoo.org/show/173964/ however there are some ideas that I didn't have time to implement yet -- namely that this type of interface to host metadata might better be defined as a 'protocol'. By this I mean, have a defined interface on the one end, so that various backends may be implemented, e.g. an ldap client and a redis client, and so in. This would allow people to integrate with existing tools better (e.g. buildbot).


on 2010-02-04 at 03:33pm EST

Member

bitprophet commented Aug 19, 2011

Erich Heine (sophacles) posted:


I came up with a way of doing this host metadata that works pretty well for the simple cases. The relevant code bits are here http://paste.pocoo.org/show/173964/ however there are some ideas that I didn't have time to implement yet -- namely that this type of interface to host metadata might better be defined as a 'protocol'. By this I mean, have a defined interface on the one end, so that various backends may be implemented, e.g. an ldap client and a redis client, and so in. This would allow people to integrate with existing tools better (e.g. buildbot).


on 2010-02-04 at 03:33pm EST

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Feb 29, 2012

Member

This is strongly related to some stuff I was experimenting with in #563, where having a real Host object would help a lot with host_string BS, and also pave the way for this kind of per-host settings overrides.

That has been punted to 2.x so this is too.

Member

bitprophet commented Feb 29, 2012

This is strongly related to some stuff I was experimenting with in #563, where having a real Host object would help a lot with host_string BS, and also pave the way for this kind of per-host settings overrides.

That has been punted to 2.x so this is too.

@mrmachine

This comment has been minimized.

Show comment
Hide comment
@mrmachine

mrmachine May 10, 2014

We currently store additional metadata about the remote host in env.server, that we pull from an inventory of server configs saved on disk as JSON. But this only works when we are running tasks on a single server, or when every task that is expected to be run with multiple servers contains code to update env.server based on the value of env.host_string.

One thing that would make this easier for us is if fabric provided a hook for when env.host_string is changed. Then we could just have one piece of code that hooks into that, and updates env.server with the additional context whenever env.host_string is changed.

Could this be implemented easily, without having to go all out and refactor host_string into a full blown Host object?

mrmachine commented May 10, 2014

We currently store additional metadata about the remote host in env.server, that we pull from an inventory of server configs saved on disk as JSON. But this only works when we are running tasks on a single server, or when every task that is expected to be run with multiple servers contains code to update env.server based on the value of env.host_string.

One thing that would make this easier for us is if fabric provided a hook for when env.host_string is changed. Then we could just have one piece of code that hooks into that, and updates env.server with the additional context whenever env.host_string is changed.

Could this be implemented easily, without having to go all out and refactor host_string into a full blown Host object?

@grantjenks

This comment has been minimized.

Show comment
Hide comment
@grantjenks

grantjenks Jun 23, 2018

Copying relevant comments from #1748, @peteruhnak commented:

"""
Hi! Is it possible to specify hosts either in a config file or the fabfile itself?

I know I can provide them via CLI (-H), but if the fabfile is designed to communicate with a specific server, then it just forces the user to do extra stuff for no good reason.

The "best" solution I could figure out was to create the connection by hand, e.g.

@task
def my_ls(c):
conn = Connection('myhost')
conn.run('ls')
but that seems quite dirty as I need to (1) duplicate it everywhere and (2) it makes the context argument pointless.
"""

And I followed up with:

"""
Agreed with all above. I really miss the old env.hosts setting. It was beautifully simple.

-1 on needing another config file like invoke.yaml or whatnot. I would much prefer to monkey-patch or some kind of import hook that let me state the hosts.

It also took me a while to figure out that ctx.local is not present if no hosts are specified :(
"""

I understand Invoke provides rich config support. I would hope that included configuring from Python's fab files.

grantjenks commented Jun 23, 2018

Copying relevant comments from #1748, @peteruhnak commented:

"""
Hi! Is it possible to specify hosts either in a config file or the fabfile itself?

I know I can provide them via CLI (-H), but if the fabfile is designed to communicate with a specific server, then it just forces the user to do extra stuff for no good reason.

The "best" solution I could figure out was to create the connection by hand, e.g.

@task
def my_ls(c):
conn = Connection('myhost')
conn.run('ls')
but that seems quite dirty as I need to (1) duplicate it everywhere and (2) it makes the context argument pointless.
"""

And I followed up with:

"""
Agreed with all above. I really miss the old env.hosts setting. It was beautifully simple.

-1 on needing another config file like invoke.yaml or whatnot. I would much prefer to monkey-patch or some kind of import hook that let me state the hosts.

It also took me a while to figure out that ctx.local is not present if no hosts are specified :(
"""

I understand Invoke provides rich config support. I would hope that included configuring from Python's fab files.

@kaptajnen

This comment has been minimized.

Show comment
Hide comment
@kaptajnen

kaptajnen Jul 19, 2018

In fabric1 I added the ability to load hosts lists with --set list=somelistfile.txt. In the fabfile I would then parse the file and modify env.hosts and env.passwords with settings from this file. This allowed me to do e.g fab task --set list=datacenter1.txt. I then had dozens of general purpose tasks I could run against predefined lists, in some cases a list per datacenter or list per logical function. I realize this may be a bit of a hack but it has suited me well for many years in my professional career.

I haven't found a way to do this in fab2 since the hosts are loaded inside the executor. I would very much like to do something similar in fab2. However it seems the paradigm in fab2 requires me to specify the hosts in the tasks. This makes it impossible to have my general purpose tasks and use it on a wide variety of servers (without using fab2 -H).

Having some kind of advanced host configuration file and being able to specify it on the command line I think would be a great addition to fabric2.

kaptajnen commented Jul 19, 2018

In fabric1 I added the ability to load hosts lists with --set list=somelistfile.txt. In the fabfile I would then parse the file and modify env.hosts and env.passwords with settings from this file. This allowed me to do e.g fab task --set list=datacenter1.txt. I then had dozens of general purpose tasks I could run against predefined lists, in some cases a list per datacenter or list per logical function. I realize this may be a bit of a hack but it has suited me well for many years in my professional career.

I haven't found a way to do this in fab2 since the hosts are loaded inside the executor. I would very much like to do something similar in fab2. However it seems the paradigm in fab2 requires me to specify the hosts in the tasks. This makes it impossible to have my general purpose tasks and use it on a wide variety of servers (without using fab2 -H).

Having some kind of advanced host configuration file and being able to specify it on the command line I think would be a great addition to fabric2.

@benzkji

This comment has been minimized.

Show comment
Hide comment
@benzkji

benzkji Aug 5, 2018

as @grantjenks mentioned, there is advanced config support in invoke: http://docs.pyinvoke.org/en/1.1/concepts/configuration.html - so we would just need to build a yaml file, and there we go. this would/could be extended (by hand, everyone on its own for now), to support something like the env concept, as in fabric one. If one doesnt like yaml, this can be completely done in python code, as I understand.

benzkji commented Aug 5, 2018

as @grantjenks mentioned, there is advanced config support in invoke: http://docs.pyinvoke.org/en/1.1/concepts/configuration.html - so we would just need to build a yaml file, and there we go. this would/could be extended (by hand, everyone on its own for now), to support something like the env concept, as in fabric one. If one doesnt like yaml, this can be completely done in python code, as I understand.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 7, 2018

Member

@benzkji Absolutely true; users can, right now, put arbitrary data in their config files (py, yml, json, eventually others) and pull that data out inside their tasks to construct Connections and Groups and whatnot. This is intentional; we don't ever want there to be only one way to handle that sort of setup, which is why the objects are all public and such.

See the very top of the ticket, too; I edited the description with a modern outlook on what this feature might look like in v2, in terms of some basic common "for un-opinionated users" helpers. I'm actually tinkering with that in a private (client-of-fabric, not fork) repo these days to see what patterns emerge.

Member

bitprophet commented Aug 7, 2018

@benzkji Absolutely true; users can, right now, put arbitrary data in their config files (py, yml, json, eventually others) and pull that data out inside their tasks to construct Connections and Groups and whatnot. This is intentional; we don't ever want there to be only one way to handle that sort of setup, which is why the objects are all public and such.

See the very top of the ticket, too; I edited the description with a modern outlook on what this feature might look like in v2, in terms of some basic common "for un-opinionated users" helpers. I'm actually tinkering with that in a private (client-of-fabric, not fork) repo these days to see what patterns emerge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment