-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement secrets/credentials container #8353
Comments
Some notes from a planning discussion with @andrewvc
|
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings. Part of elastic#8353
PR #8566 The API is general purpose, allowing arbitrary Using a versioned While it is a bit of a pain to use
The implementation provided in the PR is a JavaKeyStore of type Interestingly the |
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings. Part of elastic#8353
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings. Part of #8353
…M support * Introduce a SecretStoreFactory to allow runtime definition of SecretStore implementation. * Introduce a SecureConfig to allow simple configuration of different SecretStore implementaiton. * Introduce random default password plus obfuscation. Best attempt at security through obscurity. * Corrections / better support for x-JVM modification. Part of elastic#8353 Note - not included here: * Any means to populate the SecureConfig based YAML settings * Any command line tooling to work with the SecretStore * Any support for configuration or settings in the context of Logstash
@jordansissel / @andrewvc / @yaauie re: What do you think about using Also, I would guess that environment substitution is the general workaround for the lack of a secret store, so the migration from Further, wrapping in brackets |
I'd rather have a one byte solution instead of `$!`. Maybe `^{secret.blah}`?
Definitely +1 on the brackets.
…On Wed, Nov 15, 2017 at 7:05 PM, Jake Landis ***@***.***> wrote:
@jordansissel <https://github.com/jordansissel> / @andrewvc
<https://github.com/andrewvc> / @yaauie <https://github.com/yaauie>
re: password => $secret.blah
What do you think about using $!{blah} to represent a secret ? *(as i am
sure you know)* This differs from the environment substitution by only
the ! . In concept this and the environment entries are doing the same
thing (substitution), and in ruby-likeness adding a ! changes the
behavior slightly.
Also, I would guess that environment substitution is the general
workaround for the lack of a secret store, so the migration from ${BLAH}
to $!{BLAH} seems intuitive. ( or migrate to $!{blah} since case doesn't
matter here). *LS won't start up with a helpful message if you configure
a secret that can't be replaced*
Further, wrapping in brackets {...} greatly simplifies any edge cases
w/r/t to parsing out the values.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#8353 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAIBY5cFwzQ3rtXuiZ1tVOiNkzvu-i5Hks5s24pegaJpZM4PfmxZ>
.
|
I was thinking we could use the same syntax. It could check both env and
secret store for the name. This is what puppet does and it seems to work
well
On Wed, Nov 15, 2017 at 6:37 PM Andrew Cholakian <notifications@github.com>
wrote:
… I'd rather have a one byte solution instead of `$!`. Maybe
`^{secret.blah}`?
Definitely +1 on the brackets.
On Wed, Nov 15, 2017 at 7:05 PM, Jake Landis ***@***.***>
wrote:
> @jordansissel <https://github.com/jordansissel> / @andrewvc
> <https://github.com/andrewvc> / @yaauie <https://github.com/yaauie>
>
> re: password => $secret.blah
>
> What do you think about using $!{blah} to represent a secret ? *(as i am
> sure you know)* This differs from the environment substitution by only
> the ! . In concept this and the environment entries are doing the same
> thing (substitution), and in ruby-likeness adding a ! changes the
> behavior slightly.
>
> Also, I would guess that environment substitution is the general
> workaround for the lack of a secret store, so the migration from ${BLAH}
> to $!{BLAH} seems intuitive. ( or migrate to $!{blah} since case doesn't
> matter here). *LS won't start up with a helpful message if you configure
> a secret that can't be replaced*
>
> Further, wrapping in brackets {...} greatly simplifies any edge cases
> w/r/t to parsing out the values.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#8353 (comment)
>,
> or mute the thread
> <
https://github.com/notifications/unsubscribe-auth/AAIBY5cFwzQ3rtXuiZ1tVOiNkzvu-i5Hks5s24pegaJpZM4PfmxZ
>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#8353 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAIC6l4WUmaGIG0fC9LIKmTfkP24Ys9dks5s25_6gaJpZM4PfmxZ>
.
|
Oh, I like that idea as well. WDYT @jakelandis ? |
Re-using ...so the evaluation precedence will be:
|
It sounds like we have agreement then. |
The CLI will strongly mirror Elasticsearch's. (It won't be a perfect copy, but pretty close) Since the keystore has optional logstash.yml settings, the CLI will be implemented using the Ruby based command line (Clamp). This is to allow easy access logstash.yml and have access to all the pre-existing defaults (such as data dir), but at a slight cost of startup time (it won't need to load the whole thing, just enough to get the settings from the correct place) . This means that a user can interact with the secret store via For example: However, a shell script will be created that will convert those (kinda awkward) command's into something that matches Elasticsearch's CLI. Just wanted to solicit any feedback on this approach but I get far down this road. |
This is a weird implementation detail. This would mean there are two ways for users to invoke the keystore cli, and I'd prefer only one. We have the plugin manager as a separate command line interface, and I think we can do the same for the secret manager. I am +1 on copying the CLI that Elasticsearch has. The implementation details are up for discussion. My intuition is that the |
Agreed.
The main difference is the requirement to access to logstash.yml. However, in retrospect, the ability to read the settings is already de-coupled from the 'bin/logstash' CLI, so should be able to re-use reading without the CLI. Let me take another swing at this. thanks! |
@jakelandis I enjoy these efforts to try and reuse components :) |
Why use substitution for the configuration file instead of what ES does, which merges settings from the keystore after loading the configuration yaml? It would be nice if all our applications behaved similarly. |
@tylersmalley - We have 2 sets of configuration's to protect. One configuration sets up our pipelines, and the other for the Logstash service. The former is a set of configuration files written in Logstash's config language, and defines the pipelines (for example, use beats input with es output). The latter is a single YAML file and defines attributes specific to Logstash itself (for example which queue type to use). For our pipeline config the keys used aren't guaranteed to be unique across the set of configurations. For example two outputs could have defined For our pipeline config, I do not believe it is possible to merge without substitution placeholders. For our single YAML configuration it is possible, but we also want symmetry within our own configuration first.
Agree'd , I will look into support a merge in addition to a placeholder for our YAML configuration. |
* Adds more CRUD like operations for SecretStore API * SecretStoreFactory Mirror API's CRUD operations * Adds 'exists' to API to allow command line warning 'Overwrite ?' * Minor readabiliy Note - there should not be an large logic changes here, only refactoring and minor readability concerns. This should be the final review for this section of code. Part of elastic#8353
This is in support of a future change to also support substitution from the secret store in addition to the environment variables. Part of elastic#8353
This is support of a adding additional command line entry points that also need to settings (logstash.yml) file. Since there is a distinct before/read/after life-cycle this utility provides some direction for additional command line entry points without duplicating code and logic. Part of elastic#8353
This is support of a adding additional command line entry points that also need to settings (logstash.yml) file. Since there is a distinct before/read/after life-cycle this utility provides some direction for additional command line entry points without duplicating code and logic. Part of elastic#8353
This is support of a adding additional command line entry points that also need to settings (logstash.yml) file. Since there is a distinct before/read/after life-cycle this utility provides some direction for additional command line entry points without duplicating code and logic. Part of elastic#8353
This is support of a adding additional command line entry points that also need to settings (logstash.yml) file. Since there is a distinct before/read/after life-cycle this utility provides some direction for additional command line entry points without duplicating code and logic. Part of #8353 Fixes #8700
This is support of a adding additional command line entry points that also need to settings (logstash.yml) file. Since there is a distinct before/read/after life-cycle this utility provides some direction for additional command line entry points without duplicating code and logic. Part of #8353 Fixes #8700
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings. Part of elastic#8353
…M support * Introduce a SecretStoreFactory to allow runtime definition of SecretStore implementation. * Introduce a SecureConfig to allow simple configuration of different SecretStore implementaiton. * Introduce random default password plus obfuscation. Best attempt at security through obscurity. * Corrections / better support for x-JVM modification. Part of elastic#8353 Note - not included here: * Any means to populate the SecureConfig based YAML settings * Any command line tooling to work with the SecretStore * Any support for configuration or settings in the context of Logstash
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings. Part of #8353
…M support * Introduce a SecretStoreFactory to allow runtime definition of SecretStore implementation. * Introduce a SecureConfig to allow simple configuration of different SecretStore implementaiton. * Introduce random default password plus obfuscation. Best attempt at security through obscurity. * Corrections / better support for x-JVM modification. Part of elastic#8353 Note - not included here: * Any means to populate the SecureConfig based YAML settings * Any command line tooling to work with the SecretStore * Any support for configuration or settings in the context of Logstash
…M support (#8659) * Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support * Introduce a SecretStoreFactory to allow runtime definition of SecretStore implementation. * Introduce a SecureConfig to allow simple configuration of different SecretStore implementaiton. * Introduce random default password plus obfuscation. Best attempt at security through obscurity. * Corrections / better support for x-JVM modification. Part of #8353 Note - not included here: * Any means to populate the SecureConfig based YAML settings * Any command line tooling to work with the SecretStore * Any support for configuration or settings in the context of Logstash * Secret Store Part 2: Review changes - new Random and remove System Property support
* Adds more CRUD like operations for SecretStore API * SecretStoreFactory Mirror API's CRUD operations * Adds 'exists' to API to allow command line warning 'Overwrite ?' * Minor readabiliy Note - there should not be an large logic changes here, only refactoring and minor readability concerns. This should be the final review for this section of code. Part of elastic#8353
* Adds more CRUD like operations for SecretStore API * SecretStoreFactory Mirror API's CRUD operations * Adds 'exists' to API to allow command line warning 'Overwrite ?' * Minor readabiliy Note - there should not be an large logic changes here, only refactoring and minor readability concerns. This should be the final review for this section of code. Part of elastic#8353
…tor (#8745) * Adds more CRUD like operations for SecretStore API * SecretStoreFactory Mirror API's CRUD operations * Adds 'exists' to API to allow command line warning 'Overwrite ?' * Minor readabiliy Note - there should not be an large logic changes here, only refactoring and minor readability concerns. This should be the final review for this section of code. Part of #8353
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings. Part of elastic#8353
…M support (elastic#8659) * Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support * Introduce a SecretStoreFactory to allow runtime definition of SecretStore implementation. * Introduce a SecureConfig to allow simple configuration of different SecretStore implementaiton. * Introduce random default password plus obfuscation. Best attempt at security through obscurity. * Corrections / better support for x-JVM modification. Part of elastic#8353 Note - not included here: * Any means to populate the SecureConfig based YAML settings * Any command line tooling to work with the SecretStore * Any support for configuration or settings in the context of Logstash * Secret Store Part 2: Review changes - new Random and remove System Property support
…tor (elastic#8745) * Adds more CRUD like operations for SecretStore API * SecretStoreFactory Mirror API's CRUD operations * Adds 'exists' to API to allow command line warning 'Overwrite ?' * Minor readabiliy Note - there should not be an large logic changes here, only refactoring and minor readability concerns. This should be the final review for this section of code. Part of elastic#8353
This change introduces the command line tooling and hooks needed to allow Logstash to use the secret store. This change hooks into the same logic that the does the environment variable substitution. The commnad line mirrors the Elasticsearch command line, and is implemented primarily in Java. Part of elastic#8353
This change introduces the command line tooling and hooks needed to allow Logstash to use the secret store. This change hooks into the same logic that the does the environment variable substitution. The commnad line mirrors the Elasticsearch command line, and is implemented primarily in Java. Part of elastic#8353
This change introduces the command line tooling and hooks needed to allow Logstash to use the secret store. This change hooks into the same logic that the does the environment variable substitution. The commnad line mirrors the Elasticsearch command line, and is implemented primarily in Java. Part of elastic#8353
This change introduces the command line tooling and hooks needed to allow Logstash to use the secret store. This change hooks into the same logic that the does the environment variable substitution. The commnad line mirrors the Elasticsearch command line, and is implemented primarily in Java. Part of elastic#8353
* Integrate secret store with Logstash core This change introduces the command line tooling and hooks needed to allow Logstash to use the secret store. This change hooks into the same logic that the does the environment variable substitution. The commnad line mirrors the Elasticsearch command line, and is implemented primarily in Java. Part of #8353 * Fix javadoc build and include cli in package * 8353-part4 - review changes (will fixup prior to merge) * 8353-part4 -review changes 2
This is in support of a future change to also support substitution from the secret store in addition to the environment variables. Part of elastic#8353 Fixes elastic#8699
This is support of a adding additional command line entry points that also need to settings (logstash.yml) file. Since there is a distinct before/read/after life-cycle this utility provides some direction for additional command line entry points without duplicating code and logic. Part of elastic#8353 Fixes elastic#8700
Parent ticket: #6892
This issue is focused on designing the following:
The above could be split into separate issues if desired.
Rough specification:
Safety goals:
The text was updated successfully, but these errors were encountered: