Add "--value-file" option to "sops set [...]"

[bjornfor]

This allows running "sops set [...]" without leaking secrets in process
listings.

To read secrets from stdin, use "/dev/stdin" as the file path.

Fixes #729

I'm not a go programmer, please review carefully :-)

[bjornfor]

About the CI failure; make test succeeds on my machine.

I amended the commit to include sign-off. When is CI going to be run again? (It's been 8+ hours.)

[felixfontein]

About the CI failure; make test succeeds on my machine.

That runs the unit tests; these also pass in CI. What fails are the functional tests (make functional-tests).

When is CI going to be run again? (It's been 8+ hours.)

When someone with appropriate rights presses the buttons. Please remember that we are volunteers who do this in their free time.

[bjornfor]

About the CI failure; make test succeeds on my machine.

That runs the unit tests; these also pass in CI. What fails are the functional tests (make functional-tests).

Ok, thanks. I can reproduce the make functional-tests errors locally, so I can start looking at them now.

When is CI going to be run again? (It's been 8+ hours.)

When someone with appropriate rights presses the buttons. Please remember that we are volunteers who do this in their free time.

Oh, sorry, I thought it was automatic, and given the short time spent on "make test" locally I was surprised about 8+ hour latency on github 🙈

[bjornfor]

Functional tests are passing locally now (except for publish_json_file_vault and publish_json_file_vault_version_1 which fail in main branch for me).

I'll try to create a functional test for the new "--value-file" option tomorrow.

[bjornfor]

Hm. The first full make functional-tests fails on my new test. Then, running (cd functional-tests && cargo test set_json_file_insert) allows my new test, and the test I based it upon, to succeed. And after that make functional-tests works again.

Where's the state that makes the test non-deterministic? How to get rid of it?

[bjornfor]

Where's the state that makes the test non-deterministic? How to get rid of it?

It seems test order is random, and some orders work, some does not:

• (cd functional-tests && cargo test set_json_file_insert) seems to always work.
• (cd functional-tests && cargo test) sometimes fails.

I based the new test, set_json_file_insert_with_value_file, on the existing set_json_file_insert and I'm surprised to see this difference.

Actually, it might be more nuanced than test order: I ran for i in $(seq 1000); do echo "i=$i"; (cd functional-tests && cargo test set_json_file_insert); echo $? >> results.txt; done and got 8/1000 failures.

[bjornfor]

Here's the output of a failing/racy run:

i=930                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.02s                                                                                                                                                                                                                                                                                                                                                                                                                      
     Running unittests src/lib.rs (target/debug/deps/functional_tests-f17a3fe490e958b4)                                                                                                                                                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
running 2 tests                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
test tests::set_json_file_insert ... FAILED                                                                                                                                                                                                                                                                                                                                                                                                                                                   
test tests::set_json_file_insert_with_value_file ... FAILED                                                                                                                                                                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
failures:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
---- tests::set_json_file_insert stdout ----                                                                                                                                                                                                                                                                                                                                                                                                                                                  
stdout: , stderr: Error unmarshalling input json: invalid character '}' after top-level value                                                                                                                                                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
thread 'tests::set_json_file_insert' panicked at src/lib.rs:476:9:                                                                                                                                                                                                                                                                                                                                                                                                                            
sops didn't exit successfully                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace                                                                                                                                                                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
---- tests::set_json_file_insert_with_value_file stdout ----                                                                                                                                                                                                                                                                                                                                                                                                                                  
stdout: , stderr: Error unmarshalling input json: invalid character '}' after top-level value                                                                                                                                                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
thread 'tests::set_json_file_insert_with_value_file' panicked at src/lib.rs:522:9:                                                                                                                                                                                                                                                                                                                                                                                                            
sops didn't exit successfully                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
failures:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
    tests::set_json_file_insert                                                                                                                                                                                                                                                                                                                                                                                                                                                               
    tests::set_json_file_insert_with_value_file                                                                                                                                                                                                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 36 filtered out; finished in 0.03s                                                                                                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
error: test failed, to rerun pass `--lib`                                                                                                                                                                                                                                                                                                                                                                                                                                                     

[felixfontein]

Using different filenames from the test you modified can help.

[bjornfor]

Using different filenames from the test you modified can help.

Aha, cargo test runs tests in parallel with threads! Thanks! It all makes sense now.

The latest push fixes the tests, by using a unique file name.

[felixfontein]

How about adding a (mutually exclusive) further option that reads from stdin instead of a file? That would provide a platform independent way of reading from stdin (/dev/stdin doesn't work on Windows).

[bjornfor]

Sure, but that would take some more time for me to implement and test (noob here). Also, isn't that something that can be added on top of this PR and is not in conflict/wrong direction with this change (iow, not a blocker)?

Another option could be to make the "value" argument optional, and if missing, read from stdin. (On Linux it's trivial to pipe file contents to stdin, I don't know about Windows.)

Thoughts?