Improving the hygiene of 1Password CLI 2 integration account specification

[halostatue]

The recent release of 1Password CLI 2.0.0 changes a number of things, but the two biggest changes are:

1. op signin requires the --account param. This has been fixed by Fix op signin for v2 #1961 and chore: 1Password fixes #1966, although not yet released.
2. When using op signin without biometric authentication, the environment variable used is no longer OP_SESSION_<accountShorthand>, but OP_SESSION_<accountUUID>. That means that if I have an entry like {{ opDocument foo '' myaccount }}, the session cannot be inherited from the environment because chezmoi will try OP_SESSION_myaccount, but the session token will actually be stored in OP_SESSION_myaccountUUID. If 1password.prompt is false, then the UUID is the only value that can be safely used in op* template functions.

A solution would be to cache the account maps at the beginning of a session (on the first call to any of the op* template functions). This would be retrieved with op account list --format=json, which returns responses like this:

[
  { "url": "account1.1password.com", "email": "user@account1.example.com", "user_uuid": "account1UUID" },
  { "url": "account2.1password.ca", "email": "user@account2.example.com", "user_uuid": "account2UUID" }
]

I believe that this should be transformed into a lookup table with the following rules:

• the full URL is mapped to the UUID;
• the first word of the URL (account1.1password.com becomes account1) is mapped to the UUID;
• the email address is mapped to the UUID;
• the UUID is mapped to the UUID; and
• entries that collide for any reason are removed (see below).

The example given above would be transformed into the following (as JSON):

{
  "account1": "account1UUID",
  "account1.1password.com": "account1UUID",
  "user@account1.example.com": "account1UUID",
  "account1UUID": "account1UUID",
  "account2": "account2UUID",
  "account2.1password.ca": "account2UUID",
  "user@account2.example.com": "account2UUID",
  "account2UUID": "account2UUID"
}

I believe that any of these values could be used to specify the --account parameter, but it would probably require testing to be sure.

Collisions should be removed because they will result in ambiguous results. I believe that the order of op accounts list --format=json is stable, but is the reverse order of sign-in (that is, the first account is last in the list). Known collision cases would be:

1. myaccount.1password.com and myaccount.1password.ca or myaccount.1password.eu exist and the person has accounts on both (meaning that the short version of the URL would collide), OR
2. user@example.com exists in both account1 and account2.

In theory, I guess it’s possible for the full URL to collide as well, but in that case, the email address would definitely not collide.

This doesn’t really affect me at the moment as I’m using biometric authentication, which is all kinds of awesome for chezmoi, but I think that it will be a problem for a fairly large class of users of 1Password CLI 2.0. I’m not sure how quickly this could be implemented, but I think that some sort of solution (as I’m not getting any feedback from Agilebits themselves) is necessary.

[twpayne]

Thanks very much for opening this discussion. Personally, I'm only using 1Password with a single (work) account so my knowledge of the details is much less than yours.

This all sounds good. Note that on my machine running op account list --format=json also includes a shorthand field that can also be mapped to an account UUID.

[halostatue]

What is the output of your op --version? Mine is 2.0.0.

[twpayne]

I also have op --version equal to 2.0.0. I'm on macOS.

[halostatue]

Well, ain’t that just peachy. So far, the 1Password folks seem to be ignoring my thread on 1password.community, but it appears that there is a difference in output for op account list --format=json depending on whether biometric authentication is enabled. IMO, this is a bug in op.

I am using biometric authentication (14" MBP with M1 Max), so the keys for the account list are url, email, and user_uuid. You are not, so the keys for the account list are url, email, user_uuid, and shorthand.

The logic that I described is still sound with one tweak:

• url is mapped to user_uuid;
• the first word before the dots of url (account1.1password.com becomes account1) is mapped to user_uuid;
• email is mapped to user_uuid;
• uuid is mapped to user_uuid;
• entries that collide are removed as described; and
• shorthand is mapped to user_uuid, if present.

Shorthand overrides the collision detection mode because it is the well-known shorthand that should be used. We could also potentially add the first word before the @ of email is mapped to user_uuid.

What do you think?

[halostatue]

@twpayne I’m going to start on a draft PR that can look this stuff up. The plan is to do the following:

• url -> user_uuid
• firstword(url) -> user_uuid
• email -> user_uuid
• firstword(email) -> user_uuid
• firstword(email)@url -> user_uuid
• user_uuid -> user_uuid
• remove collisions
• shorthand -> user_uuid (if present)

My first version of this is likely to require serious review, and I’m not sure what the best way to test this would be, so I’ll be submitting it as a draft. Additionally, my Golang is fairly…I don’t use the language often. But I think we should actually implement this and move forward.

[antonalekseev]

I'm not sure it belongs here, but have noticed another caveat. My op --version is 2.0.2, but that doesn't mean that op item create ... will necessarily produce an item of "version": 2, in my case it's still "version": 1. Therefore chezmoi execute-template "{{ onepasswordItemFields \"$UUID\" | toJson }}" | jq . returns {}. Probably it happens due to the fact that I still have 1Password 7 macOS GUI app which is incompatible with "version": 2.

[halostatue]

@antonalekseev, what @SimonBarendse has said here suggests that there’s either a bug in op’s output or in chezmoi’s handling of the existing output when connected to 1P7. I have fully switched to 1P8, so that’s not something that I’m going to be easily able to debug.

[twpayne]

@antonalekseev Please can you post the output with chezmoi's --debug flag. This will show exactly how chezmoi is invoking op. You can then run those commands without chezmoi to see how op is behaving. Something like:

$ chezmoi execute-template --debug "{{ onepasswordItemFields \"$UUID\" | toJson }}" | jq

[antonalekseev]

@SimonBarendse Ah, I see. Sorry for drawing misleading conclusions RE version attribute. Of course, documentation for the JSON output would be greatly appreciated.

@twpayne Sure. Here is an item created for testing purposes

$ op item get --format json testitem | jq -rM '.id + " " + .category + " " + .fields[0].value'
oyzzjv6kpnf2xf3oincqkk72ci LOGIN testuser

$ chezmoi execute-template "{{ onepasswordItemFields \"oyzzjv6kpnf2xf3oincqkk72ci\" | toJson }}" | jq .
{}

Retrying the above with --debug flag yields tons of output, but the most interesting part in my opinion is

2022-04-27T08:27:08+03:00 INF Output args=["op","--version"] duration=25.302816ms output="2.0.2\n" path=/usr/local/bin/op size=6
2022-04-27T08:27:08+03:00 INF IdempotentCmdOutput args=["op","--version"] component=system output="2.0.2\n" path=/usr/local/bin/op
2022-04-27T08:27:09+03:00 INF Output args=["op","--session","hKujmOm6xL8Cl8NxxoguM2S474ZzEVmvkfGyh1d4BKo","item","get","--format","json","oyzzjv6kpnf2xf3oincqkk72ci"] duration=1.233045936s output="{\n  \"id\": \"oyzzjv6kpnf2xf3oincqkk72ci\",\n  \"title\": \"testitem\",\n ..." path=/usr/local/bin/op size=941
2022-04-27T08:27:09+03:00 INF IdempotentCmdOutput args=["op","--session","hKujmOm6xL8Cl8NxxoguM2S474ZzEVmvkfGyh1d4BKo","item","get","--format","json","oyzzjv6kpnf2xf3oincqkk72ci"] component=system output="{\n  \"id\": \"oyzzjv6kpnf2xf3oincqkk72ci\",\n  \"title\": \"testitem\",\n ..." path=/usr/local/bin/op
2022-04-27T08:27:09+03:00 INF Close component=persistentState

@halostatue I'm still on macOS 10.14.6 which was EOSed a year ago. 1P7 is still supported AFAIK, but not sure that on 10.14.6. It would be probably better for me to upgrade, than to spend developer's time on debugging obscure issues on an obsolete OS version.

$ uname -a
Darwin pat.local 18.7.0 Darwin Kernel Version 18.7.0: Tue Jun 22 19:37:08 PDT 2021; root:xnu-4903.278.70~1/RELEASE_X86_64 x86_64

[halostatue]

Can you compare the output of

$ chezmoi execute-template "{{ onepassword \"$UUID\") | toJson }}" | jq .

I believe that onepasswordItemFields is no longer as useful with the op v2 data format. The transforms made still make sense, but fewer items seem to produce usable values IMO. It may need a deeper restructuring than has been done to date (and might be better as a filter function than an execution function).

[antonalekseev]

Yes this one yields the proper output. The only thing is that data structure used for fields is not convenient for use in templates and requires either writing boilerplate code like suggested here #474 (comment) or hard coding fields indices. jsonpath-like function would help, but unfortunately is not supported by chezmoi as far as I know.

[SimonBarendse]

Thank you @halostatue for providing such a detailed overview of the problem! I see quite a few friction points in this thread, so going to look and see if there's any changes we can make on the 1Password side to make this easier on your end. I'll get back to you all here.

Cheers, Simon (Developer, Developer Products @ 1Password)

[halostatue]

I have not had time to look at it yet. I’ll keep this notification unread and get back in a day or two.

[gunzy83]

Thanks for the great write up @SimonBarendse, I think goals section makes perfect sense and as a user point 1. actually matches how I am trying to use it (long time chezmoi user, newer to 1password).

After some experimenting and getting over the issue I mentioned in #2353, I think the idempotent op signin or op whoami (ideally scoped to the account specified in the template, particularly if you have multiple accounts) would ensure if we are logged in already we can run chezmoi without prompting or interruption. If the user wants to pre-signin, that is up to them otherwise they get a standard op signin prompt on the first occurence.

I am interested in what @halostatue has to say as his use-cases seem a bit more advanced than mine.

[halostatue]

Hi, @SimonBarendse, and sorry for the long delay in response.

Overall, the plan sounds fine. The lack of a customizable account shorthand when using biometrics means that we will absolutely need to implement the account name aliasing described elsewhere in this discussion, because using either the User ID or Account ID values is opaque and unreadable, and the sign-in address (whether the email or the URL) could be non-unique (I could have two unique user accounts on the same 1P URL, or use the same email address on two different 1P URLs). We can also consider setting up an explicit Chezmoi configuration option to allow for manual mapping.

I do have one concern about the always run op signin option that I cannot test without disabling biometric login, and that’s whether running op signin in a subshell would carry the authorization tracking upwards. Essentially, Chezmoi is doing the following for op signin:

# …process stuff and find a case where we need `op` interactions
op signin --account ${accountId}
op document get …

# …more processing. Another `op` interaction:
op signin --account ${accountId}
op document get …

In the biometric case, this is not an issue. In the non-biometric case, at least with the versions of op and 1Password 7 or 8, the output of op signin is to be evaled because it sets OP_SESSION_x.

It might be easier for us to rely on op signin --raw --account X. If we get a non-zero-length value back, we can store the token and pass it with --session TOKEN on all calls to op. Otherwise, we will need to act sort of like direnv does.

As long as op signin --raw and op … --session TOKEN will continue to work, then I don’t think that there’s any issue. Otherwise, the fact that op signin only works by setting environment variables means that Chezmoi needs to be at least partially aware of them or it cannot call op signin during execution.

[halostatue]

I believe that this discussion is mostly resolved with the following two points:

1. In feat: 1Password account lookup table #2425 and chore: Refactor 1Password account map #2440, we have implemented the 1Password account map. pkg/cmd/onepasswordtemplatefuncs_test.go:TestOnepasswordAccountMap(t *testing.T) is have the best description of the fine details of the behaviour. It it likely that we will be able to take that and put into the 1Password template functions documentation.
2. We are already using op signin --raw --account accountUUID and op … --session TOKEN. If op signin --raw --account accountUUID is made to work in an idempotent manner when OP_SESSION_x is set, then everything works as expected.

While op whoami would be a nice thing to have, if successive calls to op signin are idempotent (generating something new—and therefore prompting—only if the session in the environment has expired) is sufficient to our purposes.

@SimonBarendse, I think we have things working very nicely now.

[SimonBarendse]

As long as op signin --raw and op … --session TOKEN will continue to work

Yes, you can rely on these remaining backwards compatible for all v2.x.y versions. 👍 We won't remove any flags.

[halostatue]

@SimonBarendse, the way that op gets used in chezmoi is through template functions. I mostly use the onepasswordDocument template function (corresponding to op document get …), but there are other functions available. You can test these with chezmoi as follows:

$ chezmoi execute-template "{{ onepasswordItemFields \"$UUID\" | toJson }}"
$ chezmoi execute-template "{{ onepasswordDocument \"$UUID\" }}"

I use document fragments to construct my ~/.aws/credentials file and loop over a set of them. Essentially, I end up filling a template with values like this: {{ onepasswordDocument "aws-credentials: personal" "Personal" "myaccount" }}. (The arguments are document name/id, vault name/id, account name/id.) This turns into an op command looking like op document get 'aws-credentials: personal' --vault Personal --account myaccount.

As long as op sees that my account has a computed (or set) shorthand of myaccount, everything is fine. If, however, with op v1, I had set the shorthand to myaccount for account foobarbaz.1password.ca, then all of a sudden my chezmoi templates no longer work under biometric authentication (because the shorthand value is no longer used or respected). I have to either switch my account parameters to foobarbaz or the account UUID.

If I’m not using biometric authentication, then OP_SESSION_myaccount is no longer set, so chezmoi has to prompt me for login (if I have that enabled; if I do not have that enabled, I can no longer login unless I explicitly do export OP_SESSION_myaccount=$OP_SESSION_{uuid}). Then, because the shorthand still exists, my account reference of myaccount still exists.

There’s at least two issues here:

1. Biometric authentication no longer exposes a user-configurable shorthand for an account. If my shorthand matched my domain prefix, there’s nothing visible. But if my domain prefix was really-awesome-company.1password.eu,but I had a shorthand of rac…I now have to change my configurations.
2. Because the OP_SESSION_ variable name has changed, non-biometric users can’t disable auto-login and have chezmoi Just Work as they could previously. The first post in this discussion (and a follow-up later) suggests a way that chezmoi can work around this to discover the correct OP_SESSION_ variable through heuristics (it would also let us use the discovered value for op --account).

It’s complex and messy and I’m probably not doing a great job of describing it, but these small changes by 1Password CLI have rippled into 3–4 different areas that require workarounds or fixes from 1Password.

I’m not saying that all of the changes made are bad (I do think that requiring --account on op signin is a wrong choice, as I described in the 1password.community thread), but they have impacts and some of them are subtle.

[SimonBarendse]

Is there a reason you don't want to use:

{{ onepasswordDocument "aws-credentials: personal" "Personal" "foobarbaz.1password.ca" }}

The main reason we dropped the concept of a "shorthand" in 1Password version 2 is that we wanted to remove confusion that occurs from the shorthand potentially deviating from what the account is known as otherwise. There's already quite a few concepts for to the sign-in (sign-in address, secret key, password, email-address) and we wanted to remove one to reduce the cognitive overhead required to get started with, use and debug the sign-in process for users. The fact that these shorthands do not exist in 1Password 8 adds to the complexity there'd be in keeping them around (as there's no longer an intuitive place to look them up).

I hope this background provides some insight to our reasoning, I'm looking forward to learn your perspective on this!

I do think that requiring --account on op signin is a wrong choice, as I described in the 1password.community thread

Thank you for providing that reasoning on the forum thread, I'm really appreciating all your feedback and thorough descriptions. I've replied on the forum to this.

[halostatue]

I can think of two reasons:

1. Until you just mentioned it, it was not clear to me that you could use the full sign-in domain name as the account name (which means we would need to add that to the logic above, @twpayne);
2. Backwards compatibility. As I said, I had to update my templates because of the upgrade to op v2, and the reasoning for such was unclear.

Ultimately, I think that a lot of what’s going on is that the changes have been made, but the reasoning behind those changes has not been communicated so that downstream projects like chezmoi could communicate that to their users, even with just a link. This might be because 1Password wasn’t aware all of the places / ways that people were using op.

[jkyjustin]

Apologies for changing the internals of the session keys, there were some technical implications of no longer being able to use the account shorthand as the env var key with biometric auth, and I share your sentiment that we should have communicated that better.

[SimonBarendse]

Hi folks, I'm sorry for my slow replies here. I'm really strapped for time and I want this thread to receive the attention it deserves, so I have asked a teammate, @jkyjustin, to jump in here to help out. You're in great hands!

[jkyjustin]

Hey @halostatue

In theory, I guess it’s possible for the full URL to collide as well, but in that case, the email address would definitely not collide.

Regarding this, yes the full URL can collide. One example of this is a 1Password employee testing across multiple CLI users on a single device - so when specifying the --account flag on signin commands, using the account_uuid or shorthand fields are preferred.

I believe of the two, account_uuid is a safer choice, as shorthand will not be printed as a response of op accout list --format json when biometrics are enabled, so whereas account_uuid is always printed.

[halostatue]

I still think that you should enable sort of shorthand for the accounts even with biometrics enabled; if I am signed in as austin@mycompany.com and admin@mycompany.com (and have different accesses), then I’d like to be able to simply say --account admin even if I’m using biometrics. I know it’s a bit harder, but I think that it could be managed by an extra field (eventually exposed on the non-CLI UIs) that is set by default but can be changed.

[jkyjustin]

Hey @halostatue

I absolutely agree that that would be the ideal behavior, however, because when auth states are shared across the CLI and the desktop applications with biometric authentication enabled, shorthands currently have no meaning to the desktop clients.

While it would be possible to achieve this by getting the desktop clients to implement some sort of meaning to a shorthand (it was previously only a concept for the CLI), we won't be able to use it for biometric auth to switch between accounts of the same signin URL.

[twpayne]

Replying to bump the conversation. I want chezmoi to have first class 1Password support.

[halostatue]

Opened draft #2425 for the 1Password account lookup behaviour.