Diagnosing an authentication failure

[larsks]

I help run an OpenStack environment at the Massachusetts Open Cloud. I am trying to help @kcondon resolve an authentication problem between a dataverse instance and our Swift service. They are seeing failures at
https://github.com/javaswift/joss/blob/master/src/main/java/org/javaswift/joss/command/shared/identity/access/KeystoneV3Access.java#L89 because the keystone response does not contain a catalog element, which suggests that the authentication request is not being made appropriately. It's unclear if this is a code error or a configuration error.

I would like to try to reproduce this failure locally. I am hoping someone can provide me with the Java equivalent -- using the joss library -- of something like the following Python code:

import openstack
client = openstack.connect()
print(client.list_containers())

[landreev]

Hi Lars,
The dataverse Java class where this is happening is SwiftAccessIO (https://github.com/IQSS/dataverse/blob/develop/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java); but I can try and reduce the important part to a more compact fragment - as in, a real equivalent of the python code above.

OK, I just saw your MOC support ticket message, that you got my attachment, with the full HTTP exchange log. Please let us know if that clarifies things further, or if there's more diagnostics we could provide.

Thank you for looking into this!

[landreev]

The issue #5478 is completely unrelated to what we are investigating here - please disregard.

[larsks]

@landreev the one thing I note in that http log is that your code is making an unscoped token request. That is, the request data does not include a project element. You have:

{
  "auth": {
    "identity": {
      "methods": [
        "password"
      ],
      "password": {
        "user": {
          "name": "lars",
          "password": "secret",
          "domain": {
            "name": "Default"
          }
        }
      }
    }
  }
}

But I think you need a scope element to scope the token to your project:

{
  "auth": {
    "identity": {
      "methods": [
        "password"
      ],
      "password": {
        "user": {
          "name": "lars",
          "password": "secret",
          "domain": {
            "name": "Default"
          }
        }
      }
    },
    "scope": {
      "project": {
        "name": "lars",
        "domain": {
          "name": "Default"
        }
      }
    }
  }
}

I don't know if this is a code or a configuration issue.

[larsks]

@pdurbin points out that auth_type can be either keystone or keystone_v3. I suspect @landreev's issue may stem from using keystone. I have suggested they try keystone_v3 and try again to see if that changes things.

[landreev]

Interesting, let me take a quick look.
If it's really a matter of adding extra parameters to how we cook that token request with Joss, that would be the best possible scenario. I was preparing myself for the possibility that we would need to go and figure out what's going on in Joss and make changes to their code, in order to fix this.

A couple of quick questions:

• In the exchange that we are looking at, with @kcondon trying to authenticate in order to access his Swift endpoint, what would be the correct project name to pass in that extra fragment, that would define the right scope, is it "CloudDV"? (that's the project name he's seeing in his Kaizen console).
  So, the scope fragment should look like this:

    {
       ...
        "scope": {
          "project": {
            "name": "CloudDV",
            "domain": {
              "name": "Default"
            }
          }
        }
      }
    }

correct?

• Any guess as to why we were able to get away without specifying the scope, with the older version(s) of the MOC? (just curious, really - it's not that important)

[landreev]

We are already using keystone_v3.

[landreev]

Another quick question: in the OpenStack lingo, are the words "project" and "tenant" interchangeable? Do they mean the same thing? ("project" as in "scope": { "project": { "name": ... above)

[larsks]

The words "project" and "tenant" are mostly interchangeable. The phrase "tenant" was deprecated a few years back, and all the new documentation and apis refer to "project".

If you see the MOC ticket, I believe we can work around this problem by setting a default project for the user; this will be used to generated project scoped tokens by default if no explicit scope is provided in the request. I believe this is a bug in either the dataverse code or the underlying joss code, which should be making appropriately scoped authentication requests (because we can use the "default project" concept to work around this particular issue, we would be out of luck if the user account used for authentication was (a) a member of multiple projects and (b) needed a token scoped to a project other than the default).

Re: your earlier question, yes, that scope fragment looks correct.

[landreev]

For reference, the MOC support ticket that @kcondon opened a week ago is https://osticket.massopen.cloud/tickets.php?id=180

A whole lot of diagnostics and back-and-forth, including switching to keystone and back to keystone_v3 was already done there.

(I'm getting mixed results accessing the ticket URL above without authenticating; most likely, it is not readable to the world by default. I'll make sure that all the info important for the Dataverse-side troubleshooting and development is duplicated here, in the Dataverse GitHub issue.

We'll use this issue to do whatever needs to be done on the Dataverse side/in the Dataverse code; and we'll spawn extra dev. issue(s) in the Joss project, if it becomes necessary. (But the hope still is that we can get away without it)

For the record, Joss does not appear to be regularly maintained. The last commit there is from @ferrys - Sarah Ferry - made exactly a year ago. So if it turns out we do need something changed there, we'll have to take matters into our own hands.

[landreev]

@larsks Thanks, that's all super useful.
Yes, I am trying to figure out how to make our requests scoped; so that we don't have to rely on something that may or may not be set on the OpenStack service end.
What's a little bit discouraging so far, is that the Dataverse code that initiates the authentication already appears to be doing this:

else if (swiftEndPointAuthMethod.equals("keystone_v3")) {
                logger.fine("Authentication type: keystone_v3");
                account = new AccountFactory()
                        .setTenantName(swiftEndPointTenantName)
                        .setUsername(swiftEndPointUsername)
                        ... etc.

and since Kevin has had that "swiftEndPointTenantName" set to "CloudDV" in his configuration all along - one would expect that it would be used to scope the request properly... So maybe this is something that will need to be tinkered with on the Joss side.

I'll try and test if the change you've made to Kevin's account makes the whole thing work, in a moment.

[landreev]

OK, we can now upload files through Dataverse, yes.
(there may still be a problem with the file ending up in a "folder" that's different from where we want it to be - but that's definitely an issue on the Dataverse side...)
And as I said earlier, I'm still looking into the issue of our requests not being explicitly scoped.

(Interestingly enough, when @kcondon and I were looking at his account info in the Kaizen console earlier today, before you made that change, it did say "CloudDV" under "Project" already)

[landreev]

OK, this is unrelated to the authentication and/or MOC accounts issues - but we also broke the way the StorageIdentifier is formed for swift-stored files. We must have broken it some time between now and the last time it was known to work with MOC (~6 months ago?). As a result, the file is uploaded onto the swift volume; you can download it from there directly. But Dataverse can't find it - so the file can't be downloaded via Dataverse. We done broke it!

Since, once again, this is unrelated to the authentication, I'm opening a new issue for this.

[PaulBoon]

@landreev it looks like the authentication problem has not been fixed in the code.
Both with 4.14 and 4.10.1 I cannot get authentication working with my Swift endpoint. I can use it with Cyberduck and curl commands, so it should be possible. Would it be best to create a new issue for this, or can I add to another existing one?

[pdurbin]

@PaulBoon I'd like to suggest creating a fresh issue. Thanks!

[pdurbin]

@larsks are you still interested in this?

[larsks]

@pdurbin I am not!

[pdurbin]

@larsks thanks for closing this. I hope we can work together again soon! 😄