Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NIFI-5973 Adds ShellUserGroupProvider #3495

Closed
wants to merge 10 commits into from
Closed

Conversation

@natural
Copy link
Contributor

natural commented May 25, 2019

Thank you for submitting a contribution to Apache NiFi.

Please provide a short description of the PR here:

Description of PR

The code in this change-set provides the functionality discussed in NIFI-5973, specifically:

  • adds a new UserGroupProvider implementation called ShellUserGroupProvider
  • shell users + groups support on OSX, Alpine Linux, CentOS, Debian, and Ubuntu
  • adds remote shell support (via ssh) for testing the above
  • adds test for all the above, some via Testcontainers

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

  • Is there a JIRA ticket associated with this PR? Is it referenced
    in the commit message?

  • Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

  • Has your PR been rebased against the latest commit within the target branch (typically master)?

  • Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

  • Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
  • Have you written or updated unit tests to verify your changes?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
  • If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
  • If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Copy link
Contributor

alopresto left a comment

Thanks Troy, this is clearly a lot of great work across a number of difficult to test environments. I tried to run this locally on my Mac OS X 10.14 environment and while I was able to use the LDAP identity provider to authenticate, the mapping to shell-provided users and groups isn't working for me yet. I'll post more details on my testing environment & process.

@@ -452,6 +452,10 @@ The LdapUserGroupProvider has the following properties:

NOTE: Any identity mapping rules specified in _nifi.properties_ will also be applied to the user identities. Group names are not mapped.

==== ShellUserGroupProvider

The ShellUserGroupProvider syncs user and groups from Unix-like systems using shell commands. It has no user-configurable properties.

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 5, 2019

Contributor

I think we may want to provide additional details about how the Shell UGP works (i.e. how it queries the OS for users and groups, and what functionality this enables).

This comment has been minimized.

Copy link
@natural

natural Jun 5, 2019

Author Contributor

I've added text regarding how and why, and a table for the config properties you pointed out in a comment below.

package org.apache.nifi.authorization;

class NssShellCommands implements ShellCommandsProvider {
public String getUsersList() {

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 5, 2019

Contributor

Also commenting on the ShellCommandsProvider interface, but I think each implementation should have generic behavior documented in class-level Javadoc, as well as each method commented with what the expected format of the response is (the format should be common across all providers; the method in which each impl achieves that would be OS-specific).

This comment has been minimized.

Copy link
@natural

natural Jun 5, 2019

Author Contributor

Added comments to the interface methods and methods of both implementations. Included text about the format in the interface.

*/
package org.apache.nifi.authorization;

class OsxShellCommands implements ShellCommandsProvider {

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 5, 2019

Contributor

Same comment from Nss impl.

package org.apache.nifi.authorization;

class OsxShellCommands implements ShellCommandsProvider {
public String getUsersList() {

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 5, 2019

Contributor

I don't think this command is correct, as on my local Mac OS X 10.14 system, this returns:

# username:uid
alopresto:502
...
nobody:-2
root:0

while I believe the expected format is:

# username:uid:gid_of_primary_group
alopresto:502:20
...
nobody:-2:-2
root:0:0

This comment has been minimized.

Copy link
@natural

natural Jun 5, 2019

Author Contributor

You're correct, that's the expected format.

I've updated the OSX shell command to include the output of the primary group at the end. I've also documented the expected format, and added additional logging statements around the output parsing.

*/
package org.apache.nifi.authorization;

interface ShellCommandsProvider {

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 5, 2019

Contributor

Same comment from Nss impl.


import java.util.Set;

import static org.junit.Assert.assertEquals;

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 5, 2019

Contributor

I think our style guide prohibits or at least frowns on static imports.

This comment has been minimized.

Copy link
@natural

natural Jun 6, 2019

Author Contributor

I've removed those static imports.

Is there a way we can add this to the checkstyle definition?

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 6, 2019

Contributor

Yes, it should involve AvoidStarImport and AvoidStaticMemberImport. I can add those to the nifi-checkstyle.xml hosted on the wiki, but I will need to check to see if it affects existing code before enforcing it across the project. I can't find at the moment where those rules are documented, but I remember it being enforced semi-rigidly throughout the life of the project.

public class ShellUserGroupProviderIT extends ShellUserGroupProviderBase {
private static final Logger logger = LoggerFactory.getLogger(ShellUserGroupProviderIT.class);

private final static String ALPINE_IMAGE = "natural/alpine-sshd:latest";

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 5, 2019

Contributor

We may want to publish the Dockerfiles for these publicly on the apache Docker account or include them locally in the build to avoid brittle dependencies on individual user accounts.

This comment has been minimized.

Copy link
@natural

natural Jun 6, 2019

Author Contributor

Agreed. I'll add a note to the ticket + to the code explaining how to do that if/when its appropriate.

}

@Test
public void testGetUsers() {

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 5, 2019

Contributor

Is there any expectation that other classes will extend ShellUserGroupProviderBase? I understand putting reusable functionality in a common location, but it makes these tests very opaque.

This comment has been minimized.

Copy link
@natural

natural Jun 6, 2019

Author Contributor

Not at this point, no. The approach seemed best to accommodate the remote commands (ssh to test containers). I can re-work the test classes if it's a pressing issue.

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 6, 2019

Contributor

Ok, it doesn't need to be refactored, but the test method names & Javadoc can be made more expressive so the "what" of what each method is asserting is clear to the caller.

This comment has been minimized.

Copy link
@natural

natural Jun 6, 2019

Author Contributor

Renamed the test methods to be more descriptive, and added a small javadoc to each.

final String osName = System.getProperty("os.name");
ShellCommandsProvider commands = getCommandsProvider();

if (commands == null) {

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 5, 2019

Contributor

I like the logic to determine the appropriate OS and translation layer for the commands, but I think this logic should be extracted to a utility method and this code can allow for injection if the configuration specifies a particular implementation (self-determining if none is provided).

This comment has been minimized.

Copy link
@natural

natural Jun 6, 2019

Author Contributor

Extracted into a static utility method.

@alopresto

This comment has been minimized.

Copy link
Contributor

alopresto commented Jun 5, 2019

Steps to test on Mac OS X 10.14

  1. Build the PR as normal
  2. Secure the instance using the TLS Toolkit (followed normal procedure to generate certificates/keystores and populate nifi.properties)
  3. Configure the LDAP login identity provider to allow for authentication (used the same resources as when validating NIFI-6085 in PR 3362)
  4. Configure the shell-user-group-provider in authorizers.xml rather than the file-user-group-provider
  5. Populate user identity mappings in nifi.properties to map the LDAP DN (cn=alopresto,ou=people,dc=nifi,dc=com) to the OS user (alopresto) (see below)
  6. Start NiFi
  7. Try to log in

Here is where I received a Unknown user with identity 'alopresto'. Contact the system administrator. error in the NiFi UI and the below logs in logs/nifi-user.log:

2019-06-04 17:22:08,433 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for cn=alopresto,ou=people,dc=nifi,dc=com
2019-06-04 17:22:08,433 INFO [NiFi Web Server-24] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[alopresto], groups[] does not have permission to access the requested resource. Unknown user with identity 'alopresto'. Returning Forbidden response.

Identity mapping transformations:

# Case-insensitive regex to extract only the CN and make that the complete identity
nifi.security.identity.mapping.pattern.dn=(?i)^CN=([^,]*),.*$
nifi.security.identity.mapping.value.dn=$1

My (generated by the code in this PR) authorizations.xml looks like this, where my OS user uid is 502:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizations>
    <policies>
        <policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
            <user identifier="502"/>
        </policy>
        <policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
            <user identifier="502"/>
        </policy>
        <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
            <user identifier="502"/>
        </policy>
        <policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
            <user identifier="502"/>
        </policy>
        <policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
            <user identifier="502"/>
        </policy>
        <policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
            <user identifier="502"/>
        </policy>
        <policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
            <user identifier="502"/>
        </policy>
        <policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
            <user identifier="502"/>
        </policy>
    </policies>
</authorizations>

I don't see any policy for the root process group (usually indicated by 2 policy elements with resource=/process-groups/UUID and action=R/action=W). I also note that the authorizer framework believes my user to belong to no groups.

@alopresto

This comment has been minimized.

Copy link
Contributor

alopresto commented Jun 5, 2019

Also I think an integration test for the providers on Mac OS X would be helpful as a lot of contributors run on that platform and it would at least demonstrate expected outputs of the OS X-specific shell commands.

@natural

This comment has been minimized.

Copy link
Contributor Author

natural commented Jun 5, 2019

Thanks Troy, this is clearly a lot of great work across a number of difficult to test environments. I tried to run this locally on my Mac OS X 10.14 environment and while I was able to use the LDAP identity provider to authenticate, the mapping to shell-provided users and groups isn't working for me yet. I'll post more details on my testing environment & process.

Andy, this is super feedback. Thank you for taking the time to review and comment with such detail. I'll update the PR with changes and replies to your comments.

@natural

This comment has been minimized.

Copy link
Contributor Author

natural commented Jun 6, 2019

I think I've addressed all of your comments with changes in b37f4c1. I'm going to circle back and revisit your setup on Mac OS.

|`Refresh Delay` | Number of seconds to pause between each listing refresh.
|==================================================================================================================================================

Like LdapUserGroupProvider, the ShellUserGroupProvider is commented out in the _nifi.properties_ file. Refer to that comment for usage examples.

This comment has been minimized.

Copy link
@andrewmlim

andrewmlim Jun 11, 2019

Contributor

Should be authorizers.xml file not nifi.properties

This comment has been minimized.

Copy link
@natural

natural Jun 12, 2019

Author Contributor

Good catch. Updated ref and added the properties to the example.

Supported systems may be configured to retrieve users and groups from an external source, such as LDAP or NIS. In these cases the shell commands
will return those external users and groups. This provides administrators another mechanism to integrate user and group directory services.

The ShellUserGroupProvider has the following properties:

This comment has been minimized.

Copy link
@andrewmlim

andrewmlim Jun 11, 2019

Contributor

These two properties are missing from the authorizers.xml file

<userGroupProvider>
<identifier>shell-user-group-provider</identifier>
<class>org.apache.nifi.authorization.ShellUserGroupProvider</class>
<property name="Initial Refresh Delay">30</property

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 13, 2019

Contributor

I think we would prefer these values to include the units like the LDAP timeouts above: https://github.com/apache/nifi/pull/3495/files#diff-8f577f792e128744ad8f8f5eea585ac4R144

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 13, 2019

Contributor

Here is the LDAP code for parsing those values into the proper numerical value given the time units:

final PropertyValue rawSyncInterval = configurationContext.getProperty(PROP_SYNC_INTERVAL);

I think the existing FormatUtils.getTimeDuration() is deprecated; use FormatUtils.getPreciseTimeDuration() instead.

This comment has been minimized.

Copy link
@natural

natural Jun 16, 2019

Author Contributor

Great idea. Added support for unit formatting.

* @return Shell command string that will return a list of groups for a user.
*/
public String getUserGroups() {
return "id -nG %s | sed s/\\ /,/g";

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 13, 2019

Contributor

I think this method should change similar to getGroupMembers(String groupName) -- getUserGroups(String user).

This comment has been minimized.

Copy link
@natural

natural Jun 15, 2019

Author Contributor

The method wasn't used, so it's been removed.

/**
* @return Shell command string that will return a list of groups for a user.
*/
public String getUserGroups() {

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 13, 2019

Contributor

Same comment from the *nix implementations -- this method should accept a String parameter.


/**
* @return Shell command string that will return a list of users for a group.
* @param groupName name of group.

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 13, 2019

Contributor

I think Javadoc style is that @param always precedes @return. The pattern is usually:

/**
 * Concise description of the method. 
 * 
 * @param param1 more info than just redundantly stating the datatype (i.e. is null valid, does a negative value have special meaning, etc.)
 * @param param2 ""
 * @return the meaning and potentially range of the return value of this method
 */
boolean isExampleMethod(int param1, int param2) {

This comment has been minimized.

Copy link
@natural

natural Jun 15, 2019

Author Contributor

Ah, that's what I get for typing the javadoc manually. :) Fixed.

private Integer remotePort;

// change to a public ctor
private RemoteShellCommands() {

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 13, 2019

Contributor

Is this supposed to be a public constructor, or is the "TODO" comment above OBE?

This comment has been minimized.

Copy link
@natural

natural Jun 15, 2019

Author Contributor

Inaccurate comment, removed.

return String.format(remoteCommand, innerProvider.getUsersList(), privateKeyPath, remotePort, remoteHost);
}

public String getUserGroups() {

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 13, 2019

Contributor

I think this method might need to accept a String parameter as well.

* `user-name-1,user-name-2,user-name-n`
*
* @return Shell command string that will return a list of users for a group.
* @param groupName name of group.

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 13, 2019

Contributor

Same Javadoc element ordering comment as above.

This comment has been minimized.

Copy link
@natural

natural Jun 15, 2019

Author Contributor

Fixed.

}
}

public ShellCommandsProvider getCommands() {

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 13, 2019

Contributor

Is this method necessary? Do we allow external classes to consume this internal object? And if it is necessary, its naming convention should be getCommandsProvider() to match setCommandsProvider().

This comment has been minimized.

Copy link
@natural

natural Jun 15, 2019

Author Contributor

That's a good catch. I've renamed the method in question to getCommandsProvider. I've also renamed the OS commands selector method to getCommandsProviderFromName.

logger.warn("Null or empty user name: " + name + " or id: " + id);
}
} else {
logger.warn("Unexpected record format. Expected 3 or more comma separated values per line.");

This comment has been minimized.

Copy link
@alopresto

alopresto Jun 13, 2019

Contributor

I think this error should be "... colon separated values ..."

This comment has been minimized.

Copy link
@natural

natural Jun 15, 2019

Author Contributor

Yes, fixed.

@alopresto

This comment has been minimized.

Copy link
Contributor

alopresto commented Jun 13, 2019

When testing (again, Mac OS X 10.14), I enabled debug logging in the nifi-user.log and tried to log in with a known user and an unknown one. For both, the authentication worked (the user did exist in LDAP), but the authorization failed ("Unknown user"). It appears there is an error occurring when loading the groups for a specific user?

🔓 57s @ 16:36:42 $ tail -f logs/nifi-user.log
2019-06-13 16:36:14,398 INFO [main] o.a.n.a.FileAccessPolicyProvider Populating authorizations for Initial Admin: alopresto
2019-06-13 16:36:14,406 INFO [main] o.a.n.a.FileAccessPolicyProvider Authorizations file loaded at Thu Jun 13 16:36:14 PDT 2019
2019-06-13 16:36:14,417 DEBUG [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.dn was found, but no transform was present. Using NONE.
2019-06-13 16:36:14,417 DEBUG [main] o.a.n.a.util.IdentityMappingUtil Found Identity Mapping with key = dn, pattern = (?i)^CN=([^,]*),.*$, value = $1, transform = NONE
2019-06-13 16:36:14,471 DEBUG [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.dn was found, but no transform was present. Using NONE.
2019-06-13 16:36:14,471 DEBUG [main] o.a.n.a.util.IdentityMappingUtil Found Identity Mapping with key = dn, pattern = (?i)^CN=([^,]*),.*$, value = $1, transform = NONE
2019-06-13 16:36:14,494 DEBUG [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.dn was found, but no transform was present. Using NONE.
2019-06-13 16:36:14,494 DEBUG [main] o.a.n.a.util.IdentityMappingUtil Found Identity Mapping with key = dn, pattern = (?i)^CN=([^,]*),.*$, value = $1, transform = NONE
2019-06-13 16:36:14,552 DEBUG [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.dn was found, but no transform was present. Using NONE.
2019-06-13 16:36:14,552 DEBUG [main] o.a.n.a.util.IdentityMappingUtil Found Identity Mapping with key = dn, pattern = (?i)^CN=([^,]*),.*$, value = $1, transform = NONE
2019-06-13 16:36:44,514 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:44,596 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:44,643 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:44,738 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:44,785 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:44,833 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:44,880 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:44,929 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:44,976 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,025 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,074 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,121 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,208 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,250 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,293 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,337 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,419 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,500 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,542 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,584 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,627 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,670 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:45,767 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:36:46,031 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:14,649 INFO [NiFi Web Server-84] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://andy.nifi:9443/nifi-api/flow/current-user (source ip: 127.0.0.1)
2019-06-13 16:37:14,653 INFO [NiFi Web Server-84] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for alopresto
2019-06-13 16:37:14,744 INFO [NiFi Web Server-84] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[alopresto], groups[] does not have permission to access the requested resource. Unknown user with identity 'alopresto'. Returning Forbidden response.
2019-06-13 16:37:16,262 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,350 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,394 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,481 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,525 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,571 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,615 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,660 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,704 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,749 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,793 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,837 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,922 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:16,964 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:17,007 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:17,050 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:17,137 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:17,226 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:17,274 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:17,319 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:17,362 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:17,405 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:17,491 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:17,733 ERROR [pool-8-thread-1] o.a.n.a.ShellUserGroupProvider refreshGroup list membership returned zero lines.
2019-06-13 16:37:30,286 INFO [NiFi Web Server-33] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://andy.nifi:9443/nifi-api/flow/current-user (source ip: 127.0.0.1)
2019-06-13 16:37:30,288 INFO [NiFi Web Server-33] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for admin
2019-06-13 16:37:30,290 INFO [NiFi Web Server-33] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[admin], groups[] does not have permission to access the requested resource. Unknown user with identity 'admin'. Returning Forbidden response.
^C
@alopresto

This comment has been minimized.

Copy link
Contributor

alopresto commented Jun 13, 2019

The relevant portions of my authorizers.xml file:

    <userGroupProvider>
        <identifier>shell-user-group-provider</identifier>
        <class>org.apache.nifi.authorization.ShellUserGroupProvider</class>
        <property name="Initial Refresh Delay">30</property>
        <property name="Refresh Delay">30</property>
    </userGroupProvider>
...
    <accessPolicyProvider>
        <identifier>file-access-policy-provider</identifier>
        <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
        <property name="User Group Provider">shell-user-group-provider</property>
        <property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Initial Admin Identity">alopresto</property>
        <property name="Legacy Authorized Users File"></property>
        <property name="Node Identity 1"></property>
        <property name="Node Group"></property>
    </accessPolicyProvider>
...
    <authorizer>
        <identifier>managed-authorizer</identifier>
        <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
        <property name="Access Policy Provider">file-access-policy-provider</property>
    </authorizer>

After initial startup (these files did not exist and were created):

No users.xml.

authorizations.xml:

🔓 0s @ 16:41:57 $ more conf/authorizations.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizations>
    <policies>
        <policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
            <user identifier="502"/>
        </policy>
        <policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
            <user identifier="502"/>
        </policy>
        <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
            <user identifier="502"/>
        </policy>
        <policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
            <user identifier="502"/>
        </policy>
        <policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
            <user identifier="502"/>
        </policy>
        <policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
            <user identifier="502"/>
        </policy>
        <policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
            <user identifier="502"/>
        </policy>
        <policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
            <user identifier="502"/>
        </policy>
    </policies>
</authorizations>

Where 502 is the uid of alopresto on this OS.

@seanorama

This comment has been minimized.

Copy link

seanorama commented Jun 15, 2019

Under the Unix NSS implementation, will it identify groups for users who do not show in the getent passwd list?

The most common scenario for the use of the shell provider is when NSS is integrated with a companies LDAP directory (typically using SSSD).

To avoid performance issues with getent showing 1000s (or more users), it's often configured and often have configured with enumerate=false which means the users/groups will not show in getent passwd/group.

However they will show when queried directly (i.e. getent passwd ${user}, getent group ${group}, id ${user}). Ideally the NSS implementation would determine at the time they do something as it would be for other OS integrated applications.

@natural

This comment has been minimized.

Copy link
Contributor Author

natural commented Jun 16, 2019

Under the Unix NSS implementation, will it identify groups for users who do not show in the getent passwd list?

Yes! As of bbbc8cf, the provider supports reading users and groups from the systems where enumerate=false.

natural added a commit to natural/nifi that referenced this pull request Jun 19, 2019
Supersedes apache#3495.
@natural natural mentioned this pull request Jun 19, 2019
8 of 11 tasks complete
@natural

This comment has been minimized.

Copy link
Contributor Author

natural commented Jun 19, 2019

Closing in favor of #3537.

@natural natural closed this Jun 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.