item.shared_with Groups permissions issue

[MetroparksGIS]

Describe Issue
There appears to be a permissions issue when accessing the groups an item is shared with. I can still access the 'everyone' and 'org' values without any issues. This only occurs for 23 out of 198 feature services. All items are owned by the same user, and after comparing all of the items that cause the error and the items that work without issue, there don't appear to be any patterns or similarities that would cause these items to fail.

History
Within the past year all layers on AGO transferred from one account to another. I have verified that the account I'm logging in through and the owner of feature services the issue occurs for are the same.

Platform
Windows 10 Pro
Python v3.7.9
ArcGIS API v1.8.5
Updated to Python v3.7.11 and API v1.9.1 and the issue remains

Example of the error

item = gis.content.search(query='id:[itemId_here]')
for k,v in item[0].shared_with.items():
	print(k,v)
	
everyone False
org True
groups Traceback (most recent call last):
  File "<pyshell#60>", line 2, in <module>
    print(k,v)
  File "C:\Users\...\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\__init__.py", line 6343, in __repr__
    return '<%s title:"%s" owner:%s>' % (type(self).__name__, self.title, self.owner)
  File "C:\Users\...\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\__init__.py", line 6321, in __getattr__
    self._hydrate()
  File "C:\Users\...\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\__init__.py", line 6314, in _hydrate
    groupdict = self._portal.get_group(self.groupid)
  File "C:\Users\...\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 949, in get_group
    return self.con.post('community/groups/' + group_id, self._postdata())
  File "C:\Users\...\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 720, in post
    force_bytes=kwargs.pop('force_bytes', False))
  File "C:\Users\...\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 514, in _handle_response
    self._handle_json_error(data['error'], errorcode)
  File "C:\Users\...\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 536, in _handle_json_error
    raise Exception(errormessage)
Exception: You do not have permissions to access this resource or perform this operation.
(Error Code: 403)

The error is exactly the same if I call 'groups' with the code below:

item[0].shared_with['groups']

To Reproduce
The below code can be run to find feature services within your AGO account that this error occurs for.

from arcgis.gis import GIS

gis = GIS('https://www.arcgis.com',[username],[password])

user = gis.users.me
folders = user.folders

for f in folders:
    items = user.items(f['title'])
    
    for item in items:
        groups = []
        
        try:
            if item.shared_with['groups'] != []:
                for g in item.shared_with['groups']:
                    groups.append(g.title)
        except:
            print(item.id)

[nanaeaubry]

@achapkowski I cannot reproduce this issue with 2.0.0 and using Online or Enterprise 10.9.1

[MetroparksGIS]

@achapkowski I cannot reproduce this issue with 2.0.0 and using Online or Enterprise 10.9.1

Maybe it's been corrected already in 2.0.0. I think ArcGIS Pro v2.9.2 installs ArcGIS API v1.9.1 by default. I'll update to the latest API version and see if the issue is still presenting. If so, maybe it's something specific to my AGO account.

[MetroparksGIS]

@achapkowski I cannot reproduce this issue with 2.0.0 and using Online or Enterprise 10.9.1

I installed v2.0.0 and reran my test script above. The same 23 out of 198 feature services are failing. I would assume something may have happened when the owner account for all of the layers was switched to the new account, but can't be sure. All 23 feature services were published prior to the switch. The only thing I find odd with this answer is it seems like any calls on one of these feature services should fail, but it's only failing for shared_with['groups'].

[nanaeaubry]

Ok I will continue taking a look and trying to reproduce, and post if I find anything. You might be right that there is something going on with those services specifically. Without having them it is hard to test.

[achapkowski]

@MetroparksGIS the groups that the Item is shared_with, are they ones within your organization? owned by you? Is your account an admin?

[MetroparksGIS]

@achapkowski Yes. I work at a small agency and all layers, groups, etc. in our AGO account are owned by one username. That account is also the admin account for the agency. The possible issue is that we used to use a different account as the admin account, and that account used to own all the layers. Within the past year, we switched everything over to a new account. I can use the API to tell me that the owner of all these layers are owned by the new admin account, but is it possible somewhere else in another file associated with the layer the old account name never got switched over? If it's only stored in the one location that the API is accessing, than I'm lost.

If I use the below script, all 23 layers that are failing for shared_with['groups'] show up in the search:

from arcgis.gis import GIS

gis = GIS('https://www.arcgis.com',[username],[password])

my_content = gis.content.search(query='owner:' + gis.users.me.username, max_items=1000)

for i in my_content:
    print(i.title)

[achapkowski]

@MetroparksGIS does the old account still own groups?

[MetroparksGIS]

@achapkowski I just checked. All groups are owned by the new account.

[achapkowski]

@MetroparksGIS if you go to the Item's page as an administrator for the 23 items that are giving errors, do they all have a common group that they belong to?

[MetroparksGIS]

@achapkowski Attached is a table showing the error items and what groups they are shared with. At the bottom is a smaller table showing other items that are shared with the same combination of groups, but that the API works as expected for.

[AndrewBuiltThis]

@MetroparksGIS - do y'all have any ArcGIS Hub groups or ArcGIS Hub partner groups? We are also experiencing this and think there is a relationship between shared groups (groups with members from different organizations) and/or Hub groups with this behavior.

[MetroparksGIS]

@AndrewBuiltThis We don't have ArcGIS Hub set up yet. We do have one group shared with outside organizations, but the feature services shared with that group actually work as expected for us.

[mcgooly]

Not sure if this applies to this bug, but we ran into the same permissions issue with some of our public items. Long story short, users from outside of our organization decided to add our items to their own external groups, and unless their groups are shared to the public, the permissions issue happens. For example, we share one of our public items with a single internal group. But this line gives the permission issue:

(Pdb) item.shared_with['groups']  # also fails with just item.shared_with
*** Exception: You do not have permissions to access this resource or perform this operation.
(Error Code: 403)

My assumption was some sort of bug broke the 'groups' entry, but that's not the case - this line shows that 'groups' is a valid key and in fact was populated with six groups(!):

(Pdb) type(item.shared_with['groups'])
<class 'list'>
(Pdb) len(item.shared_with['groups'])
6

So you can iterate over the list, then catch the exception to handles ones with the permissions error. (The arcgis code raises a general exception in these cases so you must use "except Exception" and not something more specific.)

for group in item.shared_with['groups']:
    try:
        # do whatever with the group
        print((group.title, group.owner, group.access))
    except Exception:
        print('... no permissions to this group')

It's a clunky workaround but it addresses our issue. In our case, we could see two of the six groups - one was ours, another was external but shared publicly - and four groups had permissions errors which I assume were external but not shared.

So IMO the real bug is how the arcgis code handles these cases where you have permissions to some groups but not to others. Ideally in these instances calling item.shared_with shouldn't return an error by default.

[achapkowski]

@mcgooly thank you for this example of a failure.

[MetroparksGIS]

@mcgooly - Thanks so much for this! This helped me narrow down and fix our issue. Iterating over the groups did show one group that our account didn't have access to. I decided to switch out the username/password in the script with our old admin account, and it turns out this was the issue after all. If an account linked to an organization favorites a map, layer, file, etc. it gets added to that accounts favorites list. Even if that account leaves the organization or loses access to the files, AGO maintains that favorites connection and this is what was showing up as a "group" through the API.

Luckily we could still access that old account, unfavorited all the items, and the permissions error is gone. This definitely seems like an odd way for the API to handle this, but it's good to know that items favorited or shared by external accounts will cause this issue.

[mcgooly]

@MetroparksGIS I'm just glad you opened this issue in the first place otherwise I may have abandoned looking for a solution altogether. Sounds like you had an even less obvious edge-case to deal with, yikes! I'm glad you worked it out.

[jyaistMap]

Closing.