upload fails with <Response [403]> in sharepoint ADFS.

[gauravkumar13]

I'm using below lines to upload a file to subdirectory of Sharepoint online which has ADFS in background but this is never working for me to upload.

Python code:

import sharepy 
s = sharepy.connect("xxxx.sharepoint.com", "xxxx@org.com", "password") 
r = s.post("https://xxxx.sharepoint.com/sites/*Team-Name*/_api/web/GetFolderByServerRelativeUrl('/sites/*Team-Name*/Shared Documents/*path to sub directory*')/Files/" + "add(overwrite=true,url='test.xlsx')", "testing,foo,bar")
print (r)

Above code is executed with <Response [403]> value in 'r'.

Kindly Help on this.
Thanks,
Gaurav

[JonathanHolvey]

The 403 response code means that you don't have permission to access the resource you're trying to access (or create in this case). You should probably double check the path you're trying to upload to, and check with your site administrator.

For more information on that status code, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403

Additionally, you should check your POST request call, as it doesn't look quite right to me. You should have something like this:

with open(filepath, "rb") as f:
    data = f.read()

url = "https://example.sharepoint.com/GetFolderByServerRelativeUrl('{}')/Files/add(url='{}',overwrite=true)"
r = s.post(url.format(folder, filename), data=data, headers={"content-length": len(data)})

For a full example on uploading files, see #4 (comment). I've also added this to the project readme.

[gauravkumar13]

@JonathanHolvey - I went through your example code but was just trying to test if upload is working fine.
After modifying I've below lines in my script.

import sharepy
import urllib
s = sharepy.connect("xxxx.sharepoint.com", "xxxx@org.com", "password")
folder = "/Shared Documents/xxxx/xxxx/xxxx/Reports"
filename = "test.xlsx"

with open(filename, "rb") as f:
    data = f.read()

url = "https://xxxx.sharepoint.com/_api/web/GetFolderByServerRelativeUrl('{}')/Files/add(url='{}',overwrite=true)"
r = s.post(url.format(folder, filename), data=data, headers={"content-length": str(len(data))})

print (r)

Above code is executed with <Response [404]> value in 'r'. (404 says the resource itself does not exist as per link).
Although "test.xlsx" is in current working directory and seems I've put correct value in folder variable.
I'm not able to understand what I'm missing.

[JonathanHolvey]

Can you try replacing the space in the folder name with %20? It could be that the SharePoint server expects that space to be URL-encoded.

[gauravkumar13]

I'm still getting "<Response [404]>" even after replacing space with %20.

[JonathanHolvey]

Apologies, I just realised the URL in my example above was missing /_api/web. I've corrected this now.

[gauravkumar13]

I'm back to original error "<Response [403]>".
I tried manually and I'm able to upload file perfectly to sharepoint, not sure why this is throwing 403 error.

[gauravkumar13]

When I open my company sharepoint site (xxxx.sharepoint.com) in a private browser then after login to sharepoint using xxxx.org.com this is redirecting to ADFS server where I need to login using company id (xxxx) not email (like xxxx.sharepoint.com) and password then It's landing to sharepoint site.
May be this middle adfs authentication is not happening.

[JonathanHolvey]

Are you able to access any data on the site through SharePy using a GET request?

[JhongTingDanny]

Same issue here. i run the code in Azure pipeline.

s = sharepy.connect(sharepoint_url, username=username, password=password)
update = s.post(url=f"https://myhavi.sharepoint.com/sites/XXXXXXXX/_api/web/lists/GetByTitle('{sharepoint_list_name}')/items({id})",
data=json_object,headers={'X-HTTP-Method':'MERGE', 'If-Match':'*','User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'
, 'accept': 'application/json;odata=verbose',
'Content-Type': 'application/json'
})

here is the output i use 'GET' request method to get the with status code 200
2022-02-13T16:12:46.4703270Z status code 200
2022-02-13T16:12:46.4705063Z b'{"d":{"__metadata":{"id":"https://myhavi.sharepoint.com/sites/XXXXXXXX/_api/Web/Lists(guid\'cd3ca1f0-3a85-41de-9899-a746323af7c0\')","uri":"https://myhavi.sharepoint.com/sites/XXXXXXXX/_api/Web/Lists(guid\'cd3ca1f0-3a85-41de-9899-a746323af7c0\')","etag":"\\"231\\"","type":"SP.List"},"ListItemEntityTypeFullName":"SP.Data.OFC_x005f_009_x005f_Cargowise_x0020_ShipmentListItem"}}'

but when i try to use "Post" request
here is the error 403
403
b'{"error":{"code":"-2130575251, System.Runtime.InteropServices.COMException","message":{"lang":"en-US","value":"The security validation for this page is invalid. Click Back in your Web browser, refresh the page, and try your operation again."}}}'
Unknown status code 403 encountered.

below is another pipeline has same issue.
{"error":{"code":"-2130575251, Microsoft.SharePoint.SPException","message":{"lang":"en-US","value":"The security validation for this page is invalid and might be corrupted. Please use your web browser's Back button to try your operation again."}}}
b'{"error":{"code":"-2130575251, Microsoft.SharePoint.SPException","message":{"lang":"en-US","value":"The security validation for this page is invalid and might be corrupted. Please use your web browser's Back button to try your operation again."}}}'
403

Please help!!!

[JonathanHolvey]

I'm wondering if this might be related to #49. Can you try adding the following header to your request?

headers = {
  'Authorization': f'Bearer {self.auth.digest}',
}

[gauravkumar13]

Are you able to access any data on the site through SharePy using a GET request?

@JonathanHolvey - I'm using below code which is working in my local system with 200 status (which is preauthorized to adfs server because of sso) but same is not working while running in Jenkins python container agent and throwing authentication error.

url = "https://xxxx.sharepoint.com/sites/subsite/_api/web/GetFolderByServerRelativeUrl('{}')/Files/add(url='{}',overwrite=true)"
s.headers.update({'Authorization': 'Bearer ' + s.auth.digest})
r = s.post(url.format(folder, filename), data=data)

Jenkins agent error:
Traceback (most recent call last):
File "/home/jenkins/agent/workspace/test1/upload.py", line 4, in
s = sharepy.connect("xxxx.sharepoint.com", "xxxx@org.com", "password")
File "/home/jenkins/.local/lib/python3.10/site-packages/sharepy/session.py", line 15, in connect
return SharePointSession(site, auth=autoauth)
File "/home/jenkins/.local/lib/python3.10/site-packages/sharepy/session.py", line 61, in init
self.auth.login(self.site)
File "/home/jenkins/.local/lib/python3.10/site-packages/sharepy/auth/adfs.py", line 27, in login
self._get_token()
File "/home/jenkins/.local/lib/python3.10/site-packages/sharepy/auth/adfs.py", line 63, in _get_token
raise errors.AuthError.fromxml(root)
sharepy.errors.AuthError: Unknown authentication error

I can clearly see it's failing at adfs authentication, could you please help to add one more step for adfs authentication (with credentials) in script .

[mengdl]

Same issue.
After downgrading to sharepy 1.3.0, the codes works fine.
The issue seems only happens on the new version 2.0.0. @JonathanHolvey

[JhongTingDanny]

Thanks it solved when roll back the version to 1.3.0

[gauravkumar13]

This is not working for me as I've different credentials for adfs and sharepoint.

[puntomaupunto]

@JonathanHolvey
Hi!
I am getting HTTP status 403 when trying to upload a file with sharepy 2.0.0. But when I tried to install 1.3.0 I cannot even authenticate. (if I use 2.0.0, I can authenticate and I am able to download files).
I think that there is some difference in connect method, but I don't know what it is...

[fdavidcb-1]

@JonathanHolvey Hello
I'm having simmilar behaviour:

SharePy v 1.3.0 - Can do all the tasks: connect, download, upload
SharePy v 2.0.0 - Can connect, download

Trying to cope with some issues arround file permisions I got the next message (v 2.0.0):

Traceback (most recent call last):
File "/Get_Files_From_SharePoint_debug.py", line 87, in current_session
session = sharepy.load(s_file)
File "/usr/.local/lib/python3.6/site-packages/sharepy/session.py", line 28, in load
if session.auth.refresh() or session.auth.login():
AttributeError: 'NoneType' object has no attribute 'refresh'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Get_Files_From_SharePoint_debug.py", line 310, in
s = current_session()
File "/batch/configuration/dasi/python/Get_Files_From_SharePoint_debug.py", line 95, in current_session
session.save(s_file)
File "/home/L03530558/.local/lib/python3.6/site-packages/sharepy/session.py", line 66, in save
with open(filename, mode) as file:
PermissionError: [Errno 13] Permission denied: '/~~/sp-session.pkl'

s_file is defined like this:

os.path.join(os.path.dirname(inspect.getfile(inspect.currentframe())),'sp-session.pkl')

I'm trying to force the session file, because I found out that I have multiple versions of sharepy with different users.

When I downgrade to 1.3.0 I get the results I want, but download becomes messy.

[bp-kelley]

Just to tag in here, we had the same issue in our regression tests when we upgraded sharepy to 2.0.0. 1.3.0 kept on working.

[d-english]

Same issue for our implementation. We've rolled back to 1.3.0 to fix.

[annywithy]

Hi all, I have run into the same issue with 2.0.0 but found a solution that works for me.

Essentially following the example provided by @JonathanHolvey in #4 (comment) but before the POST request to upload file, make another POST request to _api/contextinfo to obtain the FormDigestValue (parse the response like r.json()["d"]["GetContextWebInformation"]["FormDigestValue"]) and update the session with this value (e.g. s.auth.digest = [FormDigestValue].

I first rolled back to 1.3.0 but could not even authenticate. So with 2.0.0 this is the only workaround for me.
Hope it helps.

[johndecharinte]

I also had the same issue roll backed to 1.3.0 and it worked

[faytrow]

Hi all, I have run into the same issue with 2.0.0 but found a solution that works for me.

Essentially following the example provided by @JonathanHolvey in #4 (comment) but before the POST request to upload file, make another POST request to _api/contextinfo to obtain the FormDigestValue (parse the response like r.json()["d"]["GetContextWebInformation"]["FormDigestValue"]) and update the session with this value (e.g. s.auth.digest = [FormDigestValue].

I first rolled back to 1.3.0 but could not even authenticate. So with 2.0.0 this is the only workaround for me. Hope it helps.

Thank you!! That solved my issue with just a little bit of code added in.

One note regarding the code from issue #4, I did have to add a '/' to the end of the url and remove the '/' from the beginning of the Destination folder to get it to work on my end. The string concatenation did not work otherwise and I got 404 errors until swapping that around.