404 Client Error: Not Found for url: self._url("Lists")

[dHannasch]

I am trying to run the unit test suite (https://gitlab.com/dHannasch/shareplum/-/blob/gitlab-ci-yml/.gitlab-ci.yml).

On every test, when it goes to create a Site, I get an error.

Traceback (most recent call last):
  File "/builds/dHannasch/shareplum/shareplum/request_helper.py", line 17, in post
    response.raise_for_status()
  File "/usr/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: [self._url("Lists")]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/builds/dHannasch/shareplum/tests/test_site.py", line 24, in setUp
    self.site = Site(TEST_SETTINGS["site_url"], version=version, auth=auth)
  File "/builds/dHannasch/shareplum/shareplum/site.py", line 549, in Site
    return _Site365(site_url,
  File "/builds/dHannasch/shareplum/shareplum/site.py", line 406, in __init__
    super().__init__(site_url, auth, authcookie, verify_ssl, ssl_version, huge_tree, timeout)
  File "/builds/dHannasch/shareplum/shareplum/site.py", line 98, in __init__
    self.users = self.get_users()
  File "/builds/dHannasch/shareplum/shareplum/site.py", line 341, in get_users
    response = post(self._session,
  File "/builds/dHannasch/shareplum/shareplum/request_helper.py", line 20, in post
    raise ShareplumRequestError("Shareplum HTTP Post Failed", err)
shareplum.errors.ShareplumRequestError: Shareplum HTTP Post Failed : 404 Client Error: Not Found for url: [self._url("Lists")]

@isaacnorman82 Do you have any idea what could be happening? You seem knowledgeable about these requests.

What's particularly striking here is that I cannot get a 404 outside of Shareplum, even by completely skipping authentication.

python -c "from requests import post; print(post('$SHAREPOINT_SERVER_URL/sites/$SHAREPOINT_SITE_NAME//_vti_bin/lists.asmx'))"

yields a login page as HTML; that is, it returns 200 OK.
Maybe a 404 has some special meaning for a SOAP request? I don't know.

For the record, I'm pretty sure I have the Sharepoint version correct (2016), but in any case I don't...think that version should matter for this. Unless the 404 does have some special meaning.

Note that this is using #153 rather than the Office 365 authentication.

[isaacnorman82]

Just from a quick look I imagine you can't create because you don't have permissions on the site the repo owner created - I have a vague memory of hitting that myself - and then maybe the code is trying to access something after that that hasn't been created?

I can't say from the logging you included what the exactly url is that it tried to access - is the logging missing an f-string or something - why is it saying for url [self._url("Lists)] rather than the actual url? How do you know the url you tried is excactly the same?

Really the unit tests should use some shared login that has access to create or there should be an option to only run the tests that anyone can run.

[jasonrollins]

SharePlum automatically pulls the list of users when it connects to a SharePoint site. That's where you are crashing. The 404 error is a standard permission failure. As you said, your script is getting a login screen. The SharePlum request is more involved. It's actually trying to login and do things. Here is the post:

response = post(self._session, url=self._url("Lists"), headers=self._headers("GetListItems"), data=str(soap_request).encode("utf-8"), verify=self._verify_ssl, timeout=self.timeout)

It is connecting to the site and immediately pulling down the "UserInfo".

What version of SharePoint are you using? The 404 could mean your username/password is incorrect. I think you have to include the domain in the username when you are using NTLM. If your user is dHannasch then the username="DOMAIN/dHannasch".

[dHannasch]

I imagine you can't create because you don't have permissions on the site the repo owner created - I have a vague memory of hitting that myself - and then maybe the code is trying to access something after that that hasn't been created?

I have permission to edit pages on the site, and I can reach the site settings page. (Admittedly I can't be totally sure I have all permissions; you're correct that I didn't create the site myself, because, well, to be honest I don't know how to create a Sharepoint site.)

I can't say from the logging you included what the exactly url is that it tried to access - is the logging missing an f-string or something - why is it saying for url [self._url("Lists)] rather than the actual url? How do you know the url you tried is excactly the same?

Ah, sorry, I manually replaced it with self._url("Lists) because I thought that would be clearer. You wouldn't be able to access the URL, after all. I know the URL I tried is exactly the same because it did originally display the actual URL in the error message. I mean, I guess I could drop an actual request.post() into the Shareplum code itself right above this call if you're skeptical.

EDIT: I think https://github.com/jasonrollins/shareplum/pull/155/files definitively settles the question of whether the URL that gets 200 OK is the same as the URL that gets 404.

Really the unit tests should use some shared login that has access to create or there should be an option to only run the tests that anyone can run.

I mean, ideal world, but most people just don't have access to a real Sharepoint site. We can't have the tests use a shared login without, you know, actually having a Sharepoint login whose credentials are shared with the world. Money aside, I'm pretty sure that would be against the terms of service. And that's just for Office 365. While I very much don't understand Sharepoint very well, I think that non-Office-365 Sharepoint is made of separate non-communicating instances, which behave quite differently from Office 365 (looks like they're completely different Python classes), so the only way to run tests is to use people's individual credentials on individual Sharepoint instances.

The 404 error is a standard permission failure.

Ahhh.

What version of SharePoint are you using?

Sharepoint 2016.

The 404 could mean your username/password is incorrect. I think you have to include the domain in the username when you are using NTLM. If your user is dHannasch then the username="DOMAIN/dHannasch".

Oh, is this the first place the auth would fail? I was assuming that if it had gotten to the point of pulling the users, then the authorization had to be succeeding. If this is the first place the auth could fail, I kind of want to add something above this specifically to test the auth first. Something like #155.

I am using a DOMAIN\username. Admittedly I'm not 100% sure the DOMAIN is correct; I haven't figured out how to tell from poking around Sharepoint manually what the DOMAIN is. There's no place to enter a DOMAIN logging on manually through a browser. So I basically just experimented with everything that sounded likely. And now that I say that out loud, the fact that I got the same error every time should have clued me in that this was an authorization issue, huh. They can't all be correct.

(I realize that sounds strange and I should probably explain a bit. My user page has the Account showing up as i:0#.w|organizationname-dmz\dHannasch). I have no earthly idea what dmz could stand for, still less what's going on with the colon and pound at the start, so I tried organizationname by itself, ORGANIZATIONNAME by itself, dmz by itself, the whole thing including all the weird symbols, and so forth and so on. Needless to say, when I log in normally, I don't enter any of that, I just enter dHannasch. I didn't know any of that craziness existed until I started trying to set this up and learned that a DOMAIN was a thing.)

[jasonrollins]

Yes, you are getting an error on your first post. It's getting user info as part of the init process. Yes, it's a bad error. I agree with you. I've never really liked it, but I couldn't think of an alternative. If you are using Office365 you get a much better error message.

The domain thing is annoying. You are correct that you don't have to type it in when you log into the website. I wrote SharePlum back when I was using SharePoint 2013 and SharePoint 2016 sites. Which is why everything is based on SOAP. I ran into the same issue you are at first. I can't really remember how you have to specify the user name. I'm guessing your username is "organizationname-dmz\dHannasch".

Here are some things to try:
username = "organizationname-dmz\dHannasch"
username = r"organizationname-dmz\dHannasch"
username = "organizationname-dmz\dHannasch"
username = "organizationname-dmz/dHannasch"
username = "organizationname-dmz//dHannasch"
username = r"organizationname-dmz/dHannasch"

It's been too long now for me to remember how I had to specify it.

I think you can determine your domain name in the System Properties windows. Directions per Google -> "Press Windows + R keys together, type the command “sysdm. cpl” in the Run dialog box and press Enter."

Let us know what ends up working for you. It would be a good historical reference.

[dHannasch]

Well, it's definitely a single backslash we want between the DOMAIN and the username; I confirmed that by introspecting what the HttpNtlmAuth was actually parsing out as the .domain and .username. #155 is printing out the domain and username now, let me know if that's some kind of security problem.

(It also turns out capitalization doesn't matter; whatever domain you put in, it'll end up capitalized.)

I tried the domain from my own computer just to be thorough, but I think the sysdm.cpl thing is a completely different thing. I mean, the password I use to log into Sharepoint is different from the password I use to log in to my computer, you know? The user account that logs into Sharepoint is a different account on a different system with a different DOMAIN. Sharepoint doesn't know about my account on my computer, and my computer doesn't know about Sharepoint.

EDIT: for future reference, The prefix i:0#.w| before the username is just an internal code so that SharePoint can identity the type of authentication. w means windows auth, f would mean formsbased. The i in the code is to identify that it is a user. For a group, the prefix would be c instead of i.

[jasonrollins]

I did some reading and it seems my memory is a little incorrect. I've only used SharePoint 2010 and 2013, not 2016. SharePlum uses NTLM for 2010 through 2016, but I don't think 2016 supports NTLM. Here is an article I found that discussion the various authentication options. Since we are relying on requests here is the docs discussion authentication options. Maybe you can try some of the different options for requests. I'm pretty sure now that NTLM is NOT what you want. When I was using SharePoint 2010 and 2013, my username and password where the same for SharePoint and my computer. I think that is how NTLM works.

[dHannasch]

I'm confused.

On the one hand, that article you linked quite specifically says that "Windows classic mode authentication is no longer supported in SharePoint Server 2016."

On the other hand, Shareplum used to work on Sharepoint 2016 last year, right? I mean, it's specifically listed in the enumeration.

And it looks like all the people who ever succeeded were using NTLM.

https://sharepains.com/2018/01/04/sharepoint-2016-integrate-with-python-based-solutions-using-sharepoint-rest-api/

SharePoint 2016
from requests_ntlm import HttpNtlmAuth

https://docs.microsoft.com/en-us/answers/questions/142065/ways-to-accessupload-documents-to-sharepoint-sites.html

For SharePoint On-Premise(tested in SharePoint 2016 On-Premise)
from requests_ntlm import HttpNtlmAuth

It seems pretty strange and unlikely that the Sharepoint server would know or care whether your username and password were the same on Sharepoint and on your computer. I mean, you might choose to do that for convenience. But if they were actually tied together somehow, there would be no point in providing a username and password at all.

Nothing in the NTLM workflow seems to suggest that it's checking on any other computer, except of course the domain controller. https://security.stackexchange.com/questions/129832/understanding-ntlm-authentication-step-by-step