implementation of HTTP SEARCH method #1689

peysche · 2022-01-08T23:22:56Z

As of RFC 5323 (https://tools.ietf.org/html/rfc5323) a HTTP SEARCH method was proposed.

Searched in the wild for some .NET libs supporting this... nothing found ...

Will be nice if it gets implemented in RestSharp

alexeyzimarev · 2022-01-09T11:14:34Z

It's quite easy to do. As you can see, it's possible to construct a request with any method:

    static HttpMethod AsHttpMethod(Method method)
        => method switch {
            Method.Get     => HttpMethod.Get,
            Method.Post    => HttpMethod.Post,
            Method.Put     => HttpMethod.Put,
            Method.Delete  => HttpMethod.Delete,
            Method.Head    => HttpMethod.Head,
            Method.Options => HttpMethod.Options,
#if NETSTANDARD
            Method.Patch => new HttpMethod("PATCH"),
#else
            Method.Patch   => HttpMethod.Patch,
#endif
            Method.Merge => new HttpMethod("MERGE"),
            Method.Copy  => new HttpMethod("COPY"),
            _            => throw new ArgumentOutOfRangeException()
        };

Also, a change is needed to tell RestSharp that SEARCH requests support a body.

You can make a PR to add it, should not be a lot of work.

alexeyzimarev · 2022-01-09T11:21:27Z

I was wrong about the last statement, RestSharp would add content body to any request, so it's up to the server.

I added the verb, but I don't know how to test it as Kestrel doesn't support this method.

alexeyzimarev · 2022-01-09T11:32:09Z

It's in 107.0.2, without anything more than adding the verb itself.

peysche · 2022-01-09T11:32:24Z

Hi,

Thx for investigation, SEARCH method will be fine if it is Implemented, but I solved it for my Solution on an other way

As I like to sear Files on NextCloud, and NC is already supporting the SEARCH method in his WEBDAV implementation
I currently solved it with an simple WebRequest

ICredentials WebDavCredential;
SecurityProvider SecProvider = new SecurityProvider();
WebDavCredential = new NetworkCredential(Request.Configuration.WebDavUser, SecProvider.DecryptString(Request.Configuration.WebDavPassword));

string stringData = @"<?xml version=""1.0"" encoding=""UTF-8""?>
                        <d:searchrequest xmlns:d=""DAV:""
                            xmlns:oc=""http://owncloud.org/ns"">
                            <d:basicsearch>
                                <d:select>
                                    <d:prop>
                                        <d:getlastmodified/>
                                        <d:getetag/>
                                        <d:getcontenttype/>
                                        <d:getcontentlength/>
                                        <oc:id/>
                                        <oc:fileid/>
                                        <oc:size/>
                                        <d:getetag/>
                                    </d:prop>
                                </d:select>
                                <d:from>
                                    <d:scope>
                                        <d:href>/LOCATION_TO_SEARCH_FOR</d:href>
                                        <d:depth>infinity</d:depth>
                                    </d:scope>
                                </d:from>
                                <d:where>
                                    <d:like>
                                        <d:prop>
                                            <d:displayname/>
                                        </d:prop>
                                        <d:literal>%SEARCH_STRING%</d:literal>
                                    </d:like>
                                </d:where>
                            </d:basicsearch>
                        </d:searchrequest> ";

var data = Encoding.Default.GetBytes(stringData)

WebRequest request = WebRequest.Create("https://my.nextcloudserver.com/remote.php/dav");
request.Credentials = WebDavCredential;
request.Method = "SEARCH";
request.ContentType = "text/xml";
request.ContentLength = data.Length;

Stream newStream = request.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();

WebResponse response = request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();

response.Close();

Maybe it will help someone to fix the same "Problem"

THX & Regards

peysche · 2022-01-09T11:34:39Z

I will Test 107.0.2 with NextCloud !

THX

peysche · 2022-01-09T15:15:51Z

Hi,

Looks like adding only Search is not doing the Trick..

With 107.0.2 I see the Search Method, but implementing it on the similar way I did with the WebRequest above I only get BadRequest as answer.

                                            <d:searchrequest xmlns:d=""DAV:""
                                                xmlns:oc=""http://owncloud.org/ns"">
                                                <d:basicsearch>
                                                    <d:select>
                                                        <d:prop>
                                                            <d:getlastmodified/>
                                                            <d:getetag/>
                                                            <d:getcontenttype/>
                                                            <d:getcontentlength/>
                                                            <oc:id/>
                                                            <oc:fileid/>
                                                            <oc:size/>
                                                            <d:getetag/>
                                                        </d:prop>
                                                    </d:select>
                                                    <d:from>
                                                        <d:scope>
                                                            <d:href>LOCATION_TO_SEARCH_IN</d:href>
                                                            <d:depth>infinity</d:depth>
                                                        </d:scope>
                                                    </d:from>
                                                    <d:where>
                                                        <d:like>
                                                            <d:prop>
                                                                <d:displayname/>
                                                            </d:prop>
                                                            <d:literal>%FILE_TO_SEARCH_FOR%</d:literal>
                                                        </d:like>
                                                    </d:where>
                                                </d:basicsearch>
                                            </d:searchrequest> "; 

            var data = Encoding.Default.GetBytes(stringData); 

            var client = new RestClient("https://my.nextcloudserver.com/remote.php/dav");

            client.Authenticator = new HttpBasicAuthenticator("USER", "PASSWORD");

            var request = new RestRequest();

            request.Method = Method.Search;

            request.AddBody(stringData);

            var restResponse = client.ExecuteAsync(request).GetAwaiter().GetResult();

            Console.WriteLine(restResponse.Content);

I tryed several ways to handover the Body, but no luck.

Also WebDav methods PROPFIND, MKCOL, MOVE, LOCK, UNLOCK , are not implemented, so I think currently RestSharp is not the Lib to use if you deal with WebDAV.

I will go forward and use currently standard WebRequest, or I will use PowerShell 7... the Invoke-RestMethod will do it like a charm, an also the handling of XML results is even a nice thing in PowerShell...

Never the less Thx for the try on short notice.

Regards & Thx

alexeyzimarev · 2022-01-09T15:52:30Z

Yes, because you need to add the string, not bytes, using AddBody. It will then use StringContent. In your case it will be messed up.

I suggest using https://github.com/BSiLabs/HttpTracer to inspect the request being made with RestSharp. You can check my snipper here #885 (comment)

alexeyzimarev · 2022-01-09T15:54:50Z

You also don't set the XML content type, so I guess it won't ever work.

The best solution for you is to create a class that serializes to the XML you need. Then, use AddXmlBody and it will do everything for you.

If you want to post XML string as a body, you need to use AddBody(stringData, ContentType.Xml)

alexeyzimarev · 2022-01-09T16:02:16Z

Also, in the last alpha you can use AddStringBody(stringData, DataFormat.Xml)

alexeyzimarev · 2022-01-09T16:06:11Z

I honestly don't know why you decided to send bytes array to AddBody. The method is properly documented, and it says "string", not "byte array".

RestSharp/src/RestSharp/Request/RestRequestExtensions.cs

Lines 295 to 304 in 1556eed

    
           /// <summary> 
        
           /// Adds a body parameter to the request 
        
           /// </summary> 
        
           /// <param name="request">Request instance</param> 
        
           /// <param name="obj">Object to be used as the request body, or string for plain content</param> 
        
           /// <param name="contentType">Optional: content type</param> 
        
           /// <returns></returns> 
        
           /// <exception cref="ArgumentException">Thrown if request body type cannot be resolved</exception> 
        
           /// <remarks>This method will try to figure out the right content type based on the request data format and the provided content type</remarks> 
        
           public static RestRequest AddBody(this RestRequest request, object obj, string? contentType = null) {

peysche · 2022-01-09T16:54:44Z

I did not send the Byte Array..

var data = Encoding.Default.GetBytes(stringData);

that's in because I tested a lot, I only used the plain string variable stringData

request.AddBody(stringData); <-- stringData
I also did an AddBody with an XmlDocument so I converted stringData before to an XmlDocument

and played with

request.AddHeader("Content-Type", "text/xml");
request.RequestFormat = DataFormat.Xml;

I will try AddStringBody later, have to checkout RestsSharp first, i use the NugetPackage currently.

by the way https://github.com/BSiLabs/HttpTracer is cool for tests ...

peysche · 2022-01-09T19:41:05Z

Its me again ..

request.AddStringBody(stringData, contentType: "text/xml");

did the Trick,... but ...

I got this Exception in RestResponse.cs

System.InvalidOperationException
  HResult=0x80131509
  Message=The character set provided in ContentType is invalid. Cannot read content as string using an invalid character set.
  Source=mscorlib
  StackTrace:
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at RestSharp.RestResponse.<>c__DisplayClass0_0.<<FromHttpResponse>g__GetDefaultResponse|0>d.MoveNext() in C:\Users\JohnDoe\source\repos\RestSharp\src\RestSharp\Response\RestResponse.cs:line 98
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at RestSharp.RestResponse.<FromHttpResponse>d__0.MoveNext() in C:\Users\JohnDoe\source\repos\RestSharp\src\RestSharp\Response\RestResponse.cs:line 70
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at RestSharp.RestClient.<ExecuteAsync>d__0.MoveNext() in C:\Users\JohnDoe\source\repos\RestSharp\src\RestSharp\RestClient.Async.cs:line 30
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ConsoleApp1.Program.test() in C:\Users\JohnDoe\source\repos\ConsoleApp1\ConsoleApp1\Program.cs:line 88
   at ConsoleApp1.Program.Main(String[] args) in C:\Users\JohnDoe\source\repos\ConsoleApp1\ConsoleApp1\Program.cs:line 24

  This exception was originally thrown at this call stack:
    [External Code]

Inner Exception 1:
ArgumentException: utf-8 is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method.
Parameter name: name

It looks like some encoding can not be handled with the current Implementation

I changed around Line 100 in RestResponse.cs this, and it looks like the SEARCH methode is working now and delivering a result

#if NETSTANDARD            

                bytes   = await httpResponse.Content.ReadAsByteArrayAsync();
                content = Encoding.UTF8.GetString(bytes);
                //content = await httpResponse.Content.ReadAsStringAsync();
# else
                bytes = await httpResponse.Content.ReadAsByteArrayAsync(cancellationToken);
                content = Encoding.UTF8.GetString(bytes);
                //content = await httpResponse.Content.ReadAsStringAsync(cancellationToken);
#endif

I do not know if it is like you will do it, but it works ...

alexeyzimarev · 2022-01-09T19:43:26Z

Sorry I got confused by the first line of your snippet!

var data = Encoding.Default.GetBytes(stringData);

I don't think it's related to the SEARCH method. I can't the code as you did as I can't assume it's always UTF8 that is returned by the server. It seems like the server returns something that HttpResponse doesn't understand. Do you have a snapshot of the response? It's interesting to see what the server returns.

alexeyzimarev · 2022-01-09T19:55:22Z

Also, I am wondering what's your .NET version

peysche · 2022-01-09T20:52:04Z

Target Framework is 4.8 VS 2022 latest patched

Will try to get an RAW response from Nextcloud WebDav with the Search.
I think you will be probably right with

It seems like the server returns something that HttpResponse doesn't understand

Nextcloud's WebDav implementation is somehow not every time fine, and regarding issues on GitHub somehow sometimes buggy...

by the way, I also added the PROPFIND method to the Enum.cs and RestClient.Async.cs

Works like it should

Here is the httpResponse Object dumped in the Immediate Window


{StatusCode: 207, ReasonPhrase: 'Multi-Status', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Pragma: no-cache
  Content-Security-Policy: default-src 'none';
  Cache-Control: no-store, must-revalidate, no-cache
  Date: Sun, 09 Jan 2022 20:33:58 GMT
  Set-Cookie: oc_sessionPassphrase=<hidden>; path=/; secure; HttpOnly; SameSite=Lax
  Set-Cookie: __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
  Set-Cookie: __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
  Set-Cookie: ocdlkbru62ei=c68cgegpc2mca3k6pk7cf67s3s; path=/; secure; HttpOnly; SameSite=Lax
  Set-Cookie: cookie_test=test; expires=Sun, 09-Jan-2022 21:33:58 GMT; Max-Age=3600
  Server: Apache
  Referrer-Policy: no-referrer
  X-Content-Type-Options: nosniff
  X-Download-Options: noopen
  X-Frame-Options: SAMEORIGIN
  X-Permitted-Cross-Domain-Policies: none
  X-Robots-Tag: none
  X-XSS-Protection: 1; mode=block
  Upgrade: h2
  Connection: Upgrade
  Connection: Keep-Alive
  Strict-Transport-Security: max-age=15552000;includeSubdomains
  Keep-Alive: timeout=5, max=100
  Transfer-Encoding: chunked
  Expires: Thu, 19 Nov 1981 08:52:00 GMT
  Content-Type: application/xml; charset="utf-8"
}}
    Content: {System.Net.Http.StreamContent}
    Headers: {Pragma: no-cache
Content-Security-Policy: default-src 'none';
Cache-Control: no-store, must-revalidate, no-cache
Date: Sun, 09 Jan 2022 20:33:58 GMT
Set-Cookie: oc_sessionPassphrase=<hidden>; path=/; secure; HttpOnly; SameSite=Lax, __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax, __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict, ocdlkbru62ei=c68cgegpc2mca3k6pk7cf67s3s; path=/; secure; HttpOnly; SameSite=Lax, cookie_test=test; expires=Sun, 09-Jan-2022 21:33:58 GMT; Max-Age=3600
Server: Apache
Referrer-Policy: no-referrer
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: none
X-XSS-Protection: 1; mode=block
Upgrade: h2
Connection: Upgrade, Keep-Alive
Strict-Transport-Security: max-age=15552000;includeSubdomains
Keep-Alive: timeout=5, max=100
Transfer-Encoding: chunked
}
    IsSuccessStatusCode: true
    ReasonPhrase: "Multi-Status"
    RequestMessage: {Method: SEARCH, RequestUri: 'https://my.nextcloudserver.com/remote.php/dav', Version: 1.1, Content: System.Net.Http.StringContent, Headers:
{
  Authorization: Basic V2ViU2VydmVyOk15V2U4RGF2KHJlZCE=
  Accept: application/json, text/json, text/x-json, text/javascript, *+json, application/xml, text/xml, *+xml, *
  User-Agent: RestSharp/0.0.0.0
  Content-Type: text/xml; charset=utf-8
  Content-Length: 2297
}}
    StatusCode: 207
    Version: {1.1}

also in the Immediate Window the dump of httpResponse.Content.ReadAsByteArrayAsync()

?httpResponse.Content.ReadAsByteArrayAsync()
Id = 23, Status = RanToCompletion, Method = "{null}", Result = "System.Byte[]"
    AsyncState: null
    CancellationPending: false
    CreationOptions: None
    Exception: null
    Id: 23
    Result: {byte[1156]}
    Status: RanToCompletion

and the httpResponse.Content.ReadAsStringAsync()

?httpResponse.Content.ReadAsStringAsync()
Id = 26, Status = Faulted, Method = "{null}", Result = "{Not yet computed}"
    AsyncState: null
    CancellationPending: false
    CreationOptions: None
    Exception: Count = 1
    Id: 26
    Result: null
    Status: Faulted

and the Encoding.UTF8.GetString(bytes)


?Encoding.UTF8.GetString(bytes)
"<?xml version=\"1.0\"?>\n<d:multistatus xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\" xmlns:oc=\"http://owncloud.org/ns\" xmlns:nc=\"http://nextcloud.org/ns\"><d:response><d:href>/remote.php/dav/files/User/Location/691/20102x.jpg</d:href><d:propstat><d:prop><d:getlastmodified>Fri, 07 Feb 2020 08:04:13 GMT</d:getlastmodified><d:getetag>&quot;7b8b29e45f90cfe04b11e7544b302210&quot;</d:getetag><d:getcontenttype>image/jpeg</d:getcontenttype><d:getcontentlength>189957</d:getcontentlength><oc:id>02082699ocdlkbru62ei</oc:id><oc:fileid>2082699</oc:fileid><oc:size>189957</oc:size></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/files/User/Location/691/20102.JPG</d:href><d:propstat><d:prop><d:getlastmodified>Sat, 08 Jan 2022 14:34:35 GMT</d:getlastmodified><d:getetag>&quot;62f212d79969787f16c0925c83fa16a4&quot;</d:getetag><d:getcontenttype>image/jpeg</d:getcontenttype><d:getcontentlength>338684</d:getcontentlength><oc:id>04874552ocdlkbru62ei</oc:id>
<oc:fileid>4874552</oc:fileid><oc:size>338684</oc:size></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response></d:multistatus>\n"

the error in the Exception from my old Post looks like well known in the World if you Google it ...

one solution i found was this, but did not help...

Run dotnet.exe add package System.Text.Encoding.CodePages
Add using System.Text; and Encoding.RegisterProvider(CodePagesEncodingProvider.Instance)

Stragen thing ...

alexeyzimarev · 2022-01-09T20:56:39Z

It is weird because I just closed another issue where people complained that StringContent sets the charset to utf-8 in the Content-Type header :) Really weird. I don't use .NET Framework, so can't say for sure what's wrong. Why don't you try with .NET 6 instead? Any reason you prefer using .NET Framework?

peysche · 2022-01-09T21:18:55Z

I can not use a newer Framework here, its an really old Application, and currently I am ongoing to midgrade this to an completely other solution. And therefore I has to Patchwork between the 2 Solutions ...

The final Solution is running on .NET 6 an consuming Files from an Nextcloud 22 Instance.
And some Files need also to be accessible from the old System and so I need to grap Files via WebDav with the old System from the Nextcloud Instance.

And I like to spend as less as possible time in the old Application, but as usual you can see in this case the most tricky and time consuming things are those stupid things most of the time caused by some strange Implementations done by M$ ...

alexeyzimarev · 2022-01-10T11:06:24Z

I added a wrapper to get a response string from raw response bytes instead of relying on ReadAsStringAsync. If it can't figure out the encoding, it will use UTF-8. You can try the latest preview, I am quite sure it will work fine.

peysche · 2022-01-10T11:46:37Z

Hi,

Proofed it works ! THX

Maybe you can also add Methodes

PROPFIND, PROPPATCH, LOCK, MKCOL, MOVE, UNLOCK, TRACE

To on of the next Previews ..

Thx

alexeyzimarev · 2022-01-10T11:50:54Z

I'd say someone can add it :) It's literally one line of code for each method.

alexeyzimarev added feature-request good first issue labels Jan 9, 2022

alexeyzimarev closed this as completed Jan 10, 2022

freyes-lef mentioned this issue Aug 11, 2023

is possible add method propfind to enum asMethod #2124

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implementation of HTTP SEARCH method #1689

implementation of HTTP SEARCH method #1689

peysche commented Jan 8, 2022

alexeyzimarev commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022

peysche commented Jan 9, 2022

peysche commented Jan 9, 2022

peysche commented Jan 9, 2022 •

edited

Loading

alexeyzimarev commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022 •

edited

Loading

alexeyzimarev commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022

peysche commented Jan 9, 2022 •

edited

Loading

peysche commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022 •

edited

Loading

alexeyzimarev commented Jan 9, 2022

peysche commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022

peysche commented Jan 9, 2022

alexeyzimarev commented Jan 10, 2022

peysche commented Jan 10, 2022

alexeyzimarev commented Jan 10, 2022

implementation of HTTP SEARCH method #1689

implementation of HTTP SEARCH method #1689

Comments

peysche commented Jan 8, 2022

alexeyzimarev commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022

peysche commented Jan 9, 2022

peysche commented Jan 9, 2022

peysche commented Jan 9, 2022 • edited Loading

alexeyzimarev commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022 • edited Loading

alexeyzimarev commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022

peysche commented Jan 9, 2022 • edited Loading

peysche commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022 • edited Loading

alexeyzimarev commented Jan 9, 2022

peysche commented Jan 9, 2022

alexeyzimarev commented Jan 9, 2022

peysche commented Jan 9, 2022

alexeyzimarev commented Jan 10, 2022

peysche commented Jan 10, 2022

alexeyzimarev commented Jan 10, 2022

peysche commented Jan 9, 2022 •

edited

Loading

alexeyzimarev commented Jan 9, 2022 •

edited

Loading

peysche commented Jan 9, 2022 •

edited

Loading

alexeyzimarev commented Jan 9, 2022 •

edited

Loading