Crash with strange filename #151

borouhin · 2018-08-29T09:49:05Z

Bug Report Details

I happened to have a very strange file in my OneDrive folder. It's name was כץנלו-נעןהבצב.jpg (presumably, it was saved long ago from some buggy webmail interface).
This file was successfully uploaded to OneDrive Personal with official client. It was later downloaded with this client also without errors. However, after that, when running onedrive --synchronize --local-first the presence of this file crashed the program (although the file itself was not to be uploaded, it was already present in both local and remote dirs and didn't change).

Application and Operating System Details:

OS: Linux 4.15.0-32-generic Error when trying to upload a file with weird non printable characters present #35-Ubuntu SMP Fri Aug 10 17:58:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Application version: onedrive v2.1.2-3-g760f7fb
OneDrive Account Type: Personal
DMD or LDC compiler version: DMD64 D Compiler v2.081.2

To Reproduce
Steps to reproduce the behavior:

Create file כץנלו-נעןהבצב.jpg (perhaps, any filename with such entities will go).
Upload it to OneDrive via web interface or Windows client.
Sync with onedrive, make sure that the file got downloaded OK.
Stop onedrive.
Run onedrive --synchronize --local-first -v
See the crash.

Complete Verbose Log Output
Application Log Output:

Uploading new items of .
std.json.JSONException@std/json.d(1392): Unexpected character '<'. (Line 1:1)
<!DOCTYPE html>
<html>
    <head>
        <title>A potentially dangerous Request.RawUrl value was detected from the client (=&quot;...ахматова)/&amp;#1499;&amp;#1509;&amp;#1504...&quot;).</title>
        <meta name="viewport" content="width=device-width" />
        <style>
         body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;}
         p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
         b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
         H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
         H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
         pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
         .marker {font-weight: bold; color: black;text-decoration: none;}
         .version {color: gray;}
         .error {margin-bottom: 10px;}
         .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
         @media screen and (max-width: 639px) {
          pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
         }
         @media screen and (max-width: 479px) {
          pre { width: 280px; }
         }
        </style>
    </head>

    <body bgcolor="white">

            <span><H1>Server Error in '/' Application.<hr width=100% size=1 color=silver></H1>

            <h2> <i>A potentially dangerous Request.RawUrl value was detected from the client (=&quot;...ахматова)/&amp;#1499;&amp;#1509;&amp;#1504...&quot;).</i> </h2></span>

            <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">

            <b> Description: </b>ASP.NET has detected data in the request that is potentially dangerous because it might include HTML markup or script. The data might represent an attempt to compromise the security of your application, such as a cross-site scripting attack. If this type of input is appropriate in your application, you can include code in a web page to explicitly allow it. For more information, see http://go.microsoft.com/fwlink/?LinkID=212874.
            <br><br>

            <b> Exception Details: </b>System.Web.HttpRequestValidationException: A potentially dangerous Request.RawUrl value was detected from the client (=&quot;...ахматова)/&amp;#1499;&amp;#1509;&amp;#1504...&quot;).<br><br>

            <b>Source Error:</b> <br><br>

            <table width=100% bgcolor="#ffffcc">
               <tr>
                  <td>
                      <code>

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.</code>

                  </td>
               </tr>
            </table>

            <br>

            <b>Stack Trace:</b> <br><br>

            <table width=100% bgcolor="#ffffcc">
               <tr>
                  <td>
                      <code><pre>

[HttpRequestValidationException (0x80004005): A potentially dangerous Request.RawUrl value was detected from the client (=&quot;...ахматова)/&amp;#1499;&amp;#1509;&amp;#1504...&quot;).]
   System.Web.HttpRequest.ValidateString(String value, String collectionKey, RequestValidationSource requestCollection) +11917179
   System.Web.HttpRequest.get_RawUrl() +78
   Microsoft.Online.AggregatorService.WebService.ContextHelper.GetIncomingUrl(HttpContextBase httpContext) in X:\bt\960606\repo\src\dev\WebService\ContextHelper.cs:196
   Microsoft.Online.AggregatorService.WebService.AuthenticationModule.InitializeAndGetLogContext(HttpContextWrapper currentContextWrapper) in X:\bt\960606\repo\src\dev\WebService\AuthenticationModule.cs:306
   Microsoft.Online.AggregatorService.WebService.AuthenticationModule.Context_BeginRequest(Object sender, EventArgs e) in X:\bt\960606\repo\src\dev\WebService\AuthenticationModule.cs:98
   System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +139
   System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step) +195
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean&amp; completedSynchronously) +88
</pre></code>

                  </td>
               </tr>
            </table>

            <br>

            <hr width=100% size=1 color=silver>

            <b>Version Information:</b>&nbsp;Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.7.2623.0

            </font>

    </body>
</html>
<!--
[HttpRequestValidationException]: A potentially dangerous Request.RawUrl value was detected from the client (=&quot;...ахматова)/&amp;#1499;&amp;#1509;&amp;#1504...&quot;).
   at System.Web.HttpRequest.ValidateString(String value, String collectionKey, RequestValidationSource requestCollection)
   at System.Web.HttpRequest.get_RawUrl()
   at Microsoft.Online.AggregatorService.WebService.ContextHelper.GetIncomingUrl(HttpContextBase httpContext) in X:\bt\960606\repo\src\dev\WebService\ContextHelper.cs:line 196
   at Microsoft.Online.AggregatorService.WebService.AuthenticationModule.InitializeAndGetLogContext(HttpContextWrapper currentContextWrapper) in X:\bt\960606\repo\src\dev\WebService\AuthenticationModule.cs:line 306
   at Microsoft.Online.AggregatorService.WebService.AuthenticationModule.Context_BeginRequest(Object sender, EventArgs e) in X:\bt\960606\repo\src\dev\WebService\AuthenticationModule.cs:line 98
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
--><!--
This error page might contain sensitive information because ASP.NET is configured to show verbose error messages using &lt;customErrors mode="Off"/&gt;. Consider using &lt;customErrors mode="On"/&gt; or &lt;customErrors mode="RemoteOnly"/&gt; in production environments.-->
----------------
/home/alb/dlang/dmd-2.081.2/linux/bin64/../../src/phobos/std/json.d:726 pure @safe void std.json.parseJSON!(char[]).parseJSON(char[], int, std.json.JSONOptions).error(immutable(char)[]) [0x7274ea00]
/home/alb/dlang/dmd-2.081.2/linux/bin64/../../src/phobos/std/json.d:1067 pure @safe void std.json.parseJSON!(char[]).parseJSON(char[], int, std.json.JSONOptions).parseValue(ref std.json.JSONValue) [0x7274f5db]
/home/alb/dlang/dmd-2.081.2/linux/bin64/../../src/phobos/std/json.d:1073 pure @safe std.json.JSONValue std.json.parseJSON!(char[]).parseJSON(char[], int, std.json.JSONOptions) [0x7274e99d]
src/onedrive.d:491 std.json.JSONValue onedrive.OneDriveApi.perform() [0x7274e21d]
src/onedrive.d:326 std.json.JSONValue onedrive.OneDriveApi.get(const(char)[], bool) [0x7274d86a]
src/onedrive.d:192 std.json.JSONValue onedrive.OneDriveApi.getPathDetails(const(immutable(char)[])) [0x7274cc83]
src/sync.d:1145 void sync.SyncEngine.uploadNewFile(immutable(char)[]) [0x7275bd9f]
src/sync.d:1021 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x7275b4a4]
src/sync.d:1012 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x7275b383]
src/sync.d:1012 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x7275b383]
src/sync.d:1012 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x7275b383]
src/sync.d:1012 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x7275b383]
src/sync.d:1012 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x7275b383]
src/sync.d:1012 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x7275b383]
src/sync.d:736 void sync.SyncEngine.scanForDifferences(immutable(char)[]) [0x72759f89]
src/main.d:396 void main.performSync(sync.SyncEngine, immutable(char)[], bool, bool, bool) [0x7273d262]
src/main.d:269 _Dmain [0x7273cb13]

Screenshots
Not applicable.

Additional context
Not long before encountering this bug, I removed items.sqlite3 file to debug another issue. Maybe this somehow triggered the bug. I cannot test all the scenarios, because resyncing ~700.000 files in my OneDrive takes very long time.

Bug Report Checklist

[V] Detailed description
[V] Reproduction steps (if applicable)
[V] Verbose Log Output

The text was updated successfully, but these errors were encountered:

abraunegg · 2018-08-29T09:55:50Z

Can you send me an archived copy of the file - just need the exact name - not the data in the file.

Email the archived file to support@mynas.com.au

borouhin · 2018-08-29T09:59:41Z

I've emailed you the file. It's name is indeed exactly the same as I wrote above.

abraunegg · 2018-08-29T10:00:30Z

Thanks .. will use this for local testing

abraunegg · 2018-09-15T07:43:06Z

@borouhin
Can you help validate the following PR for this issue:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/178/head:pr178
git checkout pr178
make
make install

Local validation of PR:

Loading config ...
Using Config Dir: <redacted>
No config file found, using defaults
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: <redacted>
Initializing the Synchronization Engine ...
Account Type: personal
Default Drive ID: <redacted>
Default Root ID: <redacted>
Remaining Free Space: 112742891520
Fetching details for OneDrive Root
OneDrive Root exists in the database
Syncing changes from OneDrive ...
Applying changes of Path ID: <redacted>
Uploading differences of .
Processing root
The directory has not changed
Processing bad_files
The directory has not changed
Uploading new items of .
Skipping item - invalid name (Contains HTML ASCII Code): ./bad_files/&#1499;&#1509;&#1504;&#1500;&#1493;-&#1504;&#1506;&#1503;&#1492;&#1489;&#1510;&#1489;.jpg
Skipping item - invalid name (Contains HTML ASCII Code): ./bad_files/asdfasdf &#1503; sadfasdfasdf.doc
Skipping item - invalid name (Contains HTML ASCII Code): ./bad_files/asdfasdf &#1503 sadfasdfasdf.doc
Skipping item - invalid name (Contains HTML ASCII Code): ./bad_files/asdfasdf &#asdf sadfasdfasdf.doc
Applying changes of Path ID: <redacted>

abraunegg · 2018-09-20T19:48:16Z

@borouhin
Any update on testing this PR?

borouhin · 2018-09-20T19:53:13Z

@abraunegg, I'm sorry, but you shouldn't count on me for testing anytime soon.

I just want to keep my files in sync. It's 10:50 PM now here. My last sync session started at 06:30 AM and it's still (!) rescanning every single folder and file in my OneDrive directory. Yes, it's a lot of files. But OneDrive Windows client keeps them in sync without noticeable delays (after initial sync), Dropbox for Linux client (I moved to OneDrive from Dropbox after their announcement to drop other FS support except ext4) did the same job with the same files quickly enough not to notice it.
It looks like a fundamental design flaw in this program, which cannot be fixed by individual patches.

Every test leads to restart of sync job, and my home server is constantly loaded with onedrive process, which impacts performance of other tasks. Sure, I could run onedrive on another machine dedicated to testing with a single small folder, but not right now, not enough time, sorry.

* Update file handling to look for HTML ASCII codes which will cause uploads to fail

abraunegg · 2018-10-03T23:34:13Z

PR #178 merged

lock · 2019-01-05T22:35:43Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

borouhin mentioned this issue Aug 29, 2018

Validate filename length to conform with Linux FS limits before download #142

Closed

abraunegg added Bug Something isn't working Investigating labels Aug 29, 2018

abraunegg mentioned this issue Sep 11, 2018

Unexpected character '<'. (Line 1:1) #167

Closed

abraunegg self-assigned this Sep 15, 2018

abraunegg added PR Provided - Needs Testing Waiting on Response and removed Investigating labels Sep 15, 2018

abraunegg added this to the 2.1.3 milestone Oct 3, 2018

abraunegg added a commit that referenced this issue Oct 3, 2018

Resolve crash with strange filename (Issue #151) (#178)

14b2de8

* Update file handling to look for HTML ASCII codes which will cause uploads to fail

abraunegg closed this as completed Oct 3, 2018

abraunegg added Fixed and removed PR Provided - Needs Testing Waiting on Response labels Oct 3, 2018

lock bot locked and limited conversation to collaborators Jan 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash with strange filename #151

Crash with strange filename #151

borouhin commented Aug 29, 2018

abraunegg commented Aug 29, 2018

borouhin commented Aug 29, 2018

abraunegg commented Aug 29, 2018

abraunegg commented Sep 15, 2018

abraunegg commented Sep 20, 2018

borouhin commented Sep 20, 2018 •

edited

Loading

abraunegg commented Oct 3, 2018

lock bot commented Jan 5, 2019

Crash with strange filename #151

Crash with strange filename #151

Comments

borouhin commented Aug 29, 2018

Bug Report Details

Bug Report Checklist

abraunegg commented Aug 29, 2018

borouhin commented Aug 29, 2018

abraunegg commented Aug 29, 2018

abraunegg commented Sep 15, 2018

abraunegg commented Sep 20, 2018

borouhin commented Sep 20, 2018 • edited Loading

abraunegg commented Oct 3, 2018

lock bot commented Jan 5, 2019

borouhin commented Sep 20, 2018 •

edited

Loading