New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance / crash when ~/Documents folder contains lot of files / folders #405

Open
zrzka opened this Issue Sep 22, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@zrzka
Collaborator

zrzka commented Sep 22, 2017

Check this topic. How to reproduce?

Copy Pythonista.app bundle content (via StaSh, everything) into to ~/Documents. Enough for reproduction. Now, tap on + in the UI and select Empty Script or New Folder. It takes up to 5s till the second screen appears (folders tree). It's the latest iPad Pro.

Same issue has dgelessus and shaun-h. Shaun has even worse issue, crash, ... and I assume it's because it consumes all memory on his device?

@shaun-h

This comment has been minimized.

Show comment
Hide comment
@shaun-h

shaun-h Sep 22, 2017

I am on a iPad Pro, I have deleted a lot of files as a test and mine has stopped crashing, now it is just slow around 18 seconds to load the folder tree.

shaun-h commented Sep 22, 2017

I am on a iPad Pro, I have deleted a lot of files as a test and mine has stopped crashing, now it is just slow around 18 seconds to load the folder tree.

@zrzka

This comment has been minimized.

Show comment
Hide comment
@zrzka

zrzka Sep 22, 2017

Collaborator

Trying to simulate crash, not happened yet, but this ...

#!python3

import os
import shutil


ROOT = os.path.expanduser('~/Documents/tree')
COUNT = 100
MAX_DEPTH = 3


def make_dirs(parent, depth=0):
    if depth == MAX_DEPTH:
        return
        
    if depth <= 3:
        print(parent, depth)
        
    for i in range(0, COUNT):
        dir = os.path.join(parent, str(i))
        os.mkdir(dir)
        make_dirs(dir, depth + 1)

try:
    shutil.rmtree(ROOT)
except:
    pass
os.mkdir(ROOT)
make_dirs(ROOT)

... and + - New Folder takes 1m21s to display folder tree.

Collaborator

zrzka commented Sep 22, 2017

Trying to simulate crash, not happened yet, but this ...

#!python3

import os
import shutil


ROOT = os.path.expanduser('~/Documents/tree')
COUNT = 100
MAX_DEPTH = 3


def make_dirs(parent, depth=0):
    if depth == MAX_DEPTH:
        return
        
    if depth <= 3:
        print(parent, depth)
        
    for i in range(0, COUNT):
        dir = os.path.join(parent, str(i))
        os.mkdir(dir)
        make_dirs(dir, depth + 1)

try:
    shutil.rmtree(ROOT)
except:
    pass
os.mkdir(ROOT)
make_dirs(ROOT)

... and + - New Folder takes 1m21s to display folder tree.

@zrzka

This comment has been minimized.

Show comment
Hide comment
@zrzka

zrzka Sep 22, 2017

Collaborator

COUNT = 200 & MAX_DEPTH = 3 crashes after ~2 minutes.

Collaborator

zrzka commented Sep 22, 2017

COUNT = 200 & MAX_DEPTH = 3 crashes after ~2 minutes.

@zrzka

This comment has been minimized.

Show comment
Hide comment
@zrzka

zrzka Sep 22, 2017

Collaborator

There's no crash report, just lot of events of cpu usage type ...

Date/Time:       2017-09-23 00:36:05.713876 +0200
OS Version:      iPhone OS 11.0 (Build 15A372)
Architecture:    arm64
Report Version:  19

Command:         Pythonista3
Path:            /private/var/containers/Bundle/Application/0425FFD1-ED38-4C20-A5FC-39766B5FAA73/Pythonista3.app/Pythonista3
Version:         3.1.1 (311009)
Parent:          launchd [1]
PID:             2692

Event:           cpu usage
CPU:             144s cpu time over 178 seconds (81% cpu average), exceeding limit of 80% cpu over 180 seconds
Action taken:    none
Duration:        177.84s
Steps:           69

Hardware model:  iPad7,2
Active cpus:     3

... but, these lines ...

kernel[0] : EXC_RESOURCE -> Pythonista3[2776] exceeded mem limit: ActiveHard 3072 MB (fatal)
kernel[0] : 399243.947 memorystatus: killing_specific_process pid 2776 [Pythonista3] (per-process-limit 10) - memorystatus_available_pages: 42511
...
ReportCrash(CrashReporterSupport)[2782] : report not saved because it is non-actionable (ie: extension hangs or internal-only)

... and full mess around them ...

----> Sep 23 01:15:32 Roberts-iPad kernel[0] <Notice>: EXC_RESOURCE -> Pythonista3[2776] exceeded mem limit: ActiveHard 3072 MB (fatal)
Sep 23 01:15:32 Roberts-iPad kernel[0] <Notice>: 399243.947 memorystatus: killing_specific_process pid 2776 [Pythonista3] (per-process-limit 10) - memorystatus_available_pages: 42511
Sep 23 01:15:32 Roberts-iPad SpringBoard(KeyboardArbiter)[56] <Error>: HW kbd: Failed to set (null) as keyboard focus
Sep 23 01:15:32 Roberts-iPad callservicesd(CallKit)[128] <Notice>: Connection invalidated for call controller host connection: <private>
Sep 23 01:15:32 Roberts-iPad UserEventAgent(MemoryMonitor)[23] <Notice>: kernel jetsam snapshot note received
Sep 23 01:15:32 Roberts-iPad ReportCrash(CrashReporterSupport)[335] <Notice>: Saved type '298(298)' report (18 of max 25) at /var/mobile/Library/Logs/CrashReporter/JetsamEvent-2017-09-23-011532.ips
Sep 23 01:15:32 Roberts-iPad backboardd(BaseBoard)[61] <Error>: Unable to get short BSD proc info for 2776: No such process
Sep 23 01:15:32 Roberts-iPad backboardd(BaseBoard)[61] <Error>: Unable to get proc info for 2776: Undefined error: 0
Sep 23 01:15:32 Roberts-iPad ReportCrash(CrashReporterSupport)[2782] <Notice>: Trying to create CR directory structure as root
Sep 23 01:15:32 Roberts-iPad ReportCrash(CrashReporterSupport)[2782] <Notice>: cr_update: <private>
Sep 23 01:15:32 Roberts-iPad ReportCrash(CrashReporterSupport)[2782] <Notice>: cr_update: <private>
Sep 23 01:15:32 Roberts-iPad ReportCrash[2782] <Notice>: Formulating report for corpse[2776] Pythonista3
Sep 23 01:15:32 Roberts-iPad ReportCrash(MobileCoreServices)[2782] <Notice>: notify_register_check() failed with error 1000000
----> Sep 23 01:15:32 Roberts-iPad ReportCrash(CrashReporterSupport)[2782] <Notice>: report not saved because it is non-actionable (ie: extension hangs or internal-only)
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Port death watcher fired.
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: Server invalidated <BKProcessAssertion: 0x125e71680; "Resume" (activation:inf); id:\M-b\M^@\M-&DE66E056491F>
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: Client relinquished <BKProcessAssertion: 0x125e71680; "Resume" (activation:inf); id:\M-b\M^@\M-&DE66E056491F>
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Ignoring assertion remove, because we are terminated or pending termination
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: Process exited: <BKProcess: 0x125d22a40; Pythonista3; com.omz-software.Pythonista3; pid: 2776; agency: Application; visibility: foreground; task: none; hostpid: 56>
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Invalidating...
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] dump all assertions HWM:2 (CPUMON check): {
    <BKProcessAssertion: 0x125e71680; "Resume" (activation:inf); id:\M-b\M^@\M-&DE66E056491F> [active]
}
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Got exit context: <BKSProcessExitContext: 0x125d18650; reason: jetsam>
Sep 23 01:15:32 Roberts-iPad SpringBoard(FrontBoard)[56] <Notice>: <FBApplicationProcess: 0x15852b590; Pythonista3 (com.omz-software.Pythonista3); pid: 2776> was killed by jetsam.
Sep 23 01:15:32 Roberts-iPad SpringBoard(FrontBoard)[56] <Notice>: [FBProcessManager] Removing: <FBApplicationProcess: 0x15852b590; Pythonista3 (com.omz-software.Pythonista3); pid: 2776>
Sep 23 01:15:32 Roberts-iPad SpringBoard[56] <Notice>: Process exited: <FBApplicationProcess: 0x15852b590; Pythonista3 (com.omz-software.Pythonista3); pid: -1> -> <FBApplicationProcessExitContext: 0x1c923f2a0; exitReason: jetsam; terminationReason: (none)>
Sep 23 01:15:32 Roberts-iPad SpringBoard[56] <Notice>: Application process state changed for com.omz-software.Pythonista3: (null)
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: Deleted job with label: UIKitApplication:com.omz-software.Pythonista3[0x253c][67]
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Terminating because the job-submitter has disconnected.
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Deleted launchd job with label: UIKitApplication:com.omz-software.Pythonista3[0x253c][67]
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Invalidation complete.
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: Checking for deferred bootstrap request for com.omz-software.Pythonista3
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Removing client: <BKProcessInfoServerClient: 0x125e3f090; pid: 56>
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] No clients remain.
Sep 23 01:15:32 Roberts-iPad CommCenter[81] <Notice>: #I BundleID: <private> is no longer a foreground app
Sep 23 01:15:32 Roberts-iPad CommCenter[81] <Notice>: #I BundleID: <private> is no longer a foreground app
Sep 23 01:15:32 Roberts-iPad SpringBoard(WiFiPicker)[56] <Notice>: WIFI PICKER [com.omz-software.Pythonista3]: isProcessLaunch: 0,    isForegroundActivation: 0,     isForegroundDeactivation: 1
Sep 23 01:15:32 Roberts-iPad mediaserverd(CoreMedia)[27] <Notice>: -CMSessionMgr- CMSessionMgrHandleApplicationStateChange: CMSession: Client com.omz-software.Pythonista3 with pid '2776' is now Terminated. Background entitlement: NO
Sep 23 01:15:32 Roberts-iPad analyticsd[2208] <Notice>: persist purged: <private>
Sep 23 01:15:32 Roberts-iPad SpringBoard[56] <Notice>: Running <SBAppExitedWorkspaceTransaction: 0x153db7420> for transition request:
<SBMainWorkspaceTransitionRequest: 0x1c10fc500; display: Main; source: Unspecified> {
    applicationContext = <SBWorkspaceApplicationSceneTransitionContext: 0x1c0347dd0; background: NO> entities = {
        SBLayoutRolePrimary = <SBHomeScreenEntity: 0x1c066b540; ID: com.apple.SpringBoard.workspace-entity.builtin.home-screen; layoutRole: primary>;
    };
}
Sep 23 01:15:32 Roberts-iPad SpringBoard[56] <Notice>: Not animating wallpaper orientation to landscapeRight (3) in 0.00 because it's already in that orientation

... are interesting. That's new, AFAIK, there was an OOM crash when app did consume lot of memory. Maybe something new, no idea. Anyway, killed by the system.

P.S. Great times few years ago when devs were not putting stupid messages about not animating wallpaper into the system log :( How is anyone supposed to find anything in this mess? :(

Collaborator

zrzka commented Sep 22, 2017

There's no crash report, just lot of events of cpu usage type ...

Date/Time:       2017-09-23 00:36:05.713876 +0200
OS Version:      iPhone OS 11.0 (Build 15A372)
Architecture:    arm64
Report Version:  19

Command:         Pythonista3
Path:            /private/var/containers/Bundle/Application/0425FFD1-ED38-4C20-A5FC-39766B5FAA73/Pythonista3.app/Pythonista3
Version:         3.1.1 (311009)
Parent:          launchd [1]
PID:             2692

Event:           cpu usage
CPU:             144s cpu time over 178 seconds (81% cpu average), exceeding limit of 80% cpu over 180 seconds
Action taken:    none
Duration:        177.84s
Steps:           69

Hardware model:  iPad7,2
Active cpus:     3

... but, these lines ...

kernel[0] : EXC_RESOURCE -> Pythonista3[2776] exceeded mem limit: ActiveHard 3072 MB (fatal)
kernel[0] : 399243.947 memorystatus: killing_specific_process pid 2776 [Pythonista3] (per-process-limit 10) - memorystatus_available_pages: 42511
...
ReportCrash(CrashReporterSupport)[2782] : report not saved because it is non-actionable (ie: extension hangs or internal-only)

... and full mess around them ...

----> Sep 23 01:15:32 Roberts-iPad kernel[0] <Notice>: EXC_RESOURCE -> Pythonista3[2776] exceeded mem limit: ActiveHard 3072 MB (fatal)
Sep 23 01:15:32 Roberts-iPad kernel[0] <Notice>: 399243.947 memorystatus: killing_specific_process pid 2776 [Pythonista3] (per-process-limit 10) - memorystatus_available_pages: 42511
Sep 23 01:15:32 Roberts-iPad SpringBoard(KeyboardArbiter)[56] <Error>: HW kbd: Failed to set (null) as keyboard focus
Sep 23 01:15:32 Roberts-iPad callservicesd(CallKit)[128] <Notice>: Connection invalidated for call controller host connection: <private>
Sep 23 01:15:32 Roberts-iPad UserEventAgent(MemoryMonitor)[23] <Notice>: kernel jetsam snapshot note received
Sep 23 01:15:32 Roberts-iPad ReportCrash(CrashReporterSupport)[335] <Notice>: Saved type '298(298)' report (18 of max 25) at /var/mobile/Library/Logs/CrashReporter/JetsamEvent-2017-09-23-011532.ips
Sep 23 01:15:32 Roberts-iPad backboardd(BaseBoard)[61] <Error>: Unable to get short BSD proc info for 2776: No such process
Sep 23 01:15:32 Roberts-iPad backboardd(BaseBoard)[61] <Error>: Unable to get proc info for 2776: Undefined error: 0
Sep 23 01:15:32 Roberts-iPad ReportCrash(CrashReporterSupport)[2782] <Notice>: Trying to create CR directory structure as root
Sep 23 01:15:32 Roberts-iPad ReportCrash(CrashReporterSupport)[2782] <Notice>: cr_update: <private>
Sep 23 01:15:32 Roberts-iPad ReportCrash(CrashReporterSupport)[2782] <Notice>: cr_update: <private>
Sep 23 01:15:32 Roberts-iPad ReportCrash[2782] <Notice>: Formulating report for corpse[2776] Pythonista3
Sep 23 01:15:32 Roberts-iPad ReportCrash(MobileCoreServices)[2782] <Notice>: notify_register_check() failed with error 1000000
----> Sep 23 01:15:32 Roberts-iPad ReportCrash(CrashReporterSupport)[2782] <Notice>: report not saved because it is non-actionable (ie: extension hangs or internal-only)
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Port death watcher fired.
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: Server invalidated <BKProcessAssertion: 0x125e71680; "Resume" (activation:inf); id:\M-b\M^@\M-&DE66E056491F>
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: Client relinquished <BKProcessAssertion: 0x125e71680; "Resume" (activation:inf); id:\M-b\M^@\M-&DE66E056491F>
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Ignoring assertion remove, because we are terminated or pending termination
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: Process exited: <BKProcess: 0x125d22a40; Pythonista3; com.omz-software.Pythonista3; pid: 2776; agency: Application; visibility: foreground; task: none; hostpid: 56>
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Invalidating...
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] dump all assertions HWM:2 (CPUMON check): {
    <BKProcessAssertion: 0x125e71680; "Resume" (activation:inf); id:\M-b\M^@\M-&DE66E056491F> [active]
}
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Got exit context: <BKSProcessExitContext: 0x125d18650; reason: jetsam>
Sep 23 01:15:32 Roberts-iPad SpringBoard(FrontBoard)[56] <Notice>: <FBApplicationProcess: 0x15852b590; Pythonista3 (com.omz-software.Pythonista3); pid: 2776> was killed by jetsam.
Sep 23 01:15:32 Roberts-iPad SpringBoard(FrontBoard)[56] <Notice>: [FBProcessManager] Removing: <FBApplicationProcess: 0x15852b590; Pythonista3 (com.omz-software.Pythonista3); pid: 2776>
Sep 23 01:15:32 Roberts-iPad SpringBoard[56] <Notice>: Process exited: <FBApplicationProcess: 0x15852b590; Pythonista3 (com.omz-software.Pythonista3); pid: -1> -> <FBApplicationProcessExitContext: 0x1c923f2a0; exitReason: jetsam; terminationReason: (none)>
Sep 23 01:15:32 Roberts-iPad SpringBoard[56] <Notice>: Application process state changed for com.omz-software.Pythonista3: (null)
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: Deleted job with label: UIKitApplication:com.omz-software.Pythonista3[0x253c][67]
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Terminating because the job-submitter has disconnected.
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Deleted launchd job with label: UIKitApplication:com.omz-software.Pythonista3[0x253c][67]
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Invalidation complete.
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: Checking for deferred bootstrap request for com.omz-software.Pythonista3
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] Removing client: <BKProcessInfoServerClient: 0x125e3f090; pid: 56>
Sep 23 01:15:32 Roberts-iPad assertiond[67] <Notice>: [Pythonista3:2776] No clients remain.
Sep 23 01:15:32 Roberts-iPad CommCenter[81] <Notice>: #I BundleID: <private> is no longer a foreground app
Sep 23 01:15:32 Roberts-iPad CommCenter[81] <Notice>: #I BundleID: <private> is no longer a foreground app
Sep 23 01:15:32 Roberts-iPad SpringBoard(WiFiPicker)[56] <Notice>: WIFI PICKER [com.omz-software.Pythonista3]: isProcessLaunch: 0,    isForegroundActivation: 0,     isForegroundDeactivation: 1
Sep 23 01:15:32 Roberts-iPad mediaserverd(CoreMedia)[27] <Notice>: -CMSessionMgr- CMSessionMgrHandleApplicationStateChange: CMSession: Client com.omz-software.Pythonista3 with pid '2776' is now Terminated. Background entitlement: NO
Sep 23 01:15:32 Roberts-iPad analyticsd[2208] <Notice>: persist purged: <private>
Sep 23 01:15:32 Roberts-iPad SpringBoard[56] <Notice>: Running <SBAppExitedWorkspaceTransaction: 0x153db7420> for transition request:
<SBMainWorkspaceTransitionRequest: 0x1c10fc500; display: Main; source: Unspecified> {
    applicationContext = <SBWorkspaceApplicationSceneTransitionContext: 0x1c0347dd0; background: NO> entities = {
        SBLayoutRolePrimary = <SBHomeScreenEntity: 0x1c066b540; ID: com.apple.SpringBoard.workspace-entity.builtin.home-screen; layoutRole: primary>;
    };
}
Sep 23 01:15:32 Roberts-iPad SpringBoard[56] <Notice>: Not animating wallpaper orientation to landscapeRight (3) in 0.00 because it's already in that orientation

... are interesting. That's new, AFAIK, there was an OOM crash when app did consume lot of memory. Maybe something new, no idea. Anyway, killed by the system.

P.S. Great times few years ago when devs were not putting stupid messages about not animating wallpaper into the system log :( How is anyone supposed to find anything in this mess? :(

@zrzka

This comment has been minimized.

Show comment
Hide comment
@zrzka

zrzka Jan 4, 2018

Collaborator

It's not about ~/Documents only, but all other folders as well (file sharing, ...). I think that attention to this one should be increased little bit, because people can open whole folders with lot of files and Pythonista needs some optimisation, folders/files lazy loading, ... Something similar to this one I made for BM and Drag & Drop script to make it fast (just an example what I mean).

Collaborator

zrzka commented Jan 4, 2018

It's not about ~/Documents only, but all other folders as well (file sharing, ...). I think that attention to this one should be increased little bit, because people can open whole folders with lot of files and Pythonista needs some optimisation, folders/files lazy loading, ... Something similar to this one I made for BM and Drag & Drop script to make it fast (just an example what I mean).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment