Trouble adding multiple disk images

[kefir-]

I tried adding several image files to a case. The workflow:

• add image,
• wait until ingest is complete
• add next image
• wait, etc...

requires regular hands-on on the keyboard, rather than starting a process with several image ingests that can work unattended, for example overnight. So I went ahead and added an image, and disabled Hash Lookup, Keyword Search and Process Unallocated Space to avoid spending a great deal of time on each ingest.

My results are as follows. I'm not sure what's related, so I'll try to detail my observations.

1. The first image was added successfully, and ingest started.
2. The next image was added successfully, and ingest started.
3. I tried to add a third image, but the Browse button in the "Add Image" window didn't work at this point. I pasted in the full path to the third image, and clicked next.
4. On step 3 "Adding Image" for the third image, the progress bar didn't move, and after about 5 minutes Autopsy disappeared.

I restarted autopsy and reopened my case, and at this point the first two images were available, but the ingests were no longer running. Some results were available under Extracted Content.

After the crash, I was able to re-add the third image again, and this time it was processed. I was also able to add two more images, and processing of them started.

After running for a bit, I noticed some messages in the lower right hand corner. One of them was the error "Unexpected exception: unable to create native thread". I'm not sure that's a perfect word-for-word quote, but it disappeared and I don't know where to find it again.

Autopsy 3.0.4 on 64bit Win7.

[adam-m]

Thanks for the report!
Looks like a memory issue (we've been trying to resolve for a while).
Could you forward logs to confirm?

If it is memory - related, future 64 bit builds should help.

Adam

On Wed, Feb 27, 2013 at 10:52 AM, kefir- notifications@github.com wrote:

I tried adding several image files to a case. The workflow:

• add image,
• wait until ingest is complete
• add next image
• wait, etc...

requires regular hands-on on the keyboard, rather than starting a process
with several image ingests that can work unattended, for example overnight.
So I went ahead and added an image, and disabled Hash Lookup, Keyword
Search and Process Unallocated Space to avoid spending a great deal of time
on each ingest.

My results are as follows. I'm not sure what's related, so I'll try to
detail my observations.

1. The first image was added successfully, and ingest started.
2. The next image was added successfully, and ingest started.
3. I tried to add a third image, but the Browse button in the "Add
   Image" window didn't work at this point. I pasted in the full path to the
   third image, and clicked next.
4. On step 3 "Adding Image" for the third image, the progress bar
   didn't move, and after about 5 minutes Autopsy disappeared.

I restarted autopsy and reopened my case, and at this point the first two
images were available, but the ingests were no longer running. Some results
were available under Extracted Content.

After the crash, I was able to re-add the third image again, and this time
it was processed. I was also able to add two more images, and processing of
them started.

After running for a bit, I noticed some messages in the lower right hand
corner. One of them was the error "Unexpected exception: unable to create
native thread". I'm not sure that's a perfect word-for-word quote, but it
disappeared and I don't know where to find it again.

Autopsy 3.0.4 on 64bit Win7.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/154
.

[kefir-]

Haven't found any logs..? The case Log folder is empty.

I've had autopsy disappear one or two more times. Is there anything I can do to increase the memory allocated?

[adam-m]

The app logs are in c:\Users\Username\AppData\Roaming.autopsy

You can try tweaking the -J-Xmx768m value in:

C:\Program Files (x86)\Autopsy\etc\autopsy.conf

Namely, reducing it will reduce the amount allocated to java heap, and
increase the amount available for native allocations (which is what it
seems you need in your current case).

You can try changing it from 768 to say 600 or 500, but not too low.
Let me know if it improves the multi-image ingests, as maybe we can tweak
it a bit more for the next release.

On Wed, Feb 27, 2013 at 11:12 AM, kefir- notifications@github.com wrote:

Haven't found any logs..? The case Log folder is empty.

I've had autopsy disappear one or two more times. Is there anything I can
do to increase the memory allocated?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/154#issuecomment-14182270
.

[kefir-]

I have quite a lot of RAM on this machine, it's 16GB or 24GB (not at my desk now). Should I still try this? I'm on Win7 64bit, but my jre is 32 bit - would it help if I install jre x64 to be able to access more than 4GB RAM?

[adam-m]

Switching to 64 bit JRE won't help until Autopsy is packaged to fully
support 64 bit - since we are not pure java, but also using some native
libs (tsk, etc) - those need to be 64 bit as well.

We are planning a 64 bit release (or a hybrid 32/64 bit release) in the
near future. For now on Windows we are limited to 2GB virtual memory, part
of which is used for java heap and remainder for native allocations.

On Wed, Feb 27, 2013 at 4:09 PM, kefir- notifications@github.com wrote:

I have quite a lot of RAM on this machine, it's 16GB or 24GB (not at my
desk now). Should I still try this? I'm on Win7 64bit, but my jre is 32 bit

• would it help if I install jre x64 to be able to access more than 4GB RAM?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/154#issuecomment-14199813
.

[kefir-]

I'm reproducing some issues quite regularly now, with this approach:

1. Add an image, and start ingest (disabled Hash Lookup, Keyword Search and Process Unallocated Space). I have to use the "right" one for the first step to reproduce the next steps.
2. Add a new image. Now "Adding image" finishes immediately and the whole image is Unalloc.

I repeated step 2 a few times, and when adding the fifth image, the Browse button again didn't work in the Add Image dialog.

I'd be happy to dig through the logs for you, but I can't forward them. I cleaned out the log directory before this test, and the only files with data now are autopsy.log.0, autopsy_traces.log.0 and messages.log. The exceptions I've found are:

SEVERE: Exception occurred in Registry
Exception: java.lang.ClassCastException: com.sun.org.apache.xerces.internal.dom.DeferredTextImpl cannot be cast to org.w3c.dom.Element

INFO: ERROR>java.lang.IllegalArgumentException
INFO: ERROR>    at java.nio.Buffer.position(Unknown Source)
INFO: ERROR>    at isi.pasco2.io.FastReadIndexFile.seek(Unknown Source)
INFO: ERROR>    at isi.pasco2.parser.IEIndexFileParser.readFileFormatVersion(Unknown Source)
INFO: ERROR>    at isi.pasco2.parser.IEIndexFileParser.parseFile(Unknown Source)
INFO: ERROR>    at isi.pasco2.parser.IEIndexFileParser.parseFile(Unknown Source)
INFO: ERROR>    at isi.pasco2.Main.main(Unknown Source)

INFO: ERROR>java.nio.BufferUnderflowException
INFO: ERROR>    at java.nio.HeapByteBuffer.get(Unknown Source)
INFO: ERROR>    at isi.pasco2.io.FastReadIndexFile.read(Unknown Source)
INFO: ERROR>    at isi.pasco2.parser.IEIndexFileParser.readFileFormatVersion(Unknown Source)
INFO: ERROR>    at isi.pasco2.parser.IEIndexFileParser.parseFile(Unknown Source)
INFO: ERROR>    at isi.pasco2.Main.main(Unknown Source)

INFO: ERROR>java.lang.IllegalArgumentException
INFO: ERROR>    at java.nio.Buffer.position(Unknown Source)
INFO: ERROR>    at isi.pasco2.io.FastReadIndexFile.read(Unknown Source)
INFO: ERROR>    at isi.pasco2.parser.IEIndexFileParser.readFileFormatVersion(Unknown Source)
INFO: ERROR>    at isi.pasco2.parser.IEIndexFileParser.parseFile(Unknown Source)
INFO: ERROR>    at isi.pasco2.Main.main(Unknown Source)

SEVERE: Exception occurred in Internet Explorer
Exception: java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: "NULNUL"

Exception:    org.sleuthkit.datamodel.TskDataException: Errors occurred while ingesting image
1. Cannot determine file system type (Sector offset: 0, Partition Type: )

and then, about half an hour later I get a few of these in between IngestManager telling me what it's processing:

Exception:   org.sleuthkit.datamodel.TskCoreException: Insufficient memory (tsk_malloc: Not enough space (zu requested)) (  - proc_attrseq)

[kefir-]

I forgot to note that while this test was done, Windows Task Manager told me that Autopsy was using about 900MB of RAM. I had not changed the -J-Xmx768m yet, I wanted to reproduce first to find what errors I was getting before the change.

[kefir-]

I reduced -J-Xmx768m to -J-Xmx500m and tried again. I saw many of the same exceptions (including the BufferUnderflowException) in the logs, and while the second image was actually parsed, the Browse button had stopped working again when I tried adding the third image.

I also tried increasing this value to 896m, and removed -J-Xmx24m and -J-XX:MaxPermSize=256M, and while the first import worked again, the Browse button didn't work when I tried adding the second.

Seems like we're walking a fairly thin line with the memory allocation, unless autopsy can use a lot more memory. But still, 2GB does sound like enough to parse a couple of file systems and carve through the content. Are there any memory hogs or memory leaks? The only ingest modules I'm running are Recent Activity, Exif Image Parser and Thunderbird Parser.

[adam-m]

Thanks.
Those Pasco and URL encoder exceptions are unrelated - but we will clean
those up to avoid user confusion.

But there may be something going on with memory. This is a good datapoint,
as you said keyword search was not selected during ingest, and normally
that could be a memory hog (Tika).
We should look into more memory profiling / leak detection in other modules.

On Thu, Feb 28, 2013 at 5:33 AM, kefir- notifications@github.com wrote:

I reduced -J-Xmx768m to -J-Xmx500m and tried again. I saw many of the same
exceptions (including the BufferUnderflowException) in the logs, and while
the second image was actually parsed, the Browse button had stopped working
again when I tried adding the third image.

I also tried increasing this value to 896m, and removed -J-Xmx24m and
-J-XX:MaxPermSize=256M, and while the first import worked again, the Browse
button didn't work when I tried adding the second.

Seems like we're walking a fairly thin line with the memory allocation,
unless autopsy can use a lot more memory. But still, 2GB does sound like
enough to parse a couple of file systems and carve through the content. Are
there any memory hogs or memory leaks? The only ingest modules I'm running
are Recent Activity, Exif Image Parser and Thunderbird Parser.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/154#issuecomment-14226956
.

[kefir-]

Thanks for the feedback, Adam.

I experimented some more, and found that I got the same issues even when I disabled all the ingest modules. So it appears to be related to the initial file system parsing itself.

[adam-m]

We found memory leaks in add image and fixed them, we should test this again with next release

[adam-m]

Is this still issue with 3.0.5 ?

[adam-m]

fixed with mem leak fixes in 3.0.5