.DS_Store parsing

[zapbot]

How about a robots.txt parser that grabs & spiders through the disallow entries as well?
I've found some very interesting things through doing that, and sometimes the entries
are even comments! [do'oh!]

I've also found that .DS_Store files (OSX metadata) are a great method to discover
other things I shouldn't be able to find, and in most cases they're just as good as
an open directory listing!

The only problem that I've encountered is that the format is basically a binary, with
no easy method to parse out the proper unicode strings.

I have a .DS_Store file that I pulled off a server during a pen-test up on my site
as an example (to test out parsing, etc) => http://talesofacoldadmin.com/random/DS_Store

Here's the file-format specification (reverse-engineered) => https://wiki.mozilla.org/DS_Store_File_Format

Basically you just need to search for unicode strings in that file though (as i found
out through a hex editor). But that introduces some other oddities with gnu strings.

--== START OUTPUT ==--
#(^_^)-(!13899)-(XXXXX@quixote)-(04:18:23)-(XXX.XXX.XXX.XXX)
#(~)
strings -e l ./DS_Store 
event
galler
galler
galler
galler
histor
image
image
scrip
scrip
scrip
style
style
style
        Template
        Template
video
video
scrip
scrip
scrip
style
style
style
        Template
        Template
video
video
--== END OUTPUT ==--

You can see it gets most of the 'good' strings, but chops off the last character. I
usually do this, then open the DS_Store file in a hex editor to find the last character.
Totally in-efficient but it works. I've been meaning to write a parser for it, as there
don't seem to be any *anywhere* that weren't written in applescript. 

Thoughts?

Original issue reported on code.google.com by fitblip on 2010-12-17 21:32:15

[zapbot]

Analyzing the robots.txt file has been on my list for a while, but not thought about
analyzing .DS_Store files - that would be really neat.

Original issue reported on code.google.com by psiinon on 2010-12-18 10:19:38

• Labels added: Type-Enhancement
• Labels removed: Type-Defect

[zapbot]

Hi fitblip,

robots.txt and .DS_Store files are sometimes a good source for hidden direcories and
such. So, i like your idea. Fact is, that there is already an extension for Andiparos,
but we haven't managed to integrate all Andiparos features into ZAP. Anyway, you could
copy those extensions and improve it if you feel like doing it... ;)

Thanks for feeding the issue list with some nice ideas

Cheers,
Axel

Original issue reported on code.google.com by a.c.neumann on 2010-12-18 10:21:38

[zapbot]

Hi there,

Can this be split out so that this one is .DS Store, and create a new one for robots.txt
similar to the sitemap.xml I created earlier?

Original issue reported on code.google.com by vanderaj on 2012-07-20 12:37:15

[zapbot]

Yes, that makes sense.
Changed this to just cover .DS_Store parsing, as it includes more info about it.
Issue 330 (nit of a coincidence;) now covers robots.txt

Original issue reported on code.google.com by psiinon on 2012-07-20 12:50:49

[zapbot]

(No text was entered with this change)

Original issue reported on code.google.com by psiinon on 2012-07-20 12:51:04

[zapbot]

Hi

Digininja discuss .DS_Store parsing too and he releases his tools FDB:
http://www.digininja.org/projects/fdb.php
Maybe this PERL module could be useful and added to ZAP.

Original issue reported on code.google.com by segu.info on 2013-07-24 14:01:07

[zapbot]

Simon, I'd like to take a crack at this. I've looked at the passive parsers, and at
the very least, I reckon I could get the filenames out into a list that could then
be added to the sitemap. 

Original issue reported on code.google.com by vanderaj on 2014-04-11 03:22:49

[zapbot]

Go for it :)

Anything you need from me?

Original issue reported on code.google.com by psiinon on 2014-04-11 15:27:45

[kingthorin]

http://doc.kaitai.io/lang_java.html

http://formats.kaitai.io/ds_store/java.html

[kingthorin]

I started to look into this based on the kaitai library and model. My first attempt wasn't successful, thinking maybe I just had a particularly fickle example file I ran it through an online parser which had no problem with it. (I did get the pieces setup, adjusted the spider filter, etc. so the ground work is there 👍).

Bringing in the kaitai library(ies) just for this might be overkill, I may look at just strings'ing it as originally suggested (something like http://www.java2s.com/Code/Java/Data-Type/Stringsextractprintablestringsfrombinaryfile.htm) or reimplementing one of the Open Source python etc based parsers in Java?

[thc202]

This can be implemented as an add-on, the libraries would not be a big problem IMO. Have you checked their size?

[kingthorin]

I had started with it in core, but I can look at re-doing things as an add-on. I wasn't as much concerned about the size of the libraries as their usefulness, since they didn't work as advertised 'out of the box' I figured it might be more simple to build something from scratch vs. trying to debug and revise whatever was broken with kaitai.

[thc202]

If that's faster/easier, sounds good.

[SkypLabs]

Hi,

I am the author of the .DS_Store file description that you can find on the Kaitai Struct Format Gallery (http://formats.kaitai.io/ds_store/java.html). Let me know if I can be of any help here.

I've actually been thinking about writing a ZAP extension for this very purpose for a while now but I didn't realise that there was already a thread about it.

[kingthorin]

@SkypLabs Thanks for reaching out, I had been looking into using that model but encountered a few build issues. I'll revive the branch and add some notes/questions here.

[kingthorin]

@SkypLabs in the generated format, line 662:
io.seek(_root.buddyAllocatorBody().blockAddresses().get((int) blockId()).offset());

Cannot cast from Long to int

Specifically on the blockId part.

I think I had previously worked around this but I had to re-implement things this morning (I guess I'd done away with the previous feature branch.)

[SkypLabs]

Hey @kingthorin, sorry for my late answer.

I just compiled the ds_store.ksy description into a Java class without any issue:

$ cat /etc/debian_version
10.10

$ java --version
openjdk 11.0.12 2021-07-20
OpenJDK Runtime Environment (build 11.0.12+7-post-Debian-2deb10u1)
OpenJDK 64-Bit Server VM (build 11.0.12+7-post-Debian-2deb10u1, mixed mode, sharing)

$ sudo apt install kaitai-struct-compiler
...

$ ksc --version
kaitai-struct-compiler 0.9

# Download the latest version of my DS_Store description.
$ wget https://raw.githubusercontent.com/kaitai-io/kaitai_struct_formats/master/macos/ds_store.ksy

$ ksc -t java ds_store.ksy

$ head DsStore.java
// This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild

import io.kaitai.struct.ByteBufferKaitaiStream;
import io.kaitai.struct.KaitaiStruct;
import io.kaitai.struct.KaitaiStream;
import java.io.IOException;
import java.util.Arrays;
import java.util.ArrayList;
import java.nio.charset.Charset;

[kingthorin]

@SkypLabs thank you!

[kingthorin]

@SkypLabs any idea how to get some attention/movement on that? 😉

[SkypLabs]

@SkypLabs any idea how to get some attention/movement on that? 😉

I wish, but it seems the ticket finally got some traction from project members! 🤞

[kingthorin]

Also if you have a python PoC working that extracts content and builds URLs that'd help.

Sure, I can easily do that 👍

@SkypLabs it seems that the katai team has provided a fix/work around. I know you said earlier that the existing java class page provides instructions but it's still really unclear to me. If there was a working PoC of some sort I feel I'd be much better positioned to sort it out. (Getting the structure is clear'ish, I think I can turn an HTTP response into a byte array to be loaded, but interacting with it in useful ways isn't: I don't want to manipulate the file I just want to extract paths/filenames and use them.)

[SkypLabs]

Hey @kingthorin. Sorry again for the late answer.

Yes I've seen their suggestion. I'll submit a PR to update my ds_store.ksy. Then I'll provide you a PoC to extract the file and directory paths from a .DS_Store in Java 👍

[SkypLabs]

PR created: kaitai-io/kaitai_struct_formats#632

[kingthorin]

Wonderful, thanks very much!

[SkypLabs]

FYI, the PR has been merged.

[kingthorin]

@SkypLabs sorry to be a pain, but any news on this?

Then I'll provide you a PoC to extract the file and directory paths from a .DS_Store in Java

Thanks in advance!

[SkypLabs]

@SkypLabs sorry to be a pain, but any news on this?

Hi @kingthorin,

Sorry, I hadn't forgotten but I had difficulties finding the time to work on it. Furthermore, I hit a nasty bug when I tried to use the compiled KSY file. I submitted a patch to fix it: kaitai-io/kaitai_struct_formats#637.

I've started to write small code examples in both Python and Java. You can them here. I will try to finish them in the next coming days.

[kingthorin]

Thanks for sticking with it. I appreciate all your help!!

[SkypLabs]

Thanks for sticking with it. I appreciate all your help!!

I'm as eager as you to see this feature land! I just struggle with time.

My fix has been merged: kaitai-io/kaitai_struct_formats#637

For your information, you can use Kaitai Struct's online IDE to interactively parse a .DS_Store file to get more familiar with its internal structure:

If you want to try it out yourself, you can find sample files in my original PR. And don't forget to manually apply my fix as the macos/ds_store.ksy file in the online IDE is not up to date.

[kingthorin]

Cool, thanks

[SkypLabs]

Hi @kingthorin,

The Python implementation is complete: https://gist.github.com/SkypLabs/bc5510838acf7d6a7e3562999091f4c0#file-parser-py

I am now implementing the same logic in Java.

[kingthorin]

Cool, thanks!

[SkypLabs]

There you go: https://gist.github.com/SkypLabs/bc5510838acf7d6a7e3562999091f4c0#file-app-java

It's a quick-and-dirty first implementation but it does the job.

[kingthorin]

That's wonderful! I just skimmed it quickly and it seems to hit all the high points, so hopefully from there I can implement or implement and tweak any edge cases.

Thank you very much!

[SkypLabs]

Great! Keep me posted. I'd like to participate in the add-on creation.

[kingthorin]

It probably won't be an add-on itself, I plan to just add it as a new component to the spider. But I'll definitely let you know when I have something together (hopefully in the next week or so).

It'll probably be somewhat like the SvgHrefParser and add a seed similar to the robots.txt and sitemap.xml handling (Might also need to add it to the options/params, I'll have to discuss that with the team.)

Edit: Actually it'll be a different seed handling than robots and sitemap since they're root and this is more "everywhere". https://github.com/zaproxy/zap-extensions/blob/534f8a8da38d364e43cbd5c921d212feda117836/addOns/spider/src/main/java/org/zaproxy/addon/spider/Spider.java#L298-L313

[kingthorin]

I’ve got the functional bits well underway: https://github.com/kingthorin/zap-extensions/tree/ds-store

Just need to tackle the the user interface bits: options dialog/params, api support.

[SkypLabs]

That looks promising! I look forward to testing it once ready.

By the way, I meant to write a blog post about this new feature, to explain what a .DS_Store file is, its internal structure, the Kaitai Struct project, etc. Is it something you would be interested in as a guest post on the ZAP blog?

[psiinon]

Definitely!

[kingthorin]

Released and announced: https://twitter.com/zaproxy/status/1628803269362429959

[SkypLabs]

That's awesome! 🥳

I'll try to have the blog post ready shortly.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.