Skip to content

TheThing Reproducing Problems

Jump to bottom
jackfromeast edited this page Apr 19, 2024 · 2 revisions

Latest Upate:

  • Base commit: a3dcc93edc8af1c14a74d255dddebf3db25b54e5 (jaw-v3)
  • Patch commit on the base commit: 5241f016f42fb3b620b542baae508424cdd9c527
  • Adjustment:
    • support analyzing dom clobbering gadgets with ineo instead of docker (get rid of the permission issue and overhead)
    • several scratches
  • Patch Diff:
    • ./jaw-v3-thething/jaw-patch.diff

Hi Soheil @SoheilKhodayari,

First off, I want to extend a huge thanks for developing TheThing (JAW-v2). The project is not only innovative but also incredibly educational—I’ve gained so much insight from studying the source code!

I'm currently trying to run the tool on my side and have run into a few hurdles that I'm struggling to overcome. I’m hoping you could provide some guidance or suggestions on how to resolve these issues.

I clone the JAW-v2 branch on my side as I am particularly interested in detecting the DOM clobbering vulnerability. Here are some of my environments:

TheThing:

My Environment:

  • OS: ubuntu 23.04
  • USER: none-root user

My config.yaml:

# 1. which webapps to test?
testbed: 
  ## option 1: test a specifc website
  site: http://127.0.0.1:8000/domc-win-xss-1.html
  ## option 2: provide a top-site list (e.g., Alexa, Tranco, etc)
  sitelist: /input/tranco_Y3JG_unique.csv
  from_row: 1
  to_row: 5000


# 2. crawler configuration
crawler:
  # max number of urls to visit
  maxurls: 2
  # time budget for crawling each site in seconds
  sitetimeout: 1800 # 30 mins;
  # browser to use for crawling
  browser:
    name: chrome
    headless: true


# 3. static analysis configuration
staticpass:
  # time budget for static analysis of each site in seconds
  sitetimeout: 10800 # 3 hrs
  # max amount of available memory for static analysis per process
  memory: 32000

# 4. dynamic analysis configuration
dynamicpass:
  # time budget for dynamic analysis of each site in seconds
  sitetimeout: 10800 # 3 hrs
  # which browser to use
  browser:
    name: chrome
    # use remote browserstack browsers or not
    use_browserstack: false
    browserstack_username: xyz
    browserstack_password: xyz
    browserstack_access_key: xyz
  
# 5. choose the vulnerability analysis component to run
# only one component must have the `enable` option as true
general_vuln_analysis:
  # performs a general vulnerability analysis (i.e., more than one vulnerability class)
  # when enabled, other vulnerability-specific components (as configured below) 
  # will not run individually anymore
  enable: true

domclobbering:
  enabled: true
  # enable or disable the passes, useful for large-scale analysis 
  # e.g., first crawl all websites, then analyze them,
  # as opposed to crawling and analyzing sequentially at the same time
  passes:
    crawling: true
    static: true
    static_neo4j: true
    dynamic: true

cs_csrf:
  enabled: false
  passes:
    crawling: false
    static: false
    static_neo4j: false
  • The testing HTML:
<!DOCTYPE html>
<body>
<script>
    let script = document.createElement('script');
    script.src = window.clobber_prop;
    document.body.appendChild(script);
</script>
</body>
</html>

Here are some of the problems that I encountered:

1/ Use of variable before initialization

I noticed that the webpage variable is used before its declaration. I moved this inside the for-loop which seemed to resolve the issue. I attached the related code snippet for the reference.

https://github.com/SoheilKhodayari/JAW/blob/JAW-v2/analyses/domclobbering/domc_neo4j_traversals.py#L93

2/ Neo4j container startup

After I fixed the first problem, the tool outputs the following error to the console which looks like a file permission error. However, the database did not even start, when I attached it to the container and checked the log file.

Here is the content shown in /var/lib/neo4j/logs/debug.log of the neo4j container.

2024-04-14 02:53:13.543+0000 WARN  [o.n.k.d.Database] [neo4j/3048417a] Exception occurred while starting the database. Trying to stop already started components.
java.nio.file.AccessDeniedException: /data/databases/neo4j/neostore
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182) ~[?:?]
        at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
        at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:77) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:59) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.createStoreChannel(SingleFilePageSwapper.java:144) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.<init>(SingleFilePageSwapper.java:117) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.impl.SingleFilePageSwapperFactory.createPageSwapper(SingleFilePageSwapperFactory.java:57) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.impl.muninn.MuninnPagedFile.<init>(MuninnPagedFile.java:155) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.map(MuninnPageCache.java:404) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.DatabasePageCache.map(DatabasePageCache.java:72) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.PageCache.map(PageCache.java:120) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.impl.store.MetaDataStore.getRecord(MetaDataStore.java:332) ~[neo4j-record-storage-engine-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.impl.store.MetaDataStore.getStoreId(MetaDataStore.java:398) ~[neo4j-record-storage-engine-4.2.3.jar:4.2.3]
        at org.neo4j.internal.recordstorage.RecordStorageEngineFactory.storeId(RecordStorageEngineFactory.java:189) ~[neo4j-record-storage-engine-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.database.Database.checkStoreId(Database.java:571) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.database.Database.validateStoreAndTxLogs(Database.java:534) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.database.Database.start(Database.java:419) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187) ~[neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:152) ~[neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.DefaultDatabaseManager.initialiseDefaultDatabase(DefaultDatabaseManager.java:64) ~[neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.DefaultDatabaseInitializer.start0(DefaultDatabaseInitializer.java:39) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.lifecycle.SafeLifecycle.transition(SafeLifecycle.java:124) [neo4j-common-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.lifecycle.SafeLifecycle.start(SafeLifecycle.java:138) [neo4j-common-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442) [neo4j-common-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110) [neo4j-common-4.2.3.jar:4.2.3]
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198) [neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:163) [neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36) [neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:134) [neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:90) [neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35) [neo4j-4.2.3.jar:4.2.3]
2024-04-14 02:53:13.546+0000 ERROR [o.n.d.d.DefaultDatabaseManager] Failed to start DatabaseId{3048417a[neo4j]}
org.neo4j.dbms.api.DatabaseManagementException: An error occurred! Unable to start `DatabaseId{3048417a[neo4j]}`.
        at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:191) ~[neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:152) ~[neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.DefaultDatabaseManager.initialiseDefaultDatabase(DefaultDatabaseManager.java:64) ~[neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.DefaultDatabaseInitializer.start0(DefaultDatabaseInitializer.java:39) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.lifecycle.SafeLifecycle.transition(SafeLifecycle.java:124) [neo4j-common-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.lifecycle.SafeLifecycle.start(SafeLifecycle.java:138) [neo4j-common-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442) [neo4j-common-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110) [neo4j-common-4.2.3.jar:4.2.3]
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198) [neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:163) [neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36) [neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:134) [neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:90) [neo4j-4.2.3.jar:4.2.3]
        at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35) [neo4j-4.2.3.jar:4.2.3]
Caused by: java.lang.RuntimeException: java.nio.file.AccessDeniedException: /data/databases/neo4j/neostore
        at org.neo4j.kernel.database.Database.handleStartupFailure(Database.java:564) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.database.Database.start(Database.java:525) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187) ~[neo4j-4.2.3.jar:4.2.3]
        ... 13 more
Caused by: java.nio.file.AccessDeniedException: /data/databases/neo4j/neostore
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182) ~[?:?]
        at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
        at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:77) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:59) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.createStoreChannel(SingleFilePageSwapper.java:144) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.<init>(SingleFilePageSwapper.java:117) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.impl.SingleFilePageSwapperFactory.createPageSwapper(SingleFilePageSwapperFactory.java:57) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.impl.muninn.MuninnPagedFile.<init>(MuninnPagedFile.java:155) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.map(MuninnPageCache.java:404) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.DatabasePageCache.map(DatabasePageCache.java:72) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.io.pagecache.PageCache.map(PageCache.java:120) ~[neo4j-io-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.impl.store.MetaDataStore.getRecord(MetaDataStore.java:332) ~[neo4j-record-storage-engine-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.impl.store.MetaDataStore.getStoreId(MetaDataStore.java:398) ~[neo4j-record-storage-engine-4.2.3.jar:4.2.3]
        at org.neo4j.internal.recordstorage.RecordStorageEngineFactory.storeId(RecordStorageEngineFactory.java:189) ~[neo4j-record-storage-engine-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.database.Database.checkStoreId(Database.java:571) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.database.Database.validateStoreAndTxLogs(Database.java:534) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.kernel.database.Database.start(Database.java:419) ~[neo4j-kernel-4.2.3.jar:4.2.3]
        at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187) ~[neo4j-4.2.3.jar:4.2.3]
        ... 13 more

When I logged in to the neo4j browser, the query showed the following error.

tmp

I suspect the issue with the Neo4j container might be related to the sequence in which data is imported. I found a related discussion in a Neo4j commit that suggests altering the import sequence could help. Presently, the tool initiates the Docker container, including mounting the volumes, and then attempts to import the data. However, using the neo4j-admin import command requires the Neo4j process to be stopped first because of file locks.

The related code is attached here for reference: https://github.com/SoheilKhodayari/JAW/blob/JAW-v2/analyses/domclobbering/domc_neo4j_traversals.py#L120

Therefore, my solution is to try to importing the data before starting the neo4j service in the container by using my custom entrypoint bash.

echo "Starting Importing database..."
neo4j-admin import --database=neo4j --nodes=/var/lib/neo4j/import/http-127.0.0.1-8000domc-win-xss-1.html/7a3da882cf535e34d549ea52fb84a9ca7caa44958db26612251190d86c01c71a/nodes.csv --relationships=/var/lib/neo4j/import/http-127.0.0.1-8000domc-win-xss-1.html/7a3da882cf535e34d549ea52fb84a9ca7caa44958db26612251190d86c01c71a/rels.csv --delimiter='¿' --skip-bad-relationships=true --skip-duplicate-nodes=true

# Now start Neo4j
echo "Starting Neo4j..."
exec /docker-entrypoint.sh neo4j

And change the following code:

def create_neo4j_container(container_name, volume_home=VOLUME_HOME):

	if not os.path.exists(volume_home):
		os.makedirs(volume_home)


	# Should we also add the  -v {1}/neo4j/conf:/conf \ option to the below command for the neo4j conf files?

	# See: https://neo4j.com/labs/apoc/4.2/installation/#restricted
	# and https://github.com/neo4j-contrib/neo4j-apoc-procedures/issues/451
	# Should I add the option below too?
	# 	-e NEO4J_dbms_security_procedures_whitelist=apoc.coll.\\\*,apoc.load.\\\* \
	command="""docker run \
    --name {0} \
    -p7474:7474 -p7687:7687 \
    -d \
    -v {1}/neo4j/data:/data \
    -v {1}/neo4j/logs:/logs \
    -v {4}:/var/lib/neo4j/import \
    -v {1}/neo4j/plugins:/plugins \
	-v /path/to/custom_entrypoint.sh \
    -e NEO4J_apoc_export_file_enabled=true \
    -e NEO4J_apoc_import_file_enabled=true \
    -e NEO4J_apoc_import_file_use__neo4j__config=true \
    -e NEO4JLABS_PLUGINS='["apoc"]' \
    -e NEO4J_dbms_security_procedures_unrestricted=apoc.\\\* \
    -e PYTHONUNBUFFERED=1 \
    --env NEO4J_AUTH={2}/{3} \
	--entrypoint "/custom_entrypoint.sh" \
	--privileged \
    neo4j:4.2.3
	""".format(container_name, volume_home, constants.NEO4J_USER, constants.NEO4J_PASS, constants.DATA_DIR)

After implementing the suggested changes to the import sequence, the database can be queried and looks functioning correctly.

3/ File permission error due to the neo4j docker

However, after running the neo4j container, the permission of directory (i.e. /data, /docker/neo4j/volumn/neo4j/data, /docker/neo4j/volumn/neo4j/plugins, etc.) that mounted to the container are changed. Especially, the /data` folder has been changed to the following permission that prevents the python script access its data anymore.

drwx------  5 7474           7474  4096 Apr 13 22:53 data

The problem might come from this documented feature of neo4j. However, even if I set the data, logs, plugins, and import directories to 777 before starting the container, it will still change the permission in any way. (I guess run the pipeline as the root user might solve the issue?)

4/ Typographical Error in Variable Name (for dynamic verification)

In run_pipeline.py, should SITE_URL be replaced with SEED_URL for clarity? This appears to be a minor typo but could potentially confuse new contributors.

https://github.com/SoheilKhodayari/JAW/blob/JAW-v2/run_pipeline.py#L133

5/ Fail to detect the dom clobbering gadget

Unfortunately, the tool hasn’t flagged any vulnerabilities in the test scripts I’ve used. I attached the folder layout after the testing of the URL. Could there be an issue with the static or dynamic analysis configurations that I should look into?

tmp2

Navigation

icon

Clone this wiki locally