Skip to content

DirectoryStructure

Jump to bottom
mit-carbon edited this page Sep 13, 2010 · 5 revisions

This document covers the layout of Graphite’s directory structure and important files within each directory.

/

/common

  • directory_structure#/common/config: An independent parser library used to implement the Config class, confusingly also named config::Config.
  • directory_structure#/common/core: Contains code for the management of a single executing core or tile in the simulator. It contains a lot of back-end code for memory management and cache coherence protocols. Also contains the core-side (client-side) aspects of syscall modeling. In some sense, the Core class itself is simply a container for sub-classes that are responsible for different aspects of the system. Perhaps the Core class should be called Tile. Somehow, this directory ended up containing mostly memory-management files.
  • directory_structure#/common/misc: Contains miscellaneous code for a lot of different tasks; this is something of a dumping ground.
  • directory_structure#/common/network: Contains code for networks contained at each tile in the simulator.
  • directory_structure#/common/performance_model: Contains code for the performance modeling of different components of the system. There are separate models for the core (execution engine on each tile), the memory controller, dram directory, etc.. These each need to modeled separate because events are handled independently.
  • directory_structure#/common/system: Contains code for management of system-wide state. In particular, this section implements the MCP and LCPs. It also contains the SimThread, which is the system thread that runs on each tile in the simulation and manages the network and memory controller.
  • directory_structure#/common/transport: Contains the physical transport layer, the lowest-level messaging primitive. It contains several implementations, only one of which (‘’socktransport.{cc,h}’’) is currently viable.
  • directory_structure#/common/user: Contains user-level interface to the simulator back-end. By ‘user’, here we mean application writers, not the simulator front-end.

/common/core

As mentioned previously, this contains the Core class which is mostly a container class and a lot of shared memory/cache coherence code.

  • ‘’core.{cc,h}’‘: Implementation of the Core class. This class is a container of a core performance model, network, and dram directory (i.e., memory controller). Maybe it should have been named ’Tile’. It also contains a lot of other implementation-specific code specific to a tile that doesn’t clearly belong to any of the sub-components.
  • ‘’syscall_model.{cc,h}’’: The client side of the syscall model, responsible for maintaining a single OS state across a distributed simulation.
  • ‘’*.{cc,h}’‘: The other source files in ’’/common/core’’ implement shared memory in one way or another.
  • ‘’/common/core/cache’’: Implements the cache local to this tile. There is a base model and several implementations of this model.
  • ‘’/common/core/directory_schemes’’: Different directory schemes for a cache coherence protocol.
  • ‘’/common/core/old_cache’’: Obsolete code.
  • ‘’/common/core/pr_l1_pr_l2_dram_directory’‘: An implementation of a private l1, private l2 cache coherence protocol based on the directory schemes provided in ’’/common/core/directory_schemes’’.

/common/misc

This section covers a few of the important files/concepts that are contained in this directory.

  • ‘’config.{cc,h}’‘: Contains code for the configuration of the simulator, mostly the Config class which reads in configuration options from a ’*.cfg’ file at start-up. This class is instantiated as a singleton and used throughout the simulator for parameters to different modules.
  • ‘’cond.{cc,h}’‘, ’’lock.{cc,h}’’, ‘’thread.{cc,h}’‘, ’’semaphore.{cc,h}’’: These files abstract different synchronization primitives.
  • ‘’thread.{cc,h}’‘, ’’tls.{cc,h}’’: These abstract threading and thread local storage. This is necessary to distinguish between using the Pin version of these primitives and their native pthread versions. This is a little confusing, but there is an implementation of the Thread class that uses pthreads (PthreadThread) contained in this directory in the file ‘’pthread_thread.cc’‘. There is another implementation based on Pin (PinThread) contained in the directory_structure#/pin directory in the file ’’pin_thread.cc’’. At //link time//, the implementation that will be used is chosen by overriding Thread::create() if the Pin library is linked with. This is a pattern we use to choose between Pin or non-Pin versions throughout the system.
  • Other miscellaneous files, some no longer used.

/common/network

  • ‘’network.{cc,h}’’: Implementation of the Network class, which is responsible for dealing with applications and the transport layer. The NetworkModel class is responsible for modeling network behavior and routing packets.
  • ‘’network_model.{cc,h}’’: The abstract NetworkModel class.
  • ‘’network_model_*.{cc,h}’’: Different network models.

/common/system

  • ‘’mcp.{cc,h}’’: The MCP class. This is responsible for managing state that must be kept consistent across the entire simulation, such as thread-to-core mapping, operating system state, etc..
  • ‘’lcp.{cc,h}’‘: The LCP class. This is responsible for managing state on each process. This is largely done by acting as the ’representative’ of the process to the rest of the simulation, so the MCP can make requests of the process directly.
  • ‘’message_types.h’’: Message types that can be sent to the MCP.
  • ‘’core_manager.{cc,h}’’: Responsible for managing the cores in each process. Also maintains thread local storage to query which core a thread is running on when trapping into the simulator.
  • ‘’sim_thread.{cc,h}’’: The thread that runs asynchronously tied to each simulation core (i.e., application thread) to manage the network and DRAM directory.
  • ‘’sim_thread_manager.{cc,h}’’: A container class for SimThreads, similar to the CoreManager.
  • ‘’simulator.{cc,h}’’: A very high-level container class for the simulation.
  • ‘’thread_manager.{cc,h}’’: A class responsible for managing the spawning and joining of threads across a distributed simulation. Because of the distributed nature of Graphite, several steps are involved in this process. This is exaggerated by details in how Pin allows the spawning of new threads, making this process very complicated. It probably merits a separate document page of its own, but the source code attempts to comment what is happening.
  • ‘’syscall_server.{cc,h}’’: The server-side of syscall modeling. This lives in the MCP.
  • ‘’network_model_analytical_server.{cc,h}’’: An unused bit of code to approximate global network utilization for an analytical network model. This was running into floating point problems which have probably since been fixed, but it is still currently unused.
  • ‘’*sync_client.{cc,h}’’: Implementation of Graphite-specific synchronization primitives for various lax synchronization schemes.
  • Plus other miscellaneous code.

/docs

This directory was supposed to contain documentation, but never really got off the ground.

  • ’’design’’: Contains an initial design document explaining how thread distribution should work.
  • ’’memos’’: Meeting minutes from some //very// early simulator meetings.
  • ’’STYLE_GUIDELINES’’: A severely out-of-date description of style guidelines. This should be ignored. Just follow the style in each file.
  • ’’tools’’: A useful document describing how to use ‘’/tools/run_tests.py’’, along with some examples.

/pin

  • ‘’clock_skew_minimization.{cc,h}’‘: Contains insertion points for various lax synchronization schemes (i.e., ’clock-skew minimization’).
  • ‘’handle_syscalls.{cc,h}’’: The Pin side of handling system calls.
  • ‘’instruction_modeling.{cc,h}’’: The Pin side of trapping instructions to do core performance modeling.
  • ‘’pin_config.{cc,h}’‘: A Pin-specific Config class, very similar to directory_structure#/common/misc’‘/config.cc’’.
  • ‘’pin_sim.cc’’: A huge file containing the Pin entry point and various major simulator callbacks.
  • ‘’pin_{thread,tls}.{cc,h}’’: Pin implementations of abstractions provided in directory_structure#/common/misc.
  • ‘’progress_trace.{cc,h}’’: A helper tool to provide graphs of progress through the simulator, as seen in the paper.
  • ’’README’’: Completely useless.
  • ‘’redirect_memory.{cc,h}’’: Implements memory redirection using Pin to trap memory accesses and rewrite instructions. In some case, emulation of instructions in the simulator is necessary. This file is responsible for getting the application information (read? write? address?) and feeding it into the simulator back-end.
  • ‘’routine_replace.{cc,h}’’: Replaces routines that provide entry points for the application into the back-end. Also contains a few simulator functions that are trapped in the application for implementation-specific reasons. These will hopefully be removed when the Pin API expands to allow the simulator to create application threads.
  • ‘’thread_start.{cc,h}’’: Actions that must be taken when a thread is spawned in the simulation.

/tests

/tools

  • ‘’carbon_simd.py, carbon_sim_kill_all.py, carbon_sim_spawner.py’’: Obsolete spawning scripts. We now just use ssh.
  • ‘’generate_plot_data.py’’: Parses simulator output and generates tables.
  • ‘’gen_logs.sh’’: A script to parse log output and serialize it into a single file with events occurring in time-order.
  • ‘’launch.py’’: Obsolete spawning script?
  • ‘’run_tests.py’‘: The easiest way to run a batch of tests. See directory_structure#/docs’‘/tools’’.
  • ‘’schedule.py’‘, ’’tests.py’’: A more complicated scheduler script to run jobs across multiple machines. Probably best just to use ‘’run_tests.py’‘. ’’tests.py’’ contains an example of how to use ‘’schedule.py’’.
  • /tools/scripts: Other scripts. Currently contains scripts to use with progress traces (directory_structure#/pin) that parse output and generate graphs.
  • ‘’tests_infrastructure.py’‘: Common code for ’’generate_plot_data.py’’ and ‘’run_tests.py’’.

Clone this wiki locally