This project is designed to track and analyze oilfield operations by integrating diverse datasets—from well production and drilling records to GIS and seismic data. The goal is to enable informed decision-making for both operational management and strategic planning, ensuring a comprehensive view of oilfield performance and asset status.
The data pipeline consists of several stages that convert raw data into actionable insights:
-
Data Ingestion & Preprocessing
- Data Conversion: CSV files are converted into GeoDataFrames.
- Spatial Standardization: All layers are reprojected to a common coordinate system (EPSG:4326/3857) to ensure consistency.
- Data Cleaning: Null values, duplicates, and outliers are removed or corrected.
-
Database Design & Storage
- Normalization: Data is normalized into dedicated tables for wells, production, drilling, and reserves.
- Spatial Databases: Geospatial data is stored in PostGIS or using GeoParquet for efficient spatial queries.
-
ETL & Workflow Automation
- ETL Frameworks: Tools like Apache Airflow or Prefect schedule and monitor data ingestion, transformation, and syncing with field reports or partner APIs.
- Versioning: Large datasets are versioned with DVC or Git LFS.
-
Visualization & Mapping
- Mapping: Maps are created using GeoPandas, Folium, or Leaflet to visualize well statuses, drilling evolution, production heat maps, and reserve distributions.
- Interactive Dashboards: Interactive visualizations are built with Plotly, Bokeh, or web frameworks like Dash and Streamlit.
| ID | Dataset Name | Description | Format | Path | Status |
|---|---|---|---|---|---|
sedimentary_basins |
Sedimentary Basins | Sedimentary basin polygons | Shapefile | data/raw/geographical/sedimentary-basins/geo-sedimentary-basins.shp |
Collected |
basin_metadata |
Basin Metadata | Attributes describing each sedimentary basin | CSV | data/raw/geographical/basin-metadata/geo-sedimentary-basins-metadata.csv |
Collected |
| ID | Dataset Name | Description | Format | Path | Status |
|---|---|---|---|---|---|
exploitation_lots |
Exploitation Lots | Boundaries of exploitation areas | CSV | data/raw/administrative/exploitation-lots/admin-exploitation-blocks.csv |
Collected |
concessions |
Concessions | Legal exploitation concessions | Shapefile | data/raw/administrative/concessions/admin-exploitation-concessions.shp |
Collected |
satellite_rasters |
Raster Basemaps | Contextual vector data (terrain, boundaries, etc.) | Shapefile | data/raw/basemap/raster-basemaps/ (directory not yet populated) |
Optional |
| ID | Dataset Name | Description | Format | Path | Status |
|---|---|---|---|---|---|
well_shapefile |
Well Locations | All wells incl. location, type, company, depth | Shapefile | data/raw/well/locations/well-locations.shp |
Collected |
well_status_log |
Well Status Over Time | Operational status per well over time | CSV | data/raw/well/production-history/well-production-history.csv |
Collected |
| ID | Dataset Name | Description | Format | Path | Status |
|---|---|---|---|---|---|
reservoirs |
Reservoirs | Hydrocarbon reservoir polygons with operator info | Shapefile | data/raw/reservoir/locations/reservoir-locations.shp |
Collected |
reservoirs_avg_depth |
Reservoirs by Avg. Depth | Reservoirs categorized by average depth | Shapefile | data/raw/reservoir/locations-by-depth/reservoir-locations-by-depth.shp |
Collected |
reserves_volumes |
Oil & Gas Reserves | Proven/probable reserve volumes | JSON | data/raw/reservoir/reserves/reserves-metadata.json |
Collected |
| ID | Dataset Name | Description | Format | Path | Status |
|---|---|---|---|---|---|
drilled_meters_by_company |
Drilling Meterage (Company) | Meters drilled per company/month/concept (2009–2025) | CSV | data/raw/drilling/meterage-by-company/drilling-meters-by-company.csv |
Collected |
drilled_meters_before_2009 |
Drilling Meterage (Pre-2009) | Historic drilled meters by company and concept | CSV | data/raw/drilling/meterage-pre-2009/drilling-meters-1900-2009.csv |
Collected |
wells_in_drilling |
Active Drilling Wells | Active drilling wells by company/month | CSV | data/raw/drilling/active-wells/drilling-active-wells.csv |
Collected |
completed_wells |
Completed Wells | Completions with company, concept, location | CSV | data/raw/drilling/completed-wells/drilling-completed-wells.csv |
Collected |
wells_completed_by_type |
Completions by Type/Company | Well completions by type and company | CSV | data/raw/drilling/completions-by-type-company/drilling-completed-wells-by-type-and-company.csv |
Collected |
wells_completed_by_concept |
Completions by Concept/Prov. | Well completions by concept and province | CSV | data/raw/drilling/completions-by-concept-province/drilling-completed-wells-by-concept-and-province-2009-2025.csv |
Collected |
wells_completed_by_welltype |
Completions by Type/Basin | Well completions by type and basin | CSV | data/raw/drilling/completions-by-type-basin/drilling-completed-wells-by-type-and-basin.csv |
Collected |
fracture_data |
Hydraulic Fracturing | Hydraulic fracturing registry (Annex IV) | CSV | data/raw/drilling/hydraulic-fracturing/well-fracture-data.csv |
Collected |
| ID | Dataset Name | Description | Format | Path | Status |
|---|---|---|---|---|---|
production_by_reservoir |
Production by Reservoir | Oil production volumes by reservoir | CSV | data/raw/production/by-reservoir/production-oil-by-reservoir.csv |
Collected |
production_by_area |
Production by Area & Consortium | Oil production by area and consortium | CSV | data/raw/production/by-area-consortium/production-oil-by-area-and-consortium.csv |
Collected |
gas_production_daily |
Daily Gas Production by Province | Gas production daily averages by province | CSV | data/raw/production/gas-daily-by-province/production-gas-daily-by-province.csv |
Collected |
| ID | Dataset Name | Description | Format | Path | Status |
|---|---|---|---|---|---|
production_forecast_csv |
Production Forecast (CSV) | Projected oil and gas production | CSV | data/raw/forecast/csv/production-forecast.csv |
Collected |
production_forecast_xlsx |
Production Forecast (XLSX) | Detailed forecast data for 2024 | XLSX | data/raw/forecast/xlsx/production-forecast-2024.xlsx |
Collected |
| ID | Dataset Name | Description | Format | Path | Status |
|---|---|---|---|---|---|
2d_seismic_lines |
2D Seismic Lines | Seismic exploration line geometries | Shapefile | data/raw/seismic/2d-lines/seismic-2d-lines.shp |
Collected |
3d_seismic_surveys |
3D Seismic Surveys | 3D seismic survey polygons | Shapefile | data/raw/seismic/3d-surveys/seismic-3d-surveys.shp |
Collected |
-
Monitoring:
Track well statuses, drilling activity, and production trends to optimize operational efficiency. -
Spatial Analysis & Mapping:
Leverage GIS data to overlay oilfield infrastructure, geological features, and legal boundaries, aiding in asset management and risk assessment. -
Forecasting:
Integrate production forecasts to support planning, budgeting, and strategic investment decisions. -
Visualization:
Enable dynamic mapping and dashboards to support detailed analyses and presentations for internal teams and external stakeholders.
- Wells: Central entity representing each oilfield well with geospatial coordinates and operational status.
- Production: Time-series data associated with wells, tracking oil, gas, and water volumes.
- Drilling: Records documenting drilling activities, including timelines and meters drilled.
- Reserves: Data on proven and probable reserves tied to each well.
- Geography: Spatial datasets for basins, concessions, and administrative boundaries providing operational context.
ArgGIS is built on a modern stack designed for geospatial data processing:
The system generates several types of visualization outputs:
-
Well Status Map
- Interactive map with wells color-coded by status (active, suspended, abandoned)
- Filter controls for well type, operator, and time period
- Pop-up information windows with detailed well information
-
Production Heat Map
- Choropleth visualization of production intensity
- Temporal slider to view production changes over time
- Aggregation options by different geographic units
-
Drilling Activity Evolution
- Animated time-series visualization showing drilling progression
- Company-specific views of drilling activity
- Comparison of drilling activity by basin or province
-
Reserve Distribution Map
- Choropleth map showing reserve volumes by basin
- Time series visualization of reserve changes
- Comparison of different reserve categories (proven, probable, possible)
Core Technologies:
- Python 3.11+
- Pandas, GeoPandas, Shapely, Fiona, Pyproj, openpyxl
- Plotly, Bokeh, Folium/Leaflet
- Docker, Poetry (for reproducible environments)
- DVC/Git LFS (for versioning large datasets)
Enhancements:
- PostGIS / GeoParquet for spatial database storage
- Contextily for ESRI/Mapbox basemaps
ArgGIS
├── backups # Backup storage for critical project data
│ ├── manifest_backup.csv # Backup of data manifest
│ ├── reserves_backup.json # Backup of reserves data
├── data # Main data directory
│ ├── interim # Intermediate data processing outputs
│ ├── processed # Final processed datasets
│ │ └── reserves # Processed reserve calculations
│ └── raw # Original unmodified source data
│ ├── administrative # Administrative and management data
│ ├── drilling # Drilling operations data
│ ├── forecast # Production forecasting data
│ ├── geographical # Geographic and spatial data
│ ├── production # Oil/gas production data
│ ├── reserves # Hydrocarbon reserves data
│ ├── reservoir # Reservoir characteristics data
│ ├── seismic # Seismic survey data
│ └── well # Well information and logs
├── metadata # Project and data documentation
│ ├── datasets # Dataset-specific metadata
│ │ ├── documentation.md # Detailed data documentation
│ │ ├── file_mapping.csv # File relationship mappings
│ │ ├── manifest.md # Dataset manifest documentation
│ │ └── manifest.yaml # Dataset manifest configuration
│ └── reserves # Reserves-specific metadata
│ ├── reserves-metadata.json # Reserves data documentation
├── notebooks # Jupyter notebooks for analysis
│ ├── EDA # Exploratory Data Analysis
│ ├── MappingPrototypes # GIS mapping prototypes
│ ├── Reports # Analysis reports and presentations
├── outputs # Generated output files
│ ├── datastore # Processed data storage
│ ├── interactive # Interactive visualizations
│ ├── maps # Generated maps and spatial plots
│ └── tables # Tabular outputs and reports
└── pipeline # Data processing pipeline
├── config.py # Pipeline configuration settings
├── ingest.py # Data ingestion and loading
├── mapping # GIS mapping modules
│ ├── map_drilling.py # Drilling activity visualizations
│ ├── map_production.py # Production data mapping
│ ├── map_reserves.py # Reserves distribution mapping
│ └── map_well_status.py # Well status visualization
├── run.py # Pipeline execution script
├── transform # Data transformation modules
│ └── reserve_processor.py # Transformation logic for reserves
└── utils # Utility functions and helpers
├── csv_utils.md # CSV handling documentation
├── csv_utils.py # CSV processing utilities
├── file_utils.py # File management functions
├── geo_utils.py # Geospatial processing utilities
├── join_utils.py # Data joining and merging tools
├── metadata_utils.py # Metadata handling utilities
├── shapefile_utils # Shapefile processing tools
└── transform_utils.py # Data transformation utilities
- Ingest Module: Centralizes file reading and initial data formatting.
- Transform Module: Handles data normalization and business-specific transformations to create unified data models.
- Mapping Module: Focuses on geospatial rendering and visualization tasks.
- Utility Functions: Provide shared functionality across ingestion, transformation, and mapping tasks.
The project is organized into the following main components:
-
ArgGIS Directory:
Contains the core code, backups, logs, metadata, notebooks for EDA and mapping prototypes, and the pipeline scripts for ingestion, transformation, and visualization. -
Pipeline Module:
Scripts and utilities are divided into submodules for ingestion (ingest.py), transformation (transform/), and mapping (mapping/), ensuring modularity and ease of maintenance. -
Metadata & Documentation:
Comprehensive documentation and data manifests are maintained within themetadatadirectory for clarity on dataset origins, formats, and transformations. -
Visualization Outputs:
Maps and interactive visualizations are generated and stored in theoutputsdirectory, allowing for straightforward sharing and review.
- Python 3.11 or higher
- Docker (optional, for containerized deployment)
- Git (for version control)
-
Clone the repository:
git clone https://github.com/yourusername/ArgGIS.git cd ArgGIS -
Set up the environment:
# Using Poetry poetry install # Or using pip pip install -r requirements.txt
-
Run the data pipeline:
python pipeline/run.py
-
Launch the visualization interface:
# Command to launch visualization interface
This project is licensed under the [LICENSE] - see the LICENSE file for details.