Skip to content

Establish dynamic Data Model visualization.#3443

Closed
ayushgupta704 wants to merge 8 commits intointelowlproject:developfrom
ayushgupta704:dynamic-datamodel-visualization
Closed

Establish dynamic Data Model visualization.#3443
ayushgupta704 wants to merge 8 commits intointelowlproject:developfrom
ayushgupta704:dynamic-datamodel-visualization

Conversation

@ayushgupta704
Copy link
Copy Markdown
Contributor

Description

I've implemented a comprehensive refactor to resolve the "data ghosting" issue by establishing the Data Model as the unified interface for both analysis storage and visualization.

Instead of relying on brittle, hardcoded extraction logic within individual visualizers, this PR introduces a Dynamic Discovery Engine in the base Visualizer class. This engine automatically detects, humanizes, and renders all metadata stored in the additional_info JSON field of any Data Model. This ensures that no valuable API data is left hidden from the user.

Key Improvements:

  • Dynamic Metadata Rendering: Implemented _get_dynamic_metadata and _humanize_key in the Visualizer base class. This allows the UI to stay in sync with API changes automatically without requiring visualizer code updates.
  • Normalized the analyzers: I updated AbuseIPDB, GreyNoise, MaxMind, and OTX to make sure they are actually populating the Data Model correctly. I mapped the standard fields (like ISP and Country) and dumped the rest into additional_info.
  • Cleaned up the Reputation tab: I simplified the Reputation visualizer. It’s now just a "quick look" with status badges. I moved all the deep, forensic details over to the DataModel tab to keep the UI from getting cluttered.
  • Fixed false positives: I also refined the logic so that clean/trusted IPs (like 1.1.1.1) don't trigger red badges anymore.
Screen.Recording.2026-03-07.192827.mp4

Type of change

  • Bug fix (non-breaking change which fixes an issue).
  • New feature (non-breaking change which adds functionality).

Checklist

  • I read the contributing rules.
  • This is for the develop branch.
  • I followed the "How to create a Plugin" guide.
  • I added the data migration using dumpplugin.
  • Added the copyright banner.
  • Ruff linting is green (0 errors).
  • All 56 backend tests passed locally.
  • Fixed all DeepSource and CodeQL alerts (B904, B008, E501).

Closes #3420

@mlodic
Copy link
Copy Markdown
Member

mlodic commented Mar 8, 2026

thanks for the efforts but I don't think this change should be merged.
The way you populate the data models and the visualizer does compromise the original scope of them.
Data models are made to extract only some of the data, that is common to other analyzers as well, not to be filled with all the information available in the analyzer.
Also, visualizers should be predictable to the users. If dynamically created, they would create too much noise and provide too chaotic results. They should be static and clear. That is intended.
Visualizers and data models have not been created to just make the raw JSON data more affordable to read, they are made to select and organize only the right data that matters to make it more usable.

@mlodic mlodic closed this Mar 8, 2026
@mlodic
Copy link
Copy Markdown
Member

mlodic commented Mar 8, 2026

Also, you should wait to be assigned to the issue next time. I just did it 2 minutes ago but I already found the PR opened. That is what happens if you rush to implementation without waiting to have a proper discussion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants