Looking for suggestion on surface texture integration for triangular meshes

[Kaipapurandeswarreddy]

Hi @paulojamorim , @rmatsuda

Based on your suggestion regarding surface textures, I explored how texture mapping could be integrated into triangular meshes in InVesalius.

---

Current understanding

1. Already Existing Features in InVesalius3

Feature	Language
MIDA	Rust
LMIP	Available via fast_countour_mip(tmip=1), thresholds hardcoded
Fast Contour MIP	Rust
Volume Raycasting	VTK
VRML Export	VTK
OBJ Export	VTK

These are used for volume rendering (looking THROUGH the volume).

2. What Missing for Surface Texture

The current VRML/OBJ export:

• Exports plain mesh geometry only (vertices + faces)
• Does NOT apply texture coordinates from volume data
• No UV mapping from MIDA/raycasting onto surfaces

The medical_triangle_texture project provides:

• Texture coordinate generation for triangular meshes
• Surface raycasting (casting rays FROM surface INTO volume)
• Texture image generation from volume projections

---

Proposed Strategy: Two-Phase Approach

Phase 1: Complete Export Flow with Existing MIDA/Raycasting

Goal: Get a working texture export as quickly as possible

What i need to build:

1. Build the Cython extensions from medical_triangle_texture
   • These create texture coordinates + texture image from volume data
2. Integrate texture generation into InVesalius3's surface creation flow
   • Connect to existing Rust-based MIDA/Raycasting
3. The existing VRML/OBJ export will automatically include textures
   • (VRML/OBJ natively support textures - we just need to add texture coords to the mesh!)
4. Deliverable: Export textured mesh to VRML/OBJ

Why this first?

• Uses existing working code from medical_triangle_texture
• Leverages existing VRML/OBJ export (already in InVesalius3)
• Achieves the core goal: "integrate texture tool into triangular mesh and export"

Phase 2: Add Features One-by-One that are there in the medical_triangle_texture

After phase 1 is working, enhance incrementally:

1. Add interactive texture viewer
2. Add CLUT widget for texture editing
3. Add interactive texture viewer in 3D window
   4.Add CLUT widget for texture color editing
   5.Add separate LMIP function with customizable thresholds (currently hardcoded)

---

Regarding :Keep Cython

I noticed the existing MIDA/Raycasting is implemented in Rust (invesalius_rs), while medical_triangle_texture uses Cython.

i think NOT to converting Cython to Rust at this stage because:

1. The Cython code in medical_triangle_texture already works
2. Can be optimized later if performance is an issue

Modules to integrate to complete the export flow :

• gen_texture.pyx - texture coordinate generation
• interpolation.pyx - volume interpolation
• raycasting.pyx - surface raycasting

---

Files to Modify in InVesalius3

File	Purpose
invesalius/data/surface.py	Add texture generation after surface creation
invesalius/gui/task_exporter.py	Add texture export options
invesalius/gui/task_surface.py	Add texture settings to surface creation

---

Questions for Your Feedback

1. Does this strategic approach make sense?
2. Should I start with VRML or OBJ as the primary export format?
3. Any concerns about keeping Cython instead of converting to Rust?

Looking forward to your guidance!

Best regards,
k.purandeswarReddy

[Kaipapurandeswarreddy]

Hi @paulojamorim, could you please share your thoughts on this so I can proceed in the right direction?

[paulojamorim]

Hi @Kaipapurandeswarreddy

InVesalius no longer uses Cython. We moved away from it mainly because Rust provides better memory management. Additionally, from a packaging perspective, maintaining both Cython and Rust is not ideal, so we chose to standardize on Rust only.

I recommend converting the medical_triangle_texture code from Cython to Rust.

[Kaipapurandeswarreddy]

Thank you @paulojamorim! That makes sense standardizing on Rust is the right approach. I will study the existing Rust modules in invesalius_rs and port the Cython modules from medical_triangle_texture to Rust. I will share my progress soon."

[paulojamorim]

OK!

[Kaipapurandeswarreddy]

Hi @paulojamorim, @rmatsuda

Following your guidance to move toward a Rust-based implementation, I replicated the full surface texture generation pipeline by converting this repo https://github.com/tfmoraes/medical_triangle_texture.git cython code to rust. Below is the exact workflow I am currently implemented:

1. Data Preparation & Loading (Python)
   Before invoking the Rust texture module, all required data is prepared in Python:

• The mesh is loaded from a .ply file using ply_reader.py, extracting vertices, normals, and faces into NumPy arrays.
• The volumetric data (CT/MRI) is loaded into memory as a NumPy array (using np.memmap for large datasets).
• The Color Lookup Table (CLUT) is loaded from a .plist file and converted into an RGB NumPy array.

2. Spatial Alignment (Python)
   Since mesh coordinates and volume coordinates may differ:

• The mesh bounding box is computed (mesh_min).
• A spatial shift (dynamic_origin) is applied to align the mesh with the volume index space.
• This ensures correct correspondence between surface points and voxel locations.

3. Texture Generation via Rust (FFI)
   The prepared data is passed to the Rust module:

• Python calls "_native.generate_surface_texture(...)"
• Inputs include vertices, normals, faces, volume, spacing, and intensity parameters

Below is the exact step-by-step breakdown of what happens inside the Rust module when "_native.generate_surface_texture(...)" is called:

3.1. Python-to-Rust Interface (src/texture_py.rs)
The execution begins in the PyO3 wrapper function "generate_surface_texture".

• Type Validation: The wrapper ensures that incoming NumPy arrays match expected Rust types (e.g., vertices as f64, volume as i16).
• Array Conversion: The NumPy arrays are converted into efficient Rust views ("ArrayView2", "ArrayView3") without copying data.
• Dispatch: These views are passed directly to the core function "generate_surface_texture_internal".

3.2. Texture Atlas Initialization (src/texture.rs)
Inside "generate_surface_texture_internal", the algorithm prepares the texture atlas:

• Grid Layout: The number of triangles is used to compute a square packing grid (nx × ny) for the atlas.
• Memory Allocation: Three arrays are initialized:
  • "tcoords" → UV coordinates for each triangle
  • "texture_image" → RGB texture atlas
  • "texture_normals" → normal/bump map

3.3. Triangle Processing Loop
The algorithm iterates over each face (triangle):

• UV Assignment: Each triangle is assigned a region in the 2D atlas and corresponding normalized UV coordinates are stored.
• Geometry Fetch: Vertex positions (v0, v1, v2) and normals (n0, n1, n2) are retrieved for the triangle.

3.4. Pixel Rasterization Loop
For each triangle, every pixel inside its atlas region is processed:

• Barycentric Mapping: "uv_to_barycentric" computes barycentric weights for the pixel.
• 3D Reconstruction: These weights interpolate the 3D position (vx, vy, vz) and normal (inx, iny, inz).
• Volume Indexing: The 3D coordinate is mapped into voxel space (px, py, pz) for volume access.

3.5. Sampling, Raycasting, and Shading
This is the core computation stage:

• Trilinear Sampling: The volume is sampled at (px, py, pz) using trilinear interpolation.

• Gradient Estimation: Neighboring samples are used to compute gradients for normal map generation.

• Base Intensity Mapping: The sampled voxel value is normalized (or mapped via WL/WW + CLUT) to obtain base color.

• Subsurface Raycasting:

  • A ray is cast along the inverse surface normal
  • The algorithm steps into the volume (nsteps iterations)
  • At each step:
    • Voxel intensity is sampled
    • Alpha (opacity) is computed
    • Colors are composited using front-to-back accumulation

• Pixel Write: The final accumulated RGB value is written into "texture_image".

3.6. Return to Python (src/texture_py.rs)
After processing all triangles and pixels:

• The populated Rust arrays ("tcoords", "texture_image", "texture_normals") are converted back to NumPy arrays using ".to_pyarray(py)"
• These arrays are returned to Python as a tuple for further saving and visualization

Outputs returned to Python:

• UV coordinates (tcoords)
• Texture image (RGB atlas)
• Texture normal map (tnormals)

4. Saving the Assets (Python)

• The generated texture image is saved as a PNG file
• The mesh, along with UV coordinates, is saved using PlyWriter
• The saved mesh references the generated texture for rendering

5. 3D Rendering (Python + VTK)

• The textured mesh is visualized using VTK
• Geometry is reconstructed using vtkPoints, vtkTriangle, and vtkPolyData
• UV coordinates are applied using numpy_to_vtk
• The PNG texture is mapped onto the mesh and displayed interactively

This pipeline has been tested on real DICOM data, and the generated textures successfully capture anatomical structures such as the rib cage and spine.

I would appreciate your feedback on whether this implementation aligns with the intended direction for the surface texture feature, and whether any adjustments are recommended for integration into InVesalius.

Note: The current results demonstrate a working pipeline, but further refinement is needed to get production grade results

Best regards,
Kaipa Purandeswar Reddy

[tfmoraes]

This guidance makes sense. But the idea is to implement inside InVesalius. So you already have the mesh and volume in memory, you don’t need to implement it. These screenshots are from your implementation?

[Kaipapurandeswarreddy]

This guidance makes sense. But the idea is to implement inside InVesalius. So you already have the mesh and volume in memory, you don’t need to implement it. These screenshots are from your implementation?

Hi @tfmoraes,

Yes, these screenshots are from my own implementation. Before integrating into InVesalius, I first convertedyour Cython code to Rust and tested it as a standalone pipeline to validate the algorithm. This gave me confidence that the Rust implementation is correct before connecting it to InVesalius internals

5_6199378115991444317.mp4

The standalone test used:

• PLY mesh exported from InVesalius
• Raw volume extracted from DICOM
• Rust module (_native.generate_surface_texture)

Once validated, I then planned the full InVesalius
integration using existing mesh and volume from memory.

Inside InVesalius, the mesh and volume are already
available in memory:

• Mesh → Project().surface_dict[index].polydata
• Volume → Slice().matrix + Slice().spacing

My integration plan has two phases:

Phase 1 — Interactive Exploration (Pipeline 2):

• User creates surface mask in InVesalius
• Clicks "Generate Texture" button (new in task_surface.py)
• _native.generate_tcoords_hf() runs using existing
  mesh + volume from memory
• User can adjust WW/WL and projection type
  (MIP, MIDA, Contour) interactively
• These reuse existing InVesalius 2D slice controls

Phase 2 — Export (Pipeline 1):

• When user is happy with settings
• _native.generate_surface_texture() runs
• Produces RGB texture
• Exports as OBJ/VRML with texture PNG
• Uses existing InVesalius OBJ/VRML exporters

Novel Contribution — Mask-Guided Raycasting & ROI Texture:

current approach :

• Rays shoot blindly along surface normals
• Some rays miss tissue and hit air
• Results in black patches on complex surfaces( this was from my observation based)

Approach 1 — Mask-Guided Raycasting:

• InVesalius already has segmentation masks
  (Slice().current_mask.matrix)
• We pass this mask directly to the Rust
  raycasting engine
• Inside Rust, for each ray step:
  if mask[rz][ry][rx] > 0:
  sample volume
  else:
  skip (NULL_VALUE)
• Rays guaranteed to hit segmented tissue
• No black patches
• Works for both:
  Threshold-based masks (bone, soft tissue)
  Brain segmentation masks
  Any binary mask in InVesalius

Approach 2 — ROI Surface Texture:

• User selects specific Region of Interest (ROI)
  within the mask
• Only that sub-region gets texture applied
• Rest of surface stays plain color
• Example:
  Whole brain surface → plain grey
  Tumor ROI → textured with tissue density

This leverages InVesalius's existing segmentation
capabilities to produce clinically reliable and
targeted surface textures

I have already implemented:

• surface_texture.py (core integration module)
• dialogs_texture.py (texture settings dialog)
• task_surface.py (Generate Texture button)
• surface.py (pubsub integration)

Looking forward to your feedback on this approach!

Best regards,
Kaipa Purandeswar Reddy

[paulojamorim]

Hi @Kaipapurandeswarreddy

Nice!

This functionality should be outside neuronavigation. Please see the video:

The user should enable volume rendering, perform the desired adjustments, and then go to the “4. Export data” tab and select the new option “Export 3D surface from volume rendering...”.

2026-03-22.23-23-10_conv.mp4

The user should be able to choose between VRML or OBJ for now (3MF will be added soon). After selecting the file format, a window with a progress bar should be displayed. A triangular mesh with texture should be generated and exported to the location specified by the user.

[Kaipapurandeswarreddy]

Hi @paulojamorim, @rmatsuda could you please share your thoughts on this so I can proceed to next process ?

[paulojamorim]

Hi @Kaipapurandeswarreddy

Nice!

This functionality should be outside neuronavigation. Please see the video:

The user should enable volume rendering, perform the desired adjustments, and then go to the “4. Export data” tab and select the new option “Export 3D surface from volume rendering...”.

2026-03-22.23-23-10_conv.mp4

The user should be able to choose between VRML or OBJ for now (3MF will be added soon). After selecting the file format, a window with a progress bar should be displayed. A triangular mesh with texture should be generated and exported to the location specified by the user.

[Kaipapurandeswarreddy]

Thank you @paulojamorim! This is very clear. I will implement it in the Export data tab with VRML/OBJ format selection and a progress bar. Will update progress here soon.

I have one question regarding my approach. If I continue implementing this feature as a pre-GSoC contribution, is it still fine to submit a GSoC proposal based on this work?

Best regards,
Kaipa Purandeswar Reddy

[paulojamorim]

Thank you @paulojamorim! This is very clear. I will implement it in the Export data tab with VRML/OBJ format selection and a progress bar. Will update progress here soon.

I have one question regarding my approach. If I continue implementing this feature as a pre-GSoC contribution, is it still fine to submit a GSoC proposal based on this work?

Best regards, Kaipa Purandeswar Reddy

Yes! Of course.

[Kaipapurandeswarreddy]

Hi @paulojamorim, @tfmoraes,
As the GSoC submission deadline is approaching, I am currently preparing my proposal for the surface texture feature and wanted to ask for your guidance.
Between the current surface-based pipeline and the transfer-function-based approach we discussed, which direction would you recommend to write proposal?

Thank you!

[paulojamorim]

Write that you will test both approaches.

[Kaipapurandeswarreddy]

Hi @paulojamorim,
I wanted to let you know that I am currently in my semester exams until April 20, 2026.
I will push the updated Pipeline 2 code immediately after my exams. Whenever I get free time between exam preparations, I will work on improving the texture quality and update the PR with progress.

Thank you for your patience and continued guidance!

[paulojamorim]

Hi @Kaipapurandeswarreddy . Ok! Not problem! Good luck.

[Kaipapurandeswarreddy]

Hi @paulojamorim,
My exams have just been completed, and I’m resuming work on the surface texture pipeline.
Before resuming it, I wanted to check if there are any other areas you would like me to focus on first.
Thank you!

[paulojamorim]

Ok! Thanks!

[Kaipapurandeswarreddy]

Hi @paulojamorim,
I’m investigating the textured surface export for the Airways preset.

The current export uses the VR preset transfer function to derive an isovalue and generate a surface, then applies the texture. For Airways, the lungs/airways are being generated, but the same low-HU threshold also captures the outer air/body boundary, so the exported result includes a skin/body shell that covers the internal lung structures.

I tried a connected-component cleanup, but it is unreliable because in some cases the lung/airway region can be the largest connected component, so it may remove the wrong structure.

Do you have a preferred direction for this case?

you can see the lungs/airways are being generated.

[Kaipapurandeswarreddy]

Hi @paulojamorim,
Just a gentle follow-up on the Airways preset issue. Whenever you have time, could you please review it and share your thoughts on the preferred direction for handling this case?
Thank you!