Refactoring plan

[martin-raden]

here we want to

• plan how to revisit and refactor the current code base
• identify bottlenecks in current code and sketch plans how to improve performance
• identify hidden bugs and plan how to fix them

@Alexander-Mitrofanov here is the plan! :)

[martin-raden]

phase 1

we want to do a thorough reading and analysis of the current code base before we start to change it.
all observations, changes, etc. are to be rationalize and documented in respective markdown files.

0. create a new branch refactoring and work within

• create a new file doc/refactor/1-current-state.md where you will document your findings, all changes, etc.
• (!) commit and open a pull request to master branch

1. revisit the whole code base and analyze the generic interfaces and the general architecture both from the library classes as well as the binary codes

• add a summary of this analysis into the 1-current-state.md
• identify potential bugs, problems, inefficiencies and report them in 1-current-state.md
• (!) commit the updated refactor-changelog.md

2. check the test cases and identify imporant missing tests or border cases

• add missing tests, run and check them with the current code
• if you find bugs in the current code base using the tests, dont fix them yet
• add a detailed description of the changes, bugs, etc. in the commit message for reproducability in 1-current-state.md
• (!) commit the updated tests (including failing tests) as well as 1-current-state.md

3. if the last step identified minor bugs that can be easily fixed

• suggest and implement respective changes and document them in 1-current-state.md
• (!) for each bugfix do a separate commit to make the changes independent

[martin-raden]

phase 2

next we want to lift the code to the current c++ standard and make full use of up-to-date coding standards and efficient c++ programming.

1. create a new branch refactor-to-c++23 that branches of the refactoring branch and work here

• create a new file doc/refactor/2-c++23.md where you will document your findings, all changes, etc.
• (!) commit and open a pull request to refactoring branch

2. analyse the code base and adapt the code to "c++23" standard but ensure respective features are supported both by linux (gcc 14) as well as macOS (Clang), since these are used by the bioconda build environment

• changes should keep the general architecture and modularity of the code described in doc/refactor/1-current-state.md intact
• ensure that all tests that are successful in the refactoring branch are also successfully executed after the refactoring
• document all changes in 2-c++23.md
• (!) commit the test-stable refactoring

3. analyse potential optimizations of the code to make it more efficient and list and discuss them in 2-c++23.md

• distinguish between "architecture preserving" and "breaking" changes
• focus mainly on preserving changes and their potential impact on performance
• suggest a roadmap to implement the identified optimizations and add it to 2-c++23.md
• (!) commit 2-c++23.md

[martin-raden]

benchmarking

to benchmark performance changes, we want to setup a workflow that is able to automatically pull different branches from the intarna repo, run benchmarks and produce for each branch (and its time stamp/commit) a benchmark JSON information. a respective script should aggregate these individual results into an aggregated markdown report. the aggregation script should be run via a github action whenever a benchmark JSON file (in a respective subfolder) is added or changed, thus keeping the report uptodate.

• create a new private repository BackofenLab/IntaRNA-refactor-benchmark
• define a reasonable test data set (that can be easily extended/pruned)
• write the git-checkout >> benchmark >> JSON pipeline for a given branch (and commit hash)
  • run it on the master, refactoring and refactor-to-c++23 branch (last commit each)
• write the aggregation script
• create the github action
• commit all of it

the benchmark should

• measure timings (best doing some iteration using mean value)
• measure memory consumption (if possible in a simple but robust way)
• check for output stability and provide reports on result divergences

[martin-raden]

phase 3

next we want to use the suggested roadmap from phase 2 to decide what optimizations are to be implemented and in which order.

• each optimization should be done in its own branch and benchmarked respectively
  • the "root branch" of each optimization is either refactor-to-c++23 or one of its subbranches if the optimization is to be implemented in multiple levels/stages

[martin-raden]

• extend the roadmap from phase 2 with a thorough profiling of code usage for different predictors
  • store the profiling information in gzipped files in the "doc/refactor/profile" folder for later comparison
  • do a thorough analysis of the profile data to suggest additional optimization strategies
• check for inefficiencies due to obsolete deep copies instead of reference access or other bottlenecks
• consider efficiency of the used boost or stl containers and suggest alternatives if more efficient implementations or containers are available
• consider the rewrite of internal code using constructs, methods, or strategies that are enabled with the c++ standards up to c++23.
• boost-vs-stl: if stl in the c++23 standard provides adequat replacements of boost containers or utility functions, rewrite the code towards stl, to reduce the boost dependency. replace stl with boost elements, if boost provides the more efficient methods.
• test and benchmark individual potential enhancements

[martin-raden]

phase 4

finally, we want to investigate how the code base can be sped up further even considering larger changes or breaks of the current architecture.

• this should still maintain a high level of modularity, to enable upcoming implementations of new recursions (predictors), output formats/handlers, input types or energy/accessibility computations.
• tests should be updated accordingly such that the test cases stay stable (if appropriate) and pass
• if possible architectural changes should be made stepwise in individual branches or at least based on step-wise commits to make the benchmark impact and code base impact trackable

generally

• the interface of the binaries should stay stable, such that external services and scripts using IntaRNA are not impacted

[martin-raden]

refactor and bugfix "basepair probs" #217

• create a new branch "refactor-basepair-probs" that pulls in the suggestions from Basepair probs #217 from the root branch https://github.com/Fulanko/IntaRNA/tree/basepair-probs
• create a new file "doc/refactor/refactor-basepair-probs.md" for documentation of this phase
• refactor the code along the changes done in the refactoring branches "refactoring" and "refactor-to-c++23" to be compatible with the recent changes, such that the suggested changes from Basepair probs #217 are compatible with the current refactoring
• check the code for simple bugs, inefficiencies etc. and fix them with individual commits and respective documentation in "refactor-basepair-probs.md"

analysis of code extension

• do a thorough investigation of the theory behind the implementation of Basepair probs #217 and write a respective documentation in the markdown file
• check for flaws or theory issues
• correct bugs and problems with individual commits

develop theory framework to enable base pair probabilities based on heuristic predictors and/or seed constraints

• create a new document "doc/theory/basepair-prob.md"
• investigate and discuss the possibility and potential recursions to compute approximate base pair probabilities based on heuristic predictors
  • discuss limitations and speedup potentials
  • ensure correctness of the suggested theoretical approaches
• extend the branch "refactor-basepair-probs" with respective implementations and needed changes to integrate them
• define and provide tests for the new approaches and checks

[martin-raden]

phase 5

after revisiting phase 3 with the new comments from above from 16.08.2026

step 1

• integrate all beneficial changes from phase 3 and phase 4 into a new base branch "refactor-breaking-changes" defining the reference for phase 5

step 2

• consider potential architectural changes to improve the runtime of the tool that
  • maintain a high level of modularity following the current main modules (energy, accessibility, predictor, outputhandlers+trackers, ..)
  • keep the binary's interface stable
  • the API of the library is allowed to change given the constraints from above
• consider the potential of defining and using class templates instead of class hierachies to improve performance