New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SOFA-NG: Roadmap #543

Open
guparan opened this Issue Dec 13, 2017 · 14 comments

Comments

3 participants
@guparan
Copy link
Member

guparan commented Dec 13, 2017

Dear SOFA community and @sofa-framework/reviewers,

As you know, one of our main objectives is currently to propose a major evolution of SOFA: this is the project "SOFA-NG", standing for Next Gen.
Let us give you some insight about the final plan and the steps to reach these objectives. Any feedback is welcome.

Objectives

The final objective of SOFA-NG is to refactorize the current (non-core) modules and components by moving them into plugins. It will permit to:

  • compile and load only the necessary modules
  • clean all dependencies in SOFA
  • help focusing the work of validation/verification/doc/test on specific codes
  • improve the packaging of codes
  • ease understanding the API of SOFA for new users

Steps of implementation

Our concerns

  • Focus on non-core components only
  • Make the transition as smooth as possible
  • Keep the history

Steps to follow

  • Propose an architecture and a folder structure
  • Decide a place for each component (incrementally)
  • Create the folder structure in SOFA plugins directory
  • Pluginize one by one the components towards their new place

Note that this project

  • will be documented, scripts will be provided to help the transition in plugins
  • is compatible (can be run in parallel) with the deprecation task

Proposed structure

https://annuel2.framapad.org/p/sofang-v0

As you can see the structure is quite similar to the old namespace structure of SOFA.
Feel free to edit/comment this pad or to propose your own version.

@damienmarchal

This comment has been minimized.

Copy link
Contributor

damienmarchal commented Dec 13, 2017

Thank Guillaume for the effort you are doing on this hard task.

For curious people here is a small test we did on how things could look like
on the "pluginization" side (the idea is to be closer to python modules):
https://github.com/SofaDefrost/sofa/blob/addModule/applications/pluginsNG/README.md

@damienmarchal

This comment has been minimized.

Copy link
Contributor

damienmarchal commented Mar 12, 2018

Hi all,

The problem with this whole SofaNG project is that it imply a major refactoring of Sofa and thus involve a lot of changes to the existing code base and imply lot of file move (which is know to be a troublemaker with git).

To evaluate how hard it would be to have sofa that match our whishes I decided to put my hand-on
https://github.com/SofaDefrost/sofa/tree/cleanTheMessStep1/ng/kernel/Sofa

Now the problem is that it is not possible to reach this state without breaking a lot of things validating each step incrementally. I don't know how to do that but I really think we need to find either a solution or stop talking about sofang.

I have done a small tool to automate a set of changes to generate an NG directory and the corresponding cmakelists straigh from the existing master code base.
Example of a changeset:

{
  "commands" : [
    ["git", "checkout", "" ,"Sofa.Helper.Types"],
    ["spm", "package", "Sofa.Helper.Types", "init"],
    ["mkdir", "Sofa/Helper/Types/src/sofa/helper/types/"],
    ["move", "../../SofaKernel/framework/sofa/helper/OptionsGroup.h", "Sofa/Helper/Types/src/sofa/helper/types/OptionsGroup.h"],
    ["move", "../../SofaKernel/framework/sofa/helper/OptionsGroup.cpp", "Sofa/Helper/Types/src/sofa/helper/types/OptionsGroup.cpp"],
    ["move", "../../SofaKernel/framework/sofa/helper/types/RGBAColor.h", "Sofa/Helper/Types/src/sofa/helper/types/RGBAColor.h"],
    ["move", "../../SofaKernel/framework/sofa/helper/types/RGBAColor.cpp", "Sofa/Helper/Types/src/sofa/helper/types/RGBAColor.cpp"],
    ["commit", "Moving all the file to their new location"],
    ["spm", "package", "Sofa.Helper.Types", "property", "source_files", "add-to", "Sofa/Helper/Types/src/sofa/helper/types/OptionsGroup.cpp"],
    ["spm", "package", "Sofa.Helper.Types", "property", "header_files", "add-to", "Sofa/Helper/Types/src/sofa/helper/types/OptionsGroup.h"],
    ["spm", "package", "Sofa.Helper.Types", "property", "source_files", "add-to", "Sofa/Helper/Types/src/sofa/helper/types/RGBAColor.h"],
    ["spm", "package", "Sofa.Helper.Types", "property", "header_files", "add-to", "Sofa/Helper/Types/src/sofa/helper/types/RGBAColor.cpp"],
    ["commit", "Registering the file to the package..."],
    ["rename", "Sofa/Helper/Types", "#include <sofa/helper/OptionsGroup.h>", "#include <sofa/helper/types/OptionsGroup.h>"],
    ["rename", "Sofa/Helper/Types", "#include <sofa/helper/RGBAColor.h>", "#include <sofa/helper/types/RGBAColor.h>"],
    ["commit", "Change the includes"],
    ["fixheader", "Sofa/Helper/Types/src/sofa/helper/types/OptionsGroup.h", "SOFA_HELPER_TYPES_OPTIONSGROUP_H"],
    ["fixheader", "Sofa/Helper/Types/src/sofa/helper/types/RGBAColors.h", "SOFA_HELPER_TYPES_RGBACOLOR_H"],
    ["fixheader", "Sofa/Helper/Types/src/sofa/helper/types/options.h", "SOFA_HELPER_CONFIG_H", "SOFA_HELPER_TYPES_CONFIG_H"], 
    ["commit", "Change the guards"],
    ["spm", "package", "Sofa.Helper.Types", "generate-cmakelists"],
    ["commit"]
   ]
}

I'm not fully convinced by I wonder if this could help us in making the move.
Eg: we start having a sofa ng branch of sofa, this branch being automatically generated from the master branch. And, if/when we are satisfied of the generating script can be re-order adequately to make a good looking history and a recipe for users to update their code base.

I'm hesitating because it somehow remind me git rebase -i and I wonder if this is not doing in the wrong way something that having the right workflow in git would sold.

@StephaneCotin

This comment has been minimized.

Copy link
Contributor

StephaneCotin commented Mar 12, 2018

Hi Damien

Are there certain categories of components that tend to have more dependencies than others?
How much of this mess could be solved by reducing even further the functionalities of SOFA-NG?
I'd like to help with this issue if I can.

@damienmarchal

This comment has been minimized.

Copy link
Contributor

damienmarchal commented Mar 13, 2018

Hello @StephaneCotin

I'm glad you are asking.

On my side I generate the following view to visualize & navigate in the inclusion graph.:
https://htmlpreview.github.io/?https://raw.githubusercontent.com/SofaDefrost/sofa/reduceInclude2/include_master_2017_12_18.html

It was suggested by @guparan that we could aggregate the files around their cmake "components" so we could identify easily the "strengh" (the amount of .h) between the cmake package.

Then from this graph views, actions can be to guide the refactoring.
Eg:

  1. remove the includes that are not mandatory to make the graph more sparse (there is very easy cases that can be done during coding sprint).
  2. when there is cycle in the graph, serious refactoring may be needed because this indicate it is not possible to properly make package out of it (this happens in Helper & DefaultType).
  3. for the elements in the graph that are included only few time this means it is very easy to put them into "external" plugins without breaking a lot of code.

I practiced 1) in https://github.com/SofaDefrost/sofa/tree/reduceInclude2
I practiced 2) in https://github.com/SofaDefrost/sofa/tree/cleanTheMessStep1/ng/kernel
I practiced 3) in several of our pluginization's PR.

What I found very hard is to make that in coordinated way and in a smooth enough way not to kill anyone's projects.

On the existing code base:

  • make PR that unify the way to declare namespace/include guards so refactoring with string replacement instead of manually patching the code base would be faciliated.
  • make PR to reduce the include's graph pressure (this will ease to cut the code into packages)
  • make more PR to deprecates components
  • put as real plugin the leaves of the dependency graph (starting with the leaves is easier because it indicate that only a small part of our code base needs to be updated).
  • find an agreement on the resulting structure.
  • find an agreement on the amount of change and understand how this will impact third party code and code history.
  • find an agreement on the process to actually make the changes.

...

@damienmarchal

This comment has been minimized.

Copy link
Contributor

damienmarchal commented Mar 13, 2018

As @untereiner said in kind of jokes...we need a gantt diagram.
Which actually I agree because such refactoring effort requires a massive amount of work and coordination.

@guparan guparan referenced this issue Mar 29, 2018

Merged

[NG] MOVE SofaComponent* to packages #620

3 of 6 tasks complete
@guparan

This comment has been minimized.

Copy link
Member

guparan commented Mar 29, 2018

Here is a quick summary of NG project evolution.

A proof of concept for a minimal version of SOFA has been pushed: https://github.com/sofa-framework/sofa-minimal-poc

Multiple iterations were made on NG architecture:
https://annuel2.framapad.org/p/sofang-v0
https://annuel2.framapad.org/p/sofang-v1
https://annuel2.framapad.org/p/sofang-v2
https://annuel2.framapad.org/p/sofang-v3

We converged towards a namespace oriented architecture splitted in two main parts: modules and plugins.
At the end, we want modules and plugins to be strictly identical in the way they are built. Same CMake behavior, same dependency handling.
The only difference that make us separate the two is the way we see their proximity with SOFA core in a long term future.
modules = things that should stay in SOFA repository
plugins = things that should move to an external repository

Most of the work done has been offline testing and discussions on Gitter. Different refactors and different CMake behaviors were tested, trying very hard not to break SOFA (or at least not too much).

@damienmarchal wrote a tool to automatize refactoring (moved to a separated repo): https://github.com/guparan/sofa2ng
It is still in early state but will be the base of future works.

Despite all this offline testing phase, the question of a clean and generic CMakeLists template for any module/plugin remains. I guess we will converge on this by actually doing the refactoring.

Finally, I just opened a primary pull-request: #620

Next steps will mainly focus on cleaning SOFA codebase to make further refactoring as easy as possible. See previous comment for specific tasks.
In parallel, Damien's tool shall be tested and improved to move on about the CMake questions.

@guparan

This comment has been minimized.

Copy link
Member

guparan commented May 2, 2018

Hi @sofa-framework/reviewers,

Here is a follow up of NG project for April.
The project is huge and still requires a lot of iterations to get the best refactoring process but we are going in the right direction 👍

Discussions

It was decided to push all NG changes to a specific NG branch on sofa-framework.
Still, all the developments will go through the PR process.

Reminder: your feedback is VERY important, we need it to adapt our work and minimize future rollbacks. Do not hesitate to put any comment in the pull-requests.

Refactoring and adapting SPM for Sofa.Component.Utils

SPM is the set of scripts we use to make our work easily reproducible. You can check it out here.

This work made me realize that it will be very hard, if not impossible, to anticipate every subtle change needed by the refactoring.

  • Do we want SPM to be an all inclusive script to refactor completely SOFA?
    every refactoring is unique and needs to change SPM
    -> namespace handling
    -> includes refactoring
  • Or should we prefer it to be an easy-to-use script to bootstrap any refactoring?
    -> very specific changes will have to be done by hand

I propose to go for solution 2 and to provide a git patch covering the work done after SPM job (fixing includes, fixing namespaces, ...).

The progress of this task is 85% since I now have to create the patch and to provide everything in the PR.

Moving on with Sofa.Helper.Bvh

Following the latest iteration on NG architecture and Damien's first draft in his cleanTheMessStep1 branch, I started refactoring SofaFramework with Sofa.Helper.Bvh.

This task needed me to heavily change/adapt SPM. It contributed to make me prefer solution 2 above.

The progress of this task is 60%.

TODO:

  • update the recipe with latest SPM changes (namespace handling)
  • adapt SPM output for smooth transition from SOFA
  • create the patch
  • open a PR

Next steps

  • Finalize and merge Sofa.Component.Utils PR
  • Open a new PR: Sofa.Helper.Bvh
  • Continue with SofaHelper refactoring

Any feedback is more than welcome 😉

@damienmarchal

This comment has been minimized.

Copy link
Contributor

damienmarchal commented May 22, 2018

@guparan I didn't noticed your last summary. Thank for it I will try to work a bit on top of that this week;

@guparan

This comment has been minimized.

Copy link
Member

guparan commented Jun 1, 2018

Hi @sofa-framework/reviewers,

May was quite quiet for our beloved NG project.
Don't worry, I still have some news for you 😉

Discussions

Pluginization works are possible but they have to be done in-place (not moving the files). This will permit to simplify the dependencies between SOFA modules.

About the NG branch, to avoid big divergence, we propose to merge it with master as soon as some big step is done. The next big step is SofaFramework coverage.

Bootstrapping scripts

Previously named SPM, the bootstrapping scripts are now able to handle test folders.

Process

Here is the process to cover one module:

  1. Create/update a recipe based on an existing one.
  2. Run the bootstrapping scripts with the recipe.
  3. Inspect the output
  4. If something is wrong or missing, edit the bootstrapping scripts and goto 1
  5. Do specific changes by hand like namespace aliases
  6. Create a patch of all changes made by hand
  7. Open a pull-request providing the recipe and the patch (previously pushed to sofa2ng)

This process will be rewritten in Sofa.Helper.Bvh pull-request.

Sofa.Component.Utils

The first NG pull-request has been merged to the NG branch.

There is still some updates to do though, because the bootstrapping scripts changed.

Sofa.Helper.Bvh

The recipe has been updated.
This work will be the base of further contributions. I will detail all the bootstrapping + patching process in the pull-request.

Next steps

  • Pluginize without moving files: use plugins mechanism to work on modules dependencies
    Start with the less dependent ones and progress towards SOFA core.
    Objective: simplify dependencies between modules.
  • Validate Damien's architecture proposal for SofaFramework: https://github.com/SofaDefrost/sofa/tree/cleanTheMessStep1/ng/kernel/Sofa
    Particularly about Sofa.Helper.Types
  • Create Sofa.Helper.Bvh pull-request
    Will give a clear example and all the process to refactor parts of SofaFramework
  • Progress within Sofa.Helper.*
    Assign tasks to all volunteers before STC#5

As always, any feedback is more than welcome 😉

@guparan

This comment has been minimized.

Copy link
Member

guparan commented Jul 2, 2018

Hi @sofa-framework/reviewers,

Here is a follow-up of SOFA-NG for June.
Only a few news for this month.

STC#5

I presented you the evolution of SOFA-NG and the future steps.
You can retrieve the presentation here.

Modularization

Two SOFA modules have been pluginized by Damien: SofaSparseSolver and SofaPreconditioner.
The idea is to make existing modules really modular: can be disabled, clean dependencies.
This work joins NG work but starting from the top - the "leafs" - of SOFA.

NG version of SofaFramework

No notable evolution on this task. Unfortunately my time has been taken on other purposes.


Next month will hopefully be way more NG-oriented for me so don't worry, it's still alive!
As always, do not hesitate to comment. 😉

@guparan

This comment has been minimized.

Copy link
Member

guparan commented Aug 1, 2018

Hi @sofa-framework/reviewers,

Here is a follow-up of SOFA-NG for July.

sofa2ng

Previously named SPM, the NG-module boostrapper has been massively updated and upgraded.
It should now able handle any standard (following conventions) SOFA module.
Check it out: https://github.com/guparan/sofa2ng

Sofa.Component.Utils

The recipe has been updated.

Sofa.Helper.Bvh

The recipe has been updated.
A pull-request proposing Sofa.Helper.Bvh and explaining how to reproduce it with sofa2ng has been done: #741

Next steps

  • Follow Sofa.Helper.Bvh PR and answer questions about sofa2ng
  • Discuss about SofaFramework modules refactoring with vonlunteers (who does what)
  • Continue refactoring SofaHelper

As always, do not hesitate to comment. 😉

@guparan

This comment has been minimized.

Copy link
Member

guparan commented Aug 31, 2018

Hi @sofa-framework/reviewers,

August was a quiet month for SOFA-NG.

Sofa.Helper.Bvh

The pull-request has been merged.
We have now 2 NG modules in the ng branch of SOFA! 🎉

Sofa.Helper.*

Here is an overview of the other Sofa.Helper parts and how hard factorization should be.
Reminder: architecture proposal comes from SofaDefrost cleanTheMessStep1 branch.

Sofa.Helper.Image
3 classes: Image, ImageDDS, ImageRAW
sofa::helper::io -> sofa::helper::image
easy

Sofa.Helper.Types
Many classes
some sofa::helper -> sofa::helper::types
some sofa::helper::types -> sofa::helper::types
hard, may need sofa2ng modifications

Sofa.Helper.Rendering
2 classes: FrameBufferObject, Transformation
sofa::helper::gl -> sofa::helper::rendering
easy

Sofa.Helper.Rendering.Gl
Many classes
sofa::helper::gl -> sofa::helper::rendering::gl
medium

Sofa.Helper.Mesh
9 classes (everything from sofa/helper/io except Image*)
sofa::helper::io -> sofa::helper::mesh
medium

Everyone is more than welcome to do a part (or at least try to).
Keep me informed in the comments, I can help if you have trouble using sofa2ng.

Next steps

  • Continue discussions about SofaFramework refactoring with vonlunteers (who does what and how)
  • Finish refactoring SofaHelper -> insure 100% coverage
  • Move on with SofaCore

As always, do not hesitate to comment. 😉

@guparan

This comment has been minimized.

Copy link
Member

guparan commented Oct 9, 2018

Hi @sofa-framework/reviewers,

Here are some news for September.

Plugin vs Classic library

I removed the config/Sofa.*.cpp file from all modules that should not be using the PluginManager API.
This is necessary to clearly distinguish what is a Plugin (= bunch of Components that are loaded on user demand with a RequiredPlugin) and what is not (= core or misc library loaded automatically).

Sofa.Helper.Types

I started the refactoring for Sofa.Helper.Types but it implies some sofa2ng modifications.
Since the sources come from 2 different places (SofaKernel/framework/sofa/helper and SofaKernel/framework/sofa/helper/types), I decided to use the recipe for SofaKernel/framework/sofa/helper only and then to refactor SofaKernel/framework/sofa/helper/types manually.
The result will be pull-requested soon.

October objectives

  • Finish refactoring Sofa.Helper.Types
  • Open PR for Sofa.Helper.Types
  • Start another Sofa.Helper.* refactoring

As always, do not hesitate to comment. 😉

@guparan

This comment has been minimized.

Copy link
Member

guparan commented Dec 12, 2018

Hi @sofa-framework/reviewers,

Long time no see, the NG task has been quite slow this last months.
Here is a quick follow up of October and November activity.

Sofa.Helper.Types

I finished the refactoring for Sofa.Helper.Types.

Next objectives

Update NG branch
Open PR for Sofa.Helper.Types
Start another Sofa.Helper.* refactoring


As always, do not hesitate to comment. 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment