Less contentious restructure

[thejohnfreeman]

This PR implements half of the changes planned for the physical redesign. These changes, described below, are in files touched infrequently by other contributors, so I expect them to be less disruptive or contentious. (The second half of the changes will come in a later PR that I expect to be more contentious.)

• Remove several unused files, including CMake modules and years-old scripts.
• Remove Builds/containers, which I've been helping @legleux over the last couple of weeks exfiltrate to a separate project, to decouple its development cadence from that of rippled. As far as I know, only Ripple uses these files, and only to build and test packages.
• Consolidate all third-party libraries within external/ by moving secp256k1 and ed25519-donna out of src/. From now on, src/ will contain only first-party sources.
• Simplify the generation of protobufs by leveraging the standard CMake function for it. Put the generation commands next to each other in the same file. Package protobufs into a library separate from libxrpl, to avoid issues with unity builds.
• Standardize on .h as the only extension for first-party headers. This could require an importer to change its include paths, but the only affected headers are part of beast/{unit_,}test/, so I don't think any other projects are affected (validator-keys-tool is not).

Before this PR is merged, we should:

• Confirm it has at least two approvals from qualified reviewers.
• Rebase the commits on top of develop. The commit sequence that predates the review should be preserved. Fixes made since the review opened should be merged into earlier commits.
• Rewrite non-substantive (e.g. formatting) commits as authored by "Pretty Printer cpp@ripple.com".
• Add rewritten non-substantive commit IDs to .git-blame-ignore-revs.
• Confirm the branch still passes all tests after making these changes.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.9%. Comparing base (676aae2) to head (b84f7e7).

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #4966     +/-   ##
=========================================
- Coverage     71.0%   70.9%   -0.0%     
=========================================
  Files          796     796             
  Lines        66727   66727             
  Branches     10973   10978      +5     
=========================================
- Hits         47348   47336     -12     
- Misses       19379   19391     +12     

Files	Coverage Δ
src/ripple/app/main/Main.cpp	47.0% <ø> (ø)
src/ripple/app/paths/AMMLiquidity.h	100.0% <ø> (ø)
src/ripple/overlay/impl/TxMetrics.cpp	16.2% <ø> (ø)
src/ripple/overlay/impl/TxMetrics.h	100.0% <ø> (ø)
src/ripple/protocol/impl/SecretKey.cpp	83.4% <100.0%> (ø)

... and 5 files with indirect coverage changes

[godexsoft]

I like the changes 👍
Just one thing that i don't necessarily agree with is renaming to .h instead of going the other way and renaming everything to .hpp. Do you have C code in the project? Or why was .h preferred?
To me it feels like a bit of a step backwards.

One side-effect of using .h is that github will count and report some files as C code if the header is simple enough that it could pass for a C header.

[scottschurr]

This restructure looks great to me.

FWIW, unlike @godexsoft, I think changing the few *.hpp files to *.h files is a good approach. The code base has a ton of *.h files. If we're going for uniformity (and I think that's a good idea) this smaller change is better. Switching all of the *.h files in the build to *.hpp would be much more disruptive.

Almost everything in this pull request built for me out of the box. But I had to make two small changes to the source to get the build to complete. Neither file that I had to change is currently in this change set. So I'll describe the issues here.

1. In src/ripple/protocol/impl/SecretKey.cpp:194 I get the following error:

error: 'secp256k1_ec_privkey_tweak_add' is deprecated: Use secp256k1_ec_seckey_tweak_add instead [-Werror,-Wdeprecated-declarations]
            if (secp256k1_ec_privkey_tweak_add(
                ^

If I change the secp256k1_ec_privkey_tweak_add on that line to secp256k1_ec_seckey_tweak_add then that file builds and links.

2. I get the following error on src/test/protocol/MultiApiJson_test.cpp:55:

error: unused variable 'index' [-Werror,-Wunused-const-variable]
    constexpr static auto index =
                          ^

And, indeed, if I remove the five lines that declare this static lambda then everything builds and links just fine.

Once those small things are patched up everything builds, links, passes unit tests, and syncs on my Intel Mac.

[godexsoft]

@scottschurr I have no objections either way. If moving to .h is significantly easier then that's a valid strategy 👍
I'm wondering however who is it disruptive for? Does anyone but Clio actually consume libxrpl? From Clio's perspective, where we already have everything as .hpp it would look better if rippled's headers were also .hpp but that's very minor.

[scottschurr]

I have no objections either way. If moving to .h is significantly easier then that's a valid strategy 👍
I'm wondering however who is it disruptive for?

That's a fair question, @godexsoft. In terms of disruption I was thinking of folks, like me, who work in the rippled code base. Renaming fewer files means I don't have to retrain that part of my brain. I also frequently have trouble tracing history through renamed files in git. Reducing the number of renamed files will increase the likelihood that looking at history using git continues to work for more files.

But neither of those are killer arguments. It's mostly about my personal convenience. So, yeah, I'm still leaning toward the current approach that @thejohnfreeman has taken.

[thejohnfreeman]

I prefer .hpp to .h too, stylistically, but the .hpp files are in the extreme minority in this codebase, to the point that they are overlooked: they have been mistakenly excluded from the formatting requirements for years.

The next PR is about to rename every file in src/ripple. If we want to adopt .hpp, that would be the time, but I don't feel strongly enough to fight for it.

[ximinez]

Does anyone but Clio actually consume libxrpl?

Two that I know about are:

• https://github.com/ripple/validator-keys-tool
• https://github.com/ximinez/ripple-offline-tool

[ximinez]

I left a couple of little tweaks, but otherwise this looks good.

[ximinez]

Looks really good.

[ximinez]

I like how you updated this!

[Bronek]

I verified that the following commits do exactly what they claim to do, and can be added to .git-blame-ignore-revs

fa3e44cc07 Consolidate external libraries
95491c0cce Rename .hpp to .h
4de37264d4 Format formerly .hpp files
ccacc94aba Rewrite includes

I would like to know for sure that the removal of packaging files won't hurt the users - I assumed they are used to build .deb and .rpm but I guess was wrong ?

[thejohnfreeman]

@Bronek the packaging files have been moved to a separate, internal (Ripple GitLab) project. They were never used outside of Ripple. They drive our internal package pipelines. As a separate project, they can be developed outside of the rippled review and release process. I've already worked with @legleux to ensure that the new project can successfully package this branch.

[thejohnfreeman]

Now that this PR has two approvals, I will work on the finishing steps outlined at the end of the PR description.