Revisit Nuttx Build #6881
Comments
|
What about applying the patches in place? |
We did that before cmake.
No and since we iterate board configs as we build. in tre would be |
|
@davids5 I think the way patches are applied done in #6878 is a great improvement so this can be in for now. I'd still like to keep the patches on a branch in our fork of NuttX... this would ease how we apply newer patches, not needing the steps for "1. create patch with right prefix; 2) possibly edit the path; 3) copy do PX4/Firmware". Since now the patches are sorted, we can easily create a branch on NuttX that has these patches applied. Then we can remove the infra to apply the patches on CMake and rather work with submodule updates. After that I think the next step is to add support for building NuttX out of tree, so we don't need the export + unzip sequences. This may give us some trouble if upstream doesn't care about it. If it proves too troublesome we can think of a way to have in-tree build with the right set of .gitignore files so it doesn't cause the dirty submodule you mentioned. |
|
I am really glad you are looking this. I have a workable solution that allows me to continue to develop, with now a minimal impact on non NuttX developers.
For a fork that the maintainer updates they can do as they like. But this will not work for my work flow and development needs.
While upstream does look out for thing that my creator PX4 and give me heads up, we can not change the upstream workflow or practices.
Please keep in mind - we have no upstream support in changing the build. It is recursive Make with no degree of freedom. The export build it the only upstream supported facility (and we carry a patch for that) Also I need to have a functional git environment so if there is a conditional .gitignore - that would be great. If you would propose a step-by-step workflow - I would be happy to see if I can use it in all the use cases I have. My current work flow on the upstream branch of NuttX and it's submodules.
rinse-lather-force push and repeat. :) Out of the above I get the applicable changes and backports this to FMU/Firmware master.
Removed cruft and breaking changes from patches. Once CI is happy |
|
Hi @davids5 .. I think what I'm suggesting is basically replacing the step 10 and 11. I prototyped it here to see if it would work and apparently it did. I'm pushing for you to take a look. Instead of generating the patch, copying it to nuttx-patches dir and making the build system to manually apply them, this would be replaced with:
With this there's no need anymore to create the patch files and instruct the build system to apply them. It's implicitly done by updating the submodule. Let me know what you think. https://github.com/lucasdemarchi/Firmware/commits/pr-build-nuttx Since you already have to split the commits per submodule in order to send upstream, I don't see this as a problem, but as I said I don't want to disrupt your workflow since you are the one doing the heavy lift work. |
|
And for example, to add the changes from @zehortigoza would be basically to apply the branch That branch had one conflict when I was backporting it, which is easier to solve with the git tools. And the branch is built-tested only. @zehortigoza can you take a look if I got the conflict correctly? |
|
@lucasdemarchi Can we discuss this on a call? |
|
I was thinking about the discussion we had, and I am looking forward to seeing the workflow (with commands) laid out. Please set me straight if I am missing the point, or some git skills/tools that invalidate the following: But this is what I see: Assuming I have only: A needs all I think the proposal would require me to carry, maintain and keep in sync 10 instances of branches. A) 5 repos with one workflow - commits history is are applied patches. With my current solution I have 2 to branches to maintain and (2 A&B) single directories to manage. These 2 patch directories answers the question: How is this PX4 branch of nuttx different from upstream NuttX? Patches: I think I can never force push on A. - correct? Patches: On B When a patch fails can see the intent in the patch and simply redo it. Also to benefit from CI on A I would need to feature branch to not clobber master after updating the PXNuttX submodules and the NuttX submodule commit. Do I have this right? |
|
@dagar - I think the parallel build is not respecting the patch ordering. Per our chat I am noting that here. |
|
@davids5 @lucasdemarchi I was playing around with git and it makes it really easy to go back and forth between commits and patch files. For example, if you were working in a branch on top of upstream (I tagged it upstream) you could turn all commits into patches with a single git format-patch. I omitted some details like path setup. You can also apply all patches as a commit each with one command. Each patch file is the same diff, but would also include the committer, date, and commit message. So we could really get the best of both worlds with this approach. @davids5 could shuffle patches to keep track of these things, but also trivially apply them to a branch off of PX4's NuttX repo. Additionally, if we don't have to handle patches for every single NuttX build I have a solution that will allow you to edit NuttX files in place. |
|
@dagar yep. It much easier to keep the commits on a git branch. This is basically what I did on the branches pointed above. As I said the difference is basically if patches are kept as separate files or in a branch. Keeping them on a branch means we don't have to apply them at every build and avoid all headaches that this entails (see the amount of issues that this causes). The only downside is that Nuttx itself has submodules and thus you can't have patches crossing submodules. However since the end goal is to upstream the patches, they should already be split on submodule boundary anyway |
Please see the #6881 (comment) If you both a have a` complete solution, that supports the work I have described above. Please answer the questions and provide a complete workflow with all the commands. I need to see the end-to-end solution. The pieces are nice. But without a complete picture. I can not tell if you have complete solution that is more or less work with the same number of branches and level of support. In your scenarios: Assume the current nuttx (44ad7e224c1ef17911ab8b4101fd624ad9ee4177) The upstream branch of NuttX - is always the latest upstream. That has say PX4 master_nxphlite tied to it. When dev is done on NuttX upstream it is branched as upstream_kinetis While that PR (opr patch set) is inflight to upstream, master_nxphlite is tied back to NuttX upstream and the patches are placed in a pending (wip_inflight) patch file. Once that PR is merge into master upstream. wip_inflight is deleted and master_nxphlite is then tied and built against upstream. This can happen 5 time a day. Only the rejects and pending patches are in PX4/Firmware master_nxphlite . master_nxphlite is constantly rebased on PX4 master. Using 3 px4 repos say PX4, NXP, INTEL with only PX4 having a NuttX submodules. NXP and INTEL just reference PX4/NuttX. INTEL and NXP repos may have different patches, depending on the development phase. Please describe in detail the work flow in maintaining PX4, NXP, INTEL. for current NuttX and upstream |
|
Basically what I'm suggesting is "Don't change your workflow, just change how patches are applied in the end". This means that you only replace the last step on your workflow: instead of copying a a) you added a bug fix You would change (d) and (e) by: d) switch to px4 branch Notice that on (d) you will already be on top of your pile of changes before, so you can check for conflicts with previous changes at time of applying the patch rather than having to build it and hope things apply cleanly. With this little change it's possible to remove all the infrastructure from the build system to apply patches (these branches are outdated, but I can easily update them if we want to at least try this approach). All the issues with patch order, handling dependencies, etc, go away. Look how this has nothing to do with having repos PX4, NXP, INTEL... this is a non-issue: there are only 2 interesting repositories: upstream NuttX and px4 branch. |
|
This was referenced (and closed automatically) by #7007 because the original reason this was opened was my suggestion to replace our custom NuttX build wrapper with cmake ExternalProject_Add. I decided to just fix the dependencies in our custom wrapper instead. Reopening. |
We have the ordering issue resolved now. I think what you are proposing is only applicable to the non development version (current) of NuttX. Right? It does not make the workflow I have for "upstream" work the same. This means I still have to carry a set of commits or patches to support rejected and wip. In the case of commits - I would have to constantly rebase the affected NuttX submodules on upstream in multiple repos. With patches I can avoid that. I am not seeing the advantage to having 2 ways to do the same thing. |
Not really. It applicable to easily manage multiple versions/branches of NuttX.
You don't really have to rebase. Merge and/or revert still work. Note that we are not changing the NuttX revision for very long times, so I don't see where this "constantly rebase" is coming from. Looking at the log, the last time we synced with upstream was in 07923a8 (Upgrade to Nuttx 7.18+ ==upstream) |
|
@lucasdemarchi see https://github.com/PX4/Firmware/tree/master_nxphlite (I have not commited the latest there yet as the HW is in WIP) It flops between: |
|
Done in 1.7 NuttX upgrade |
@dagar
Moving this here.
Revisiting the build is fine.
We still need to maintain the make oldconfig and menuconfig in whatever we do.
We need to keep in mind that a defconfig changes need a clean build.
We need to keep in mind that a patch change needs a clean copy of the affected files in the patch.
Getting the incremental build to work when switching branches would be nice if it can be done.
Getting the incremental build to work making edits in the NuttX subtree would be nice.
One gotcha is debugging via eclipse in the build dirs. If one edits the build dirs copy of the file without realizing it you can lose your edits (and your mind).
It would great if we could fix that or offer to merge the changes. The problem will be the patched files.
Still a copy back would be prefered.
The text was updated successfully, but these errors were encountered: