Discrete Adjoint Memory Usage

[jayantmukho]

Hey everyone,

We (@bmunguia and I) ran the discrete adjoint on the ONERAM6 wing mesh with ~500, 000 elements and noticed the following memory usage:

MG_LEVEL	DIRECT	DISCRETE_ADJOINT
0	1.3GB	16GB
2	1.5GB	33GB

Is this increase in memory consumption expected?

I ran the computation on the develop branch. The mesh is in TestCases/cont_adj_rans/oneram6/

Cheers,
Jayant

[erangit]

We have also encountered the significant memory requirements of the discrete adjoint. Actually, this requirement prevented the use of fine mesh (of 10-20M cells) in optimizations process. This is not the case while using continuous adjoint but the effectiveness of the discrete adjoint is higher by far.
I think this is a very important issue in the actual use of the method. Unfortunately, I do not have any solution to suggest.
Best,
Eran

[pcarruscag]

Hi Jayant,
Your figures for MG 0 are inline with what Tim reports in his 2015 paper (10-12 times increase).
As for improving it, the computation of the various residual contributions already uses pre-accumulation, so the only way I see to make that even more effective is to pre-accumulate the sum of all residuals for each edge (that would probably make the direct solver a bit faster too).
In looking at the code I noticed that some areas of the pre-processing like setting sensors, eigenvalues, etc. do not use pre-accumulation, maybe someone already tried and saw it did not make a difference? Same goes for the MG prolongation, restriction, smoothing routines.
Cheers,
Pedro

[bmunguia]

Another potential issue is unnecessary recording of computations. Jayant and I noticed that on his fix_wall_distance branch, the MG 2 case mentioned above was using 8GB more than on develop because during the tree search, the distance computations to all possible bounding boxes were being recorded, when the only necessary computation for recording is the final computation (once the minimum distance node is determined).

So I wonder if there are other places in the code where we can turn off recording to reduce memory usage. I'm curious what @talbring's thoughts are.

Best,
Brian

[hlkline]

In terms of improvements on a larger scale, this work by Tom Taylor et al. describes a hybrid approach that could take down that memory requirement quite a bit more:
https://arc.aiaa.org/doi/abs/10.2514/6.2012-3342
In short, there are theoretically ways to get at least some of the advantages of both worlds (continuous vs discrete). I recall seeing some work that got only 4x more memory required (I think it was out of Jaoquim Martin's group? And I think they did it in part by implementing boundary conditions manually and using auto diff for jacobians) but I wasn't able to find the citation as quickly as I'd like. I hope that helps. If I manage to find that paper again I'll post again.

[talbring]

@ALL, I agree that it might be a problem for bigger cases and that we have to think of some ways to reduce the memory requirements for the discrete adjoint.

One approach that I started already, is to template the linear solver in order to use double datatype also when running the differentiated version of the code. However, that was more involved than I thought, but still its on my agenda. The nice thing is that this also reduces the runtime a lot.

The next thing is, like @pcarruscag correctly pointed out, the use of preaccumulation. However, this also requires a more detailed analysis, since it can also lead to an increased memory footprint if not applied carefully. Currently, a lot of memory is used for the geometrical derivatives, i.e. the derivatives with respect to the mesh coordinates (this requires approx 1.5 times the memory needed for derivatives with respect to the conservative variables). But since these derivatives are only needed when writing the solution we could think of doing some recomputation as runtime is not that important.

So you see we have some ideas and I am sure we can reduce the memory by approx 50% if we apply all of them. However, time is unfortunately limited at the moment, but I will definitely continue improving the adjoint solver.

Tim

[pcarruscag]

I'm pinning this issue, to my knowledge this is not being worked on directly but it should be something we keep in mind as we refactor code.

[pcarruscag]

#1022 helps with this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is still a relevant issue please comment on it to restart the discussion. Thank you for your contributions.