Skip to content

C++: Project away why to prevent tuple duplication in Buffer.qll #14382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 6, 2023

Conversation

MathiasVP
Copy link
Contributor

Iteration 3 performed really poorly on basically all forks of https://github.com/OISF/suricata:

Tuple counts for Buffer#61e3d199::getBufferSize#2#fff/3@i3#ffc522h5 after 7m31s:
  13721000    ~3205%       {2} r1 = SCAN Buffer#61e3d199::getBufferSize#2#fff#prev_delta OUTPUT In.0, In.1 'why'
  1315097     ~403%        {2} r2 = JOIN r1 WITH DataFlowUtil#47741e1f::ExprFlowCached::localExprFlowStep#2#ff ON FIRST 1 OUTPUT Rhs.1 'bufferExpr', Lhs.1 'why'
  40865305    ~365%        {3} r3 = JOIN r2 WITH __Buffer#61e3d199::getBufferSize#2#fff#prev_Buffer#61e3d199::getBufferSize#2#fff#prev_delta_DataFlow__#join_rhs ON FIRST 1 OUTPUT Lhs.0 'bufferExpr', Lhs.1 'why', Rhs.1 'result'
  
  7479500     ~662397%     {1} r4 = SCAN Buffer#61e3d199::getBufferSize#2#fff#prev_delta OUTPUT In.0
  53000       ~5094%       {1} r5 = JOIN r4 WITH DataFlowUtil#47741e1f::ExprFlowCached::localExprFlowStep#2#ff ON FIRST 1 OUTPUT Rhs.1 'bufferExpr'
  70000       ~550%        {2} r6 = JOIN r5 WITH __Buffer#61e3d199::getBufferSize#2#fff#prev_Buffer#61e3d199::getBufferSize#2#fff#prev_delta_DataFlow__#join_rhs ON FIRST 1 OUTPUT Lhs.0 'bufferExpr', Rhs.1 'result'
  802603      ~1166%       {3} r7 = JOIN r6 WITH DataFlowUtil#47741e1f::ExprFlowCached::localExprFlowStep#2#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.0 'bufferExpr', Lhs.1 'result'
  13769627773 ~226508%     {3} r8 = JOIN r7 WITH Buffer#61e3d199::getBufferSize#2#fff#prev ON FIRST 1 OUTPUT Lhs.1 'bufferExpr', Rhs.1 'why', Lhs.2 'result'
  
  13810493078 ~156967%     {3} r9 = r3 UNION r8
  
  0           ~0%          {3} r10 = SCAN Buffer#61e3d199::getBufferSize#2#fff#prev_delta OUTPUT _, In.0, In.2
  0           ~0%          {3} r11 = REWRITE r10 WITH Out.0 := -1
  0           ~0%          {3} r12 = JOIN r11 WITH Expr#ef463c5d::Expr::getChild#1#dispred#fff_120#join_rhs ON FIRST 2 OUTPUT Rhs.2 'bufferExpr', Lhs.1, Lhs.2
  0           ~0%          {3} r13 = JOIN r12 WITH Access#8878f617::VariableAccess#f ON FIRST 1 OUTPUT Lhs.0 'bufferExpr', Lhs.1, Lhs.2
  0           ~0%          {4} r14 = JOIN r13 WITH Access#8878f617::Access::getTarget#0#dispred#ff ON FIRST 1 OUTPUT Rhs.1 'why', Lhs.1, Lhs.2, Lhs.0 'bufferExpr'
  0           ~0%          {5} r15 = JOIN r14 WITH Buffer#61e3d199::memberMayBeVarSize#2#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1, Lhs.2, Lhs.3 'bufferExpr', Lhs.0 'why', Rhs.1
  0           ~0%          {5} r16 = JOIN r15 WITH Access#8878f617::Access::getTarget#0#dispred#ff ON FIRST 1 OUTPUT Lhs.4, Lhs.1, Lhs.2 'bufferExpr', Lhs.3 'why', Rhs.1 'why'
  0           ~0%          {6} r17 = JOIN r16 WITH Type#2e8eb3ef::DerivedType::getBaseType#0#dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2 'bufferExpr', Lhs.3 'why', Lhs.0, Lhs.4
  0           ~0%          {6} r18 = JOIN r17 WITH Type#2e8eb3ef::PointerType#f ON FIRST 1 OUTPUT Lhs.5, Lhs.0, Lhs.1, Lhs.2 'bufferExpr', Lhs.3 'why', Lhs.4
  0           ~0%          {4} r19 = JOIN r18 WITH Variable#7a968d4e::Variable::getUnspecifiedType#0#dispred#ff ON FIRST 2 OUTPUT Lhs.5, Lhs.2, Lhs.3 'bufferExpr', Lhs.4 'why'
  
  0           ~0%          {5} r20 = JOIN r19 WITH Type#2e8eb3ef::Type::getSize#0#dispred#ff ON FIRST 1 OUTPUT Lhs.1, Lhs.2 'bufferExpr', Lhs.3 'why', Rhs.1, _
  0           ~0%          {5} r21 = REWRITE r20 WITH Tmp.4 := 0, Out.4 := (InOut.0 + Tmp.4)
                            {5} r22 = r21 AND NOT _Access#8878f617::Access::getTarget#0#dispred#ff_Access#8878f617::VariableAccess#f_Buffer#61e3d199::__#antijoin_rhs(Lhs.0, Lhs.1 'bufferExpr', Lhs.2 'why', Lhs.3)
  0           ~0%          {5} r23 = SCAN r22 OUTPUT In.1 'bufferExpr', In.2 'why', _, In.4, In.3
  0           ~0%          {3} r24 = REWRITE r23 WITH Out.2 'result' := (In.3 - In.4) KEEPING 3
  
  0           ~0%          {4} r25 = JOIN r19 WITH Type#2e8eb3ef::Type::getSize#0#dispred#ff ON FIRST 1 OUTPUT Lhs.3 'why', Lhs.1, Lhs.2 'bufferExpr', Rhs.1
  0           ~0%          {5} r26 = JOIN r25 WITH Variable#7a968d4e::Variable::getType#0#dispred#ff ON FIRST 1 OUTPUT Lhs.0 'why', Lhs.1, Lhs.2 'bufferExpr', Lhs.3, Rhs.1
  0           ~0%          {6} r27 = JOIN r26 WITH Variable#7a968d4e::Variable::getType#0#dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2 'bufferExpr', Lhs.0 'why', Lhs.3, Lhs.4
  0           ~0%          {5} r28 = JOIN r27 WITH project#Type#2e8eb3ef::Type::getSize#0#dispred#ff ON FIRST 1 OUTPUT Lhs.5, Lhs.1, Lhs.2 'bufferExpr', Lhs.3 'why', Lhs.4
  0           ~0%          {6} r29 = JOIN r28 WITH Type#2e8eb3ef::Type::getSize#0#dispred#ff ON FIRST 1 OUTPUT Lhs.2 'bufferExpr', Lhs.3 'why', _, Lhs.1, Rhs.1, Lhs.4
  0           ~0%          {3} r30 = REWRITE r29 WITH Tmp.2 := (In.3 + In.4), Out.2 'result' := (Tmp.2 - In.5) KEEPING 3
  
  0           ~0%          {3} r31 = r24 UNION r30
  13810493078 ~156967%     {3} r32 = r9 UNION r31
  0           ~0%          {3} r33 = r32 AND NOT Buffer#61e3d199::getBufferSize#2#fff#prev(Lhs.0 'bufferExpr', Lhs.1 'why', Lhs.2 'result')
                            return r33

(I cancelled the pipeline midway.)

After this change we can fully evaluate predicate and we get:

Pipeline standard for Buffer#61e3d199::getBufferSize#2#fff@5441bw59 was evaluated in 87 iterations totaling 10027ms (delta sizes total: 12964791).
    14403190  ~2720%    {2} r1 = SCAN Buffer#61e3d199::getBufferSize#2#fff#prev_delta OUTPUT In.0, In.1
    2806382   ~322%    {2} r2 = JOIN r1 WITH DataFlowUtil#47741e1f::ExprFlowCached::localExprFlowStep#2#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1
    39977376   ~355%    {3} r3 = JOIN r2 WITH __Buffer#61e3d199::getBufferSize#2#fff#prev_delta_Buffer#61e3d199::getBufferSizeProj#1#ff#prev_DataF__#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1, Rhs.1
                    
      101888   ~332%    {1} r4 = SCAN Buffer#61e3d199::getBufferSizeProj#1#ff#prev_delta OUTPUT In.0
      15895    ~76%    {1} r5 = JOIN r4 WITH DataFlowUtil#47741e1f::ExprFlowCached::localExprFlowStep#2#ff ON FIRST 1 OUTPUT Rhs.1
      34730    ~22%    {2} r6 = JOIN r5 WITH __Buffer#61e3d199::getBufferSizeProj#1#ff#prev_Buffer#61e3d199::getBufferSizeProj#1#ff#prev_delta_Da__#join_rhs ON FIRST 1 OUTPUT Lhs.0, Rhs.1
      157715    ~64%    {3} r7 = JOIN r6 WITH DataFlowUtil#47741e1f::ExprFlowCached::localExprFlowStep#2#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Lhs.1
    81632413   ~496%    {3} r8 = JOIN r7 WITH Buffer#61e3d199::getBufferSize#2#fff#prev ON FIRST 1 OUTPUT Lhs.1, Rhs.1, Lhs.2
                    
  121609789   ~450%    {3} r9 = r3 UNION r8
                    
      101888     ~1%    {3} r10 = SCAN Buffer#61e3d199::getBufferSizeProj#1#ff#prev_delta OUTPUT _, In.0, In.1
      101888     ~0%    {3} r11 = REWRITE r10 WITH Out.0 := -1
        1483     ~0%    {3} r12 = JOIN r11 WITH Expr#ef463c5d::Expr::getChild#1#dispred#fff_120#join_rhs ON FIRST 2 OUTPUT Rhs.2, Lhs.1, Lhs.2
        1483     ~0%    {3} r13 = JOIN r12 WITH Access#8878f617::VariableAccess#f ON FIRST 1 OUTPUT Lhs.0, Lhs.1, Lhs.2
        1483     ~0%    {4} r14 = JOIN r13 WITH Access#8878f617::Access::getTarget#0#dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2, Lhs.0
          0     ~0%    {5} r15 = JOIN r14 WITH Buffer#61e3d199::memberMayBeVarSize#2#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1, Lhs.2, Lhs.3, Lhs.0, Rhs.1
          0     ~0%    {5} r16 = JOIN r15 WITH Access#8878f617::Access::getTarget#0#dispred#ff ON FIRST 1 OUTPUT Lhs.4, Lhs.1, Lhs.2, Lhs.3, Rhs.1
          0     ~0%    {6} r17 = JOIN r16 WITH Type#2e8eb3ef::DerivedType::getBaseType#0#dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2, Lhs.3, Lhs.0, Lhs.4
          0     ~0%    {6} r18 = JOIN r17 WITH Type#2e8eb3ef::PointerType#f ON FIRST 1 OUTPUT Lhs.5, Lhs.0, Lhs.1, Lhs.2, Lhs.3, Lhs.4
          0     ~0%    {4} r19 = JOIN r18 WITH Variable#7a968d4e::Variable::getUnspecifiedType#0#dispred#ff ON FIRST 2 OUTPUT Lhs.5, Lhs.2, Lhs.3, Lhs.4
                    
          0     ~0%    {5} r20 = JOIN r19 WITH Type#2e8eb3ef::Type::getSize#0#dispred#ff ON FIRST 1 OUTPUT Lhs.1, Lhs.2, Lhs.3, Rhs.1, _
          0     ~0%    {5} r21 = REWRITE r20 WITH Tmp.4 := 0, Out.4 := (InOut.0 + Tmp.4)
                        {5} r22 = r21 AND NOT _Access#8878f617::Access::getTarget#0#dispred#ff_Access#8878f617::VariableAccess#f_Buffer#61e3d199::__#antijoin_rhs(Lhs.0, Lhs.1, Lhs.2, Lhs.3)
          0     ~0%    {5} r23 = SCAN r22 OUTPUT In.1, In.2, _, In.4, In.3
          0     ~0%    {3} r24 = REWRITE r23 WITH Out.2 := (In.3 - In.4) KEEPING 3
                    
          0     ~0%    {4} r25 = JOIN r19 WITH Type#2e8eb3ef::Type::getSize#0#dispred#ff ON FIRST 1 OUTPUT Lhs.3, Lhs.1, Lhs.2, Rhs.1
          0     ~0%    {5} r26 = JOIN r25 WITH Variable#7a968d4e::Variable::getType#0#dispred#ff ON FIRST 1 OUTPUT Lhs.0, Lhs.1, Lhs.2, Lhs.3, Rhs.1
          0     ~0%    {6} r27 = JOIN r26 WITH Variable#7a968d4e::Variable::getType#0#dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2, Lhs.0, Lhs.3, Lhs.4
          0     ~0%    {5} r28 = JOIN r27 WITH project#Type#2e8eb3ef::Type::getSize#0#dispred#ff ON FIRST 1 OUTPUT Lhs.5, Lhs.1, Lhs.2, Lhs.3, Lhs.4
          0     ~0%    {6} r29 = JOIN r28 WITH Type#2e8eb3ef::Type::getSize#0#dispred#ff ON FIRST 1 OUTPUT Lhs.2, Lhs.3, _, Lhs.1, Rhs.1, Lhs.4
          0     ~0%    {3} r30 = REWRITE r29 WITH Tmp.2 := (In.3 + In.4), Out.2 := (Tmp.2 - In.5) KEEPING 3
                    
          0     ~0%    {3} r31 = r24 UNION r30
  121609789   ~450%    {3} r32 = r9 UNION r31
    79073251   ~507%    {3} r33 = r32 AND NOT Buffer#61e3d199::getBufferSize#2#fff#prev(Lhs.0, Lhs.1, Lhs.2)
                        return r33

@MathiasVP MathiasVP requested a review from alexet October 5, 2023 11:35
@MathiasVP MathiasVP requested a review from a team as a code owner October 5, 2023 11:35
@github-actions github-actions bot added the C++ label Oct 5, 2023
@MathiasVP MathiasVP added the no-change-note-required This PR does not need a change note label Oct 5, 2023
@MathiasVP
Copy link
Contributor Author

DCA looks good. So this should be perfectly safe to merge as a fix if QA comes back unhappy

@MathiasVP MathiasVP merged commit a083569 into github:codeql-cli-2.15.0 Oct 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C++ no-change-note-required This PR does not need a change note
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants