Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After 2.2.6 upgrade reports generates enormous .tar file in tmp/analytics folder in root #18243

Closed
Jilco opened this Issue Sep 25, 2018 · 38 comments

Comments

@Jilco
Copy link

Jilco commented Sep 25, 2018

Preconditions

  1. Magento 2.2.6
  2. Ngingx
  3. Redis

Steps to reproduce

  1. Upgrade to 2.2.6
  2. Switch on reports (or don't switch of if you already switched it on in earlier version)

Expected result

  1. I don't realy know, i think the files are needed for advance reports

Actual result

  1. In the /tmp/analytics folder in de root of the webserver an huge (1.7 Gb) .tar (~tmp-1537867209.3073tar.tar) file is generated every 24 hours (happend 2 times now after upgrade and exact the same time). Because it is in the tmp folder of the root/webserver Nginx cant make new sessions and the site goes down.

When i look in the .tar (i downloaded it) i see a lot of the same files and the same size. Maybe generating the file goes in a loop somewhere?

@magento-engcom-team

This comment has been minimized.

Copy link
Contributor

magento-engcom-team commented Sep 25, 2018

Hi @Jilco. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

  • Summary of the issue
  • Information on your environment
  • Steps to reproduce
  • Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me $VERSION instance

where $VERSION is version tags (starting from 2.2.0+) or develop branches (for example: 2.3-develop).
For more details, please, review the Magento Contributor Assistant documentation.

@Jilco do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

  • yes
  • no
@Jilco

This comment has been minimized.

Copy link
Author

Jilco commented Sep 25, 2018

@magento-engcom-team give me 2.2.6 instance

@magento-engcom-team

This comment has been minimized.

Copy link
Contributor

magento-engcom-team commented Sep 25, 2018

Hi @Jilco. Thank you for your request. I'm working on Magento 2.2.6 instance for you

@magento-engcom-team

This comment has been minimized.

Copy link
Contributor

magento-engcom-team commented Sep 25, 2018

Hi @Jilco, here is your Magento instance.
Admin access: https://i-18243-2-2-6.engcom.dev.magento.com/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

@Jilco

This comment has been minimized.

Copy link
Author

Jilco commented Sep 25, 2018

Settings in the profided instance are the same as in my situation, but i can't see what happens when generation the report files

@engcom-backlog-nazar engcom-backlog-nazar self-assigned this Sep 26, 2018

@magento-engcom-team

This comment has been minimized.

Copy link
Contributor

magento-engcom-team commented Sep 26, 2018

Hi @engcom-backlog-nazar. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

  • 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

    DetailsIf the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

  • 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

  • 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

  • 4. Verify that the issue is reproducible on 2.3-develop branch

    Details- Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
    - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

  • 5. Verify that the issue is reproducible on 2.2-develop branch.

    Details- Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

  • 6. Add label Issue: Confirmed once verification is complete.

  • 7. Make sure that automatic system confirms that report has been added to the backlog.

@engcom-backlog-nazar

This comment has been minimized.

Copy link
Contributor

engcom-backlog-nazar commented Sep 26, 2018

HI @Jilco seems like this bug is acknowledged -> #11543

@Jilco

This comment has been minimized.

Copy link
Author

Jilco commented Sep 26, 2018

@snyderm

This comment has been minimized.

Copy link

snyderm commented Sep 26, 2018

This issue is 2.2.6 specific, and appears to be generated by magento advanced analytics. This issue did not occur in 2.3.

@engcom-backlog-nazar

This comment has been minimized.

Copy link
Contributor

engcom-backlog-nazar commented Sep 26, 2018

@Jilco As a temporary solution try disable advanced reporting.

@Jilco

This comment has been minimized.

Copy link
Author

Jilco commented Sep 26, 2018

@Jilco

This comment has been minimized.

Copy link
Author

Jilco commented Sep 26, 2018

@engcom-backlog-nazar

This comment has been minimized.

Copy link
Contributor

engcom-backlog-nazar commented Sep 26, 2018

@Jilco Problem in this file
selection_138
But i can't understand why this method uses in loop.

@Jilco

This comment has been minimized.

Copy link
Author

Jilco commented Sep 26, 2018

@engcom-backlog-nazar

This comment has been minimized.

Copy link
Contributor

engcom-backlog-nazar commented Sep 26, 2018

@Jilco Problem in Cron for Analytics module this cron runs report to write file, if you turn off reports, they not be generate files.

@engcom-backlog-nazar

This comment has been minimized.

Copy link
Contributor

engcom-backlog-nazar commented Oct 1, 2018

Hi @Jilco This may be fix your issue -> #18322

@engcom-backlog-nazar

This comment has been minimized.

Copy link
Contributor

engcom-backlog-nazar commented Oct 4, 2018

Hi @Jilco This issue was fixed by this commit -> 8e1a5d3

@Flowlance

This comment has been minimized.

Copy link

Flowlance commented Oct 8, 2018

Same problem. A 83GB tar file in /tmp/analytics...

screen shot 2018-10-08 at 10 22 57

@engcom-backlog-nazar

This comment has been minimized.

Copy link
Contributor

engcom-backlog-nazar commented Oct 8, 2018

Hi @Flowlance This issue has been fixed by this commit -> 8e1a5d3

@maderlock

This comment has been minimized.

Copy link

maderlock commented Dec 4, 2018

Is this fixed in 2.2.7 then?

@Jilco

This comment has been minimized.

Copy link
Author

Jilco commented Dec 4, 2018

@maderlock

This comment has been minimized.

Copy link

maderlock commented Dec 10, 2018

Sadly, saving the settings again has not helped. Two days later the disk filled up with 200Gb of rubbish. Going to have to turn this off in 2.2.7.

Would this fix be hard to back-port to 2.2?

@maderlock

This comment has been minimized.

Copy link

maderlock commented Dec 10, 2018

Apparently there is a back-port to 2.2 that got merged in October, but this has oddly not ended up in 2.2.7.

I've tried to override \Magento\Framework\Archive\Tar to patch this myself (pulling in everything by composer means I cannot just change the file), but sadly this has not worked because this class is pull in via a low-level "new" statement that bypasses DI.

Any advice on how this can be fixed in 2.2.7 when using composer would be gratefully received. Currently my only option looks to be to tell composer to skip the entire framework so I can copy in the framework locally and patch these few line :(

@DanielRuf

This comment has been minimized.

Copy link
Member

DanielRuf commented Dec 10, 2018

@engcom-backlog-nazar not sure why you linked my clone and commit but this is clearly not the cause. Also this was not what we've merged in the end.

@DanielRuf

This comment has been minimized.

Copy link
Member

DanielRuf commented Dec 10, 2018

@maderlock you can use composer-patches and / or the patch file of the commit.

@stu177

This comment has been minimized.

Copy link

stu177 commented Jan 18, 2019

I just had this occur on 2.3.0 on Ubuntu 18.04.1.

@Ctucker9233 Ctucker9233 reopened this Jan 22, 2019

@Ctucker9233

This comment has been minimized.

Copy link

Ctucker9233 commented Jan 22, 2019

Not fixed as of 2.2.7.

@engcom-backlog-nazar

This comment has been minimized.

Copy link
Contributor

engcom-backlog-nazar commented Jan 22, 2019

@Ctucker9233 this fix will be available on 2.2.8 release and 2.3.1 release

@engcom-backlog-nazar

This comment has been minimized.

Copy link
Contributor

engcom-backlog-nazar commented Jan 22, 2019

@smadasam

This comment has been minimized.

Copy link

smadasam commented Jan 26, 2019

I am having the same issue on my 2.3.0 site. How do I patch it until the next release comes out?

:/tmp/analytics# ll -ah
total 62G
drwxrwxr-x 2 sw_temp psacln 4.0K Jan 25 10:02 .
drwxrwxrwt 11 root root 20K Jan 26 00:41 ..
-rw-rw-r-- 1 sw_temp psacln 101K Jan 22 10:00 ~tmp-1548151203.3161tar.tar
-rw-rw-r-- 1 sw_temp psacln 71M Jan 23 10:00 ~tmp-1548237603.0528tar.tar
-rw-rw-r-- 1 sw_temp psacln 41G Jan 24 10:02 ~tmp-1548324002.9252tar.tar
-rw-rw-r-- 1 sw_temp psacln 22G Jan 25 10:02 ~tmp-1548410402.865tar.tar

@GreenGinkgo

This comment has been minimized.

Copy link

GreenGinkgo commented Jan 29, 2019

Just as a note, storing data in /tmp is misguided.

Shared servers that have a single /tmp would be co-mingling different user data together. I suggest that this data be stored within the Magento installation itself. Maybe in (magento)/var/analytics or similar.

@smadasam

This comment has been minimized.

Copy link

smadasam commented Feb 1, 2019

@magento magento deleted a comment from engcom-backlog-nazar Feb 6, 2019

@srenon

This comment has been minimized.

Copy link
Member

srenon commented Feb 8, 2019

Just upgraded to 2.2.7 and run into this issue with over 50gb of diskspace disappear in 2 days.

If you are not using this feature, you can disable "Advanced REporting Service" in admin and flush cache.

Stores > Configuration > General > Advanced Reporting

image

@DanielRuf

This comment has been minimized.

Copy link
Member

DanielRuf commented Feb 8, 2019

@DanielRuf

This comment has been minimized.

Copy link
Member

DanielRuf commented Feb 8, 2019

I'm locking this issue as it is already fixed in the next releases and we have linked the patches and all needed steps to either disable Advanced Reporting or apply the patches.

@magento magento locked as resolved and limited conversation to collaborators Feb 8, 2019

@DanielRuf

This comment has been minimized.

Copy link
Member

DanielRuf commented Feb 8, 2019

Shared servers that have a single /tmp would be co-mingling different user data together.

They have a much more critical issue - a security issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.