Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleantest blows up /tmp with files #21

Closed
NucciTheBoss opened this issue Dec 18, 2022 · 2 comments · Fixed by #36
Closed

Cleantest blows up /tmp with files #21

NucciTheBoss opened this issue Dec 18, 2022 · 2 comments · Fixed by #36

Comments

@NucciTheBoss
Copy link
Owner

Cleantest blows up the /tmp directory on Linux systems with the various pickles, scriptlets, and tarballs that it needs to upload into the test environment providers. /tmp is emptied out after every reboot, but that does not do you much good if you almost never reboot your system.

As such, rather than just getting the name of the temporary directory on the host, instead use a named temporary file created with the tempfile module. This should cut down on the amount of space being consumed by cleantest on /tmp. Lastly, this will allow for better organization of files within the test environment.

@NucciTheBoss
Copy link
Owner Author

Looking at this issue, I have realized that much more work is need to improve how cleantest handles data.

I came to this revelation after experimenting with pickles and JSON packets. The TL;DR is that I found that I would like much more for cleantest to send out "REST" requests rather than directly uploading into the test environment instances. I am thinking something more along the likes of objects dumping json packets that can be processed by cleantest rather than just raw Python scripts. Needs some careful thought], but I believe I can use a mixture of pickles and base64 encoding to accomplish this.

@NucciTheBoss
Copy link
Owner Author

I decided on a new data model for cleantest, rather than uploading raw pickles. I am instead going to use JSON packets. I will still use pickles (it's a good data format; haters gonna hate), but encode them using base64. A packet being sent from the host to the test environment instance will look like the following:

{
    "checksum": "Some sha224 hash",
    "data": "base64 encoded pickle",
    "injectable": "Python script that tells the test instance what to do with the object its just received",
}

The packet sent back from the test environment instance will look like the following:

{
    "checksum": "Some sha224 hash",
    "result": "base64 encoded Result object",
}

The idea behind this "data model" is to eliminate the dependence on dumping objects directly into /tmp.

NucciTheBoss added a commit that referenced this issue Jan 17, 2023
Notes:
* Rather than dumping out to `/tmp`, cleantest now encodes itself and dependencies as base64 objects that are then decoded in the test environment instances. This is an improvement on just dumping out a tar archive out to `/tmp` and then uploading that archive to the test environment instance.
NucciTheBoss added a commit that referenced this issue Jan 18, 2023
Notes:
* Wrapping NamedTemporaryFile with Path messed with the context manager, causing the file to linger past the files use. This created the same problem as before where /tmp would get blown up. I plugged the leak by wrapping the NamedTemporaryFile after creating it with the context manager.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant