File in gem is causing security scanner issues

[matobinder]

So we have a application that gets built, that includes mini_portile, and it runs through a security scanner before being deployed.

The scanner gets upset by this file:

/test/assets/test-download-archive.tar.gz

As, it it looks like a tar.gz file, but is really just plain text file
Does this file really need to get delivered with the gem?

For now, after I bundle install it I am just going to delete the file from my vendored dir

[flavorjones]

Hi, @matobinder. Thanks for opening this issue, and sorry you're having this problem.

It's a common convention to include tests in a rubygem package, and occasionally I've had conversations with downstream packagers (e.g., Debian package managers) about how they have commonly-used scripts that rely on tests being present (to validate the final package). But we could probably remove the tests if we judge that the inconvenience by packagers is outweighed by solving this problem.

Can you help me understand why you're having this problem? How is your application using mini_portile? Is it getting pulled in via a dependency from another gem (like nokogiri)?

[matobinder]

Yeah its getting pulled in via nokogiri
The issue really is the scanning tool, sees teh tar.gz extension, and wants to scan inside it. However since its not really a tar.gz file, it chokes and errors. I'm going to see if I can reach with the folks on the scanning tool to better handle this.

I"m not sure how that file is used, but basically its not really a tar.gz file. Which is causing the issue we are seeing

[flavorjones]

@matobinder OK, that's helpful to understand, thanks.

We have a few options:

1. make it a real tarball (easy!)
2. you could remove mini_portile2 from your deployment artifact (since it's only needed at install time)
3. you could use a native precompiled version of nokogiri that doesn't depend on mini_portile2

I'm happy to do 1, but you might want to try 3 because there's no reason to not use the precompiled version if you can.

[matobinder]

I was looking at trying out option #1, looking at how it ran, I figured I could turn it into a real tar.gz file it would be fine. But having some issues getting unit tests to work on my host. Anything special that needs to be doen before handle other than basically a "bundle install; rake"?

As for option #2, the way we have our CICD deploy pipeline kind of makes that difficult. I can do that, but we need to make a slight change to support this. (We probably will end up supporting the capability to do this, as I see this won't be the only time we have a issue like this)

Option #3 is interesting. I'll have to look into that. I normally just add nokogiri to my Gemfile, and let bundler take care of it.

If I can figure out how to get unit tests running, I"d be happy to do a PR for the option number 1, but I figure you can do that pretty quick too. I"m kind of curious as to what is failing on the unit tests for me. Not knowing the tests, its a bit hard to know which are "normal" errors. Basically its fails with this

Activating libiconv 1.15 (from ports/x86_64-redhat-linux/libiconv/1.15)...
2 retrie(s) left for sqlite-autoconf-3350400.tar.gz
1 retrie(s) left for sqlite-autoconf-3350400.tar.gz
0 retrie(s) left for sqlite-autoconf-3350400.tar.gz
SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)
Extracting sqlite-autoconf-3350400.tar.gz into tmp/x86_64-redhat-linux/ports/sqlite3/3.35.4... ERROR, review '/home/gschoep/git/mini_portile/examples/tmp/x86_64-redhat-linux/ports/sqlite3/3.35.4/extract.log' to see what happened. Last lines are:
========================================================================
tar (child): ports/archives/sqlite-autoconf-3350400.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
gtar: Child returned status 2
gtar: Error is not recoverable: exiting now

Anyways, I'll see how option #3 if I can make it work. But I'd love it if we could do option #1.
Might end up helping someone else out in same case I have.

[flavorjones]

The error you're seeing:

SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)

looks like your machine may not have up-to-date CA certificates and so downloading the sqlite tarball is failing because SSL can't verify the cert.

But you don't need to run the full test suite. You can just run the unit tests with rake test:unit and it will skip the integration ("examples") tests.

[flavorjones]

I just created a PR at #109

[flavorjones]

I've released v2.7.1 with this change. Please let me know whether it works for you!

[matobinder]

Will check it out Thanks