Skip to content

SOLR-15950: Fix auto-creation of filestore in constructor.#653

Merged
HoustonPutman merged 3 commits into
apache:mainfrom
HoustonPutman:filestore-dynamic
Feb 21, 2022
Merged

SOLR-15950: Fix auto-creation of filestore in constructor.#653
HoustonPutman merged 3 commits into
apache:mainfrom
HoustonPutman:filestore-dynamic

Conversation

@HoustonPutman
Copy link
Copy Markdown
Contributor

@HoustonPutman HoustonPutman commented Feb 18, 2022

https://issues.apache.org/jira/browse/SOLR-15950

The filestore lazy-creation change made previously didn't actually make it lazy, since the directory is ensured in the constructor of the class. By merely removing that constructor call/method, it is now lazy.

Note that I am also removing the need to create a SolrCore in the Managed Resource test. This is completely unrelated, but I saw that it wasn't needed when debugging why the test was failing. Can split this into a separate PR.

@risdenk
Copy link
Copy Markdown
Contributor

risdenk commented Feb 18, 2022

I can confirm this at least makes org.apache.solr.rest.TestManagedResourceStorage pass which has been failing on main.

./gradlew :solr:core:test --tests "org.apache.solr.rest.TestManagedResourceStorage" -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=8833B2251FFB9A75 -Ptests.badapples=false -Ptests.file.encoding=UTF-8

Checked on main again and on this pr - passes with changes from this PR.

@HoustonPutman
Copy link
Copy Markdown
Contributor Author

Opened a separate PR for the test cleanup: #654

try {
Files.createDirectories(path);
log.info("Created filestore folder {}", path);
} catch (IOException e) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we catch AccessControlException here too?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ambivalent, it'll be thrown anyways and honestly shouldn't happen in a real deployment. Feel free to add it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is thrown by Security Manager, right? Which is default enabled in 9.0 (but deprecated in JDK17). The path that was denied was

java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/core/build/resources/test/solr/filestore" "write")

So obviously our code tried to create that folder in the "build/resources/test/solr/" folder, which must have been disallowed by our solr-tests.policy

If we need to check for AccessControlException here, we'd likely need the same in 100 other locations across our code base?

Copy link
Copy Markdown
Contributor

@janhoy janhoy Feb 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The policy already have this line

permission java.io.FilePermission "${solr.solr.home}", "read,write,delete,readlink";

So that should be allowed -- if sysProp solr.solr.home is set to the same location as solrHome in the test that failed (TestManagedResourceStorage) that is, and I guess it is not...

Anyway, this PR avoids that now.

@janhoy
Copy link
Copy Markdown
Contributor

janhoy commented Feb 18, 2022

I started looking into it but reverted original commit on main. Then I learned about this PR, which is of course now broken. I'll un-revert again to enable this PR as a solution instead.

@janhoy
Copy link
Copy Markdown
Contributor

janhoy commented Feb 19, 2022

I would also expect enable.packages=false to avoid loading DistribPackageStore completely, but it may be needed for other features than pacakge loading?

@dsmiley
Copy link
Copy Markdown
Contributor

dsmiley commented Feb 19, 2022

The filestore is its own feature that could be used by something other than packages.

@janhoy
Copy link
Copy Markdown
Contributor

janhoy commented Feb 19, 2022

The filestore is its own feature that could be used by something other than packages.

Is the DistribPackageStore class ill named and should have been DistributedFileStore?

@dsmiley
Copy link
Copy Markdown
Contributor

dsmiley commented Feb 19, 2022

Is the DistribPackageStore class ill named and should have been DistributedFileStore

Yes absolutely (including the interface PackageStore). The re-imagination of the distributed package store as the general purpose file store came relatively late in the new package system design, so the rename didn't happen yet. Now would be a great time to make this change.

CC @noblepaul

@janhoy
Copy link
Copy Markdown
Contributor

janhoy commented Feb 21, 2022

Is the DistribPackageStore class ill named and should have been DistributedFileStore

Yes absolutely (including the interface PackageStore). The re-imagination of the distributed package store as the general purpose file store came relatively late in the new package system design, so the rename didn't happen yet. Now would be a great time to make this change.

CC @noblepaul

Spun this off to https://issues.apache.org/jira/browse/SOLR-16038 - @noblepaul do you want to pick that up?

Is this PR ready?

Copy link
Copy Markdown
Contributor

@janhoy janhoy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked out the branch, started Solr on empty SOLR_HOME, looks good.

@HoustonPutman HoustonPutman merged commit 03efd00 into apache:main Feb 21, 2022
@HoustonPutman HoustonPutman deleted the filestore-dynamic branch February 21, 2022 21:53
HoustonPutman added a commit that referenced this pull request Feb 21, 2022
HoustonPutman added a commit that referenced this pull request Feb 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants