calcjob test failure with sqlite backend

[danielhollas]

I was trying to run the test suite using a profile setup with verdi presto, i.e. sqlite backend and no broker.
The only thing needed make this work was to comment auto the redefined aiida_profile fixture in tests/conftest.

Obviously many tests fail in this setup as they either require rabbitMQ or PSQL backend. But one test failure caught my attention as rather suspicious as it is not clear to me why it should fail, specifically test cmdline/commands/test_calcjob.py::TestVerdiCalculation::test_calcjob_cleanworkdir

    def test_calcjob_cleanworkdir(self):
        """Test verdi calcjob cleanworkdir"""
        # Specifying no filtering options and no explicit calcjobs should exit with non-zero status
        options = []
        result = self.cli_runner.invoke(command.calcjob_cleanworkdir, options)
        assert result.exception is not None
    
        # Without the force flag it should fail
        options = [str(self.result_job.uuid)]
        result = self.cli_runner.invoke(command.calcjob_cleanworkdir, options)
        assert result.exception is not None
    
        # With force flag we should find one calcjob
        options = ['-f', str(self.result_job.uuid)]
        result = self.cli_runner.invoke(command.calcjob_cleanworkdir, options)
        assert result.exception is None, result.output
    
        # Do it again should fail as the calcjob has been cleaned
        options = ['-f', str(self.result_job.uuid)]
        result = self.cli_runner.invoke(command.calcjob_cleanworkdir, options)
>       assert result.exception is not None, result.output
E       AssertionError: Success: 1 remote folders cleaned on localhost
E         
E       assert None is not None
E        +  where None = <Result okay>.exception

The failure suggest that cleaning a working directory of a calcjob does not work as expected?

[sphuber]

This is due to the fact that not all querybuilder syntax is supported by the sqlite implementation. In this case particularly, the has_key filter does not work properly. The test that fails relies on the following query not returning anything (aiida.orm.utils.remote:get_calcjob_remote_paths):

    if only_not_cleaned is True:
        filters_remote['or'] = [
            {f'extras.{RemoteData.KEY_EXTRA_CLEANED}': {'!==': True}},
            {'extras': {'!has_key': RemoteData.KEY_EXTRA_CLEANED}},
        ]

not returning anything. But for the sqlite implementation it ignores that statement.

I am not sure if the implementation could in principle support this filtering key. Haven't looked into it in too much detail either, so might still be worth a shot.

[sphuber]

Closing as duplicate of #6256

[danielhollas]

One thing that makes me worried here: This seems to be a silent error? Should we raise explicitly if user tries to use code that uses this unsupported feature?

[sphuber]

That would perhaps be a necessary stop-gap solution until it can be implemented for sqlite, if ever. Let's discuss that in #6256