Sometimes getting the error while executing Pentaho Job. Unable to create lock file: <KETTLE_HOME_DIRECOTRY_PATH>\.kettle\environment\metastore\.lock

[rajkumarvenkatasamy]

Environment detail:

PDI Version 9 (Community Edition)

Kettle Environment plugin : 1.7.1

Details

At random manner, getting the below error while executing pentaho job. This error especially occurs in any of the Mapping - Sub transformation steps that exists in the project. The frequency of this error appearing is so far random.

Note: The same job runs successfully without any errors at times as well by making calls to the same mapping - sub transformation steps.

`--- Log Snippet starts ---

2020/12/15 12:04:28 - Populate_fact_table.0 - Connected to database [dw] (commit=10000)
2020/12/15 12:04:28 - Populate_fact_table.1 - Connected to database [dw] (commit=10000)
2020/12/15 12:04:28 - Populate_fact_table.2 - Connected to database [dw] (commit=10000)
2020/12/15 12:04:28 - Populate_fact_table.3 - Connected to database [dw] (commit=10000)
2020/12/15 12:04:28 - t_fn_get_dw_biz_entity_skey - Validating active environment 'PTP'
2020/12/15 12:04:28 - t_fn_get_dw_emp_skey - Validating active environment 'PTP'
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : Unable to load the mapping transformation because of an error : org.pentaho.di.core.exception.KettleException:
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Unable to prepare (allocate, initialize) the execution of the mapping (sub-transformation)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 -
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Validation error against transformation 'D:\pdi\data-integration\workspace\analytics_etl/common/app_framework_utils/get_dim_skeys/t_fn_get_dw_scope_skey.ktr' in active environment
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Unable to create lock file: D:\pdi\data-integration\workspace\analytics_etl.kettle\environment\metastore.lock
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : org.pentaho.di.core.exception.KettleException:
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Unable to prepare (allocate, initialize) the execution of the mapping (sub-transformation)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 -
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Validation error against transformation 'D:\pdi\data-integration\workspace\analytics_etl/common/app_framework_utils/get_dim_skeys/t_fn_get_dw_scope_skey.ktr' in active environment
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Unable to create lock file: D:\pdi\data-integration\workspace\analytics_etl.kettle\environment\metastore.lock
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 -
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 -
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.di.trans.steps.mapping.Mapping.prepareMappingExecution(Mapping.java:294)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.di.trans.steps.mapping.Mapping.init(Mapping.java:582)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.di.trans.step.StepInitThread.run(StepInitThread.java:69)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at java.lang.Thread.run(Unknown Source)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Caused by: org.pentaho.di.core.exception.KettleException:
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Validation error against transformation 'D:\pdi\data-integration\workspace\analytics_etl/common/app_framework_utils/get_dim_skeys/t_fn_get_dw_scope_skey.ktr' in active environment
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Unable to create lock file: D:\pdi\data-integration\workspace\analytics_etl.kettle\environment\metastore.lock
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 -
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.kettle.env.xp.TransStartCheckEnvironmentExtensionPoint.callExtensionPoint(TransStartCheckEnvironmentExtensionPoint.java:43)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.di.core.extension.ExtensionPointMap.callExtensionPoint(ExtensionPointMap.java:151)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.di.core.extension.ExtensionPointHandler.callExtensionPoint(ExtensionPointHandler.java:45)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.di.trans.Trans.prepareExecution(Trans.java:770)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.di.trans.steps.mapping.Mapping.prepareMappingExecution(Mapping.java:292)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - ... 3 more
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Caused by: org.pentaho.metastore.api.exceptions.MetaStoreException: Unable to create lock file: D:\pdi\data-integration\workspace\analytics_etl.kettle\environment\metastore.lock
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.metastore.stores.xml.XmlMetaStore.lockStore(XmlMetaStore.java:672)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.metastore.stores.xml.XmlMetaStore.getElementTypes(XmlMetaStore.java:169)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.metastore.stores.xml.XmlMetaStore.getElementTypes(XmlMetaStore.java:163)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.metastore.stores.xml.XmlMetaStore.getElementTypeByName(XmlMetaStore.java:239)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.metastore.stores.xml.XmlMetaStore.getElementTypeByName(XmlMetaStore.java:41)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.metastore.persist.MetaStoreFactory.loadElement(MetaStoreFactory.java:89)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.kettle.env.util.EnvironmentUtil.validateFileInEnvironment(EnvironmentUtil.java:295)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.kettle.env.xp.TransStartCheckEnvironmentExtensionPoint.callExtensionPoint(TransStartCheckEnvironmentExtensionPoint.java:41)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - ... 7 more
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - Caused by: java.io.IOException: Access is denied
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at java.io.WinNTFileSystem.createFileExclusively(Native Method)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at java.io.File.createNewFile(Unknown Source)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - at org.pentaho.metastore.stores.xml.XmlMetaStore.lockStore(XmlMetaStore.java:668)
2020/12/15 12:04:28 - t_fn_get_dw_scope_skey.0 - ... 14 more

--- Log Snippet ends ---`

[mattcasters]

I've hit it in the past as well. It's most likely caused by some background process in PDI itself not this plugin. When I ported this plugin to Hop on the new metadata back-end the issue completely went away.

[rajkumarvenkatasamy]

Hi Matt,

Thanks for getting back.

Any idea on how to prevent this issue altogether with PDI setup ?

[mattcasters]

None at all. Our inability (read: not allowed) to fix bugs in the core of PDI led to hop.apache.org

[mattcasters]

I guess thinking about it we could reduce locking by only checking the environment to which a transformation/job belongs to for top-level files, not its children.