Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

readMeta fails to read Landsat MSS metadata (*MTL.txt) files #7

Closed
aszeitz opened this issue May 25, 2016 · 31 comments
Closed

readMeta fails to read Landsat MSS metadata (*MTL.txt) files #7

aszeitz opened this issue May 25, 2016 · 31 comments

Comments

@aszeitz
Copy link

aszeitz commented May 25, 2016

Hi,

Thanks bleutner for this package, it has been very helpful so far in processing Landsat scenes!

An issue I have encountered is that readMeta fails to properly load the *MTL.txt metadata files from the various Landsat MSS scenes I have tried to process.

Here is a basic example of trying to load the metadata and the error it throws:

meta <- readMeta("./MSS/1978/LM30520251978217PAC03_MTL.txt")

Error in `rownames<-`(`*tmp*`, value = "B6_dn") : 
attempt to set 'rownames' on an object with no dimensions

and if I use the example you provide in the documentation, readMeta throws a different error because mtlFile is only an empty character vector.

mtlFile <- system.file("./MSS/1978/LM30520251978217PAC03_MTL.txt", package = "RStoolbox")
print(mtlFile)
[1] ""

meta <- readMeta(mtlFile)

Error: Metadata file does not exist. Looking for:

I have attached one of the metadata .txt files as obtained when the MSS scene was downloaded from the USGS. Any suggestions on how to get around this issue if I am trying to run radCor and topCor on the MSS data?

LM30520251978217PAC03_MTL.txt

Thanks,
Andras
@bleutner
Copy link
Owner

bleutner commented Jun 1, 2016

Hi Andras,

thanks for reporting. I think it should work now if you install the current Github version.

Out of interest: where did you download this data from? I wasn't aware that USGS are now offering reflectance scaling coefficients for MSS data, which is pretty cool. I tried to find it on EarthExplorer but couldn't.

Let me know if it works

@aszeitz
Copy link
Author

aszeitz commented Jun 1, 2016 •
edited

Hi,

Thank you for your work to provide the update. readMeta does indeed work with MSS metadata now. However, another issue popped up as I re-ran my processing script with the MSS data. radCor only converts the first of the four MSS bands, regardless of whether any other processing has been done, like topCor or if it's directly fed from stackMeta.

For example:

mss_sref <- radCor(mss_stack, mtl, method = "costz", bandSet = "full",
                                verbose = TRUE)

Bands to convert to reflectance: B4_dn
Excluding bands: B5_dn, B6_dn, B7_dn
hazeValues was not provided -> Estimating hazeValues automatically
hazeValues estimated as: 19
Selecting atmosphere: 'moderate'
Processing radiance / reflectance

Explicitly setting the bands for bandSet makes no difference either. I'm unsure if this might be intentional (if there is some reason that invalidates the other bands from having radiometric corrections applied), or a bug, or user error. On a side note, I also applied topCor to the MSS data and it worked fine.

As for your question, I downloaded the MSS data from the USGS's bulk data downloader:
Landsat Data Bulk Download *login required

And for the Data Set: "Landsat Archive:L1-5 MSS"
and I provided a Scene List with the scene ID's.

@bleutner
Copy link
Owner

bleutner commented Jun 2, 2016

There was a look-up mismatch with the haze look-up tables which expected bands 1 to 4, but in Landsat 1-3 these bands are designated bands 4 to 7...
It should be fixed now.

@aszeitz
Copy link
Author

aszeitz commented Jun 3, 2016

It works great now. Thank you very much!

@aszeitz aszeitz closed this as completed Jun 3, 2016
@sethilana
Copy link

I have tried installing RStoolbox as a zip file from github and it installed but when I load the MTL file I get an empty "" FILE. Iam using R version 3.3.4 in windows 10. Please help
Sethi

@bleutner
Copy link
Owner

Could you please provide the MTL file which doesn't work or are you referring to the one in this issue?

@sethilana
Copy link

sethilana commented Jan 30, 2017 via email

@sethilana
Copy link

I wish I could open these in Rstudio and start working on my analysis

@bleutner
Copy link
Owner

Both files are read just fine, even on R-devel in Windows.
Something must be wrong with your installation. Are you sure you followed the installation instructions correctly?

@sethilana
Copy link

sethilana commented Jan 30, 2017 via email

@bleutner
Copy link
Owner

I don't understand your last question.

Why don't you use the regular release of R (3.2.2)? Then you can install directly from CRAN, i.e. with install.packages(RStoolbox), without needing to compile it yourself.

@sethilana
Copy link

sethilana commented Jan 30, 2017 via email

@bleutner
Copy link
Owner

oh sorry, thats what I meant. 3.3.2

@sethilana
Copy link

sethilana commented Jan 30, 2017 via email

@sethilana
Copy link

sethilana commented Jan 30, 2017 via email

@sethilana
Copy link

sethilana commented Jan 31, 2017 via email

@bleutner
Copy link
Owner

I see. The system.file function is only needed to load the built-in example file.
What happened was that you were requesting a file which did not exist in the installation directory and hence system.file returned an empty character: ""

All you have to do is provide the path to your file without the system.file function:
readMeta("path/to/your/file_MTL.txt")

@sethilana
Copy link

sethilana commented Jan 31, 2017 via email

@sethilana
Copy link

sethilana commented Jan 31, 2017 via email

@bleutner
Copy link
Owner 

readMeta("C:/Users/Sethi/Documents/R/Landsat/Juneimages/LC81710742015170LGN00/LC81710742015170LGN00_MTL.txt")

@sethilana
Copy link

Thanx it has imported

@sethilana
Copy link

sethilana commented Jan 31, 2017 via email

@sethilana
Copy link

sethilana commented Feb 6, 2017 via email

@bleutner
Copy link
Owner

bleutner commented Feb 7, 2017

Please open a new issue. Every issue should address only one single problem.
Also please provide a complete, reproducible example, including the original error messages.

@Romaysa
Copy link

Romaysa commented Apr 4, 2017

Dear Bleutner,
hope you don't mind writing you here so new in Github and i coulden't find how to open a new issue, any how this is my problem have a look to the errors. same problem as Sethi had, the metadata file is not existed.
mtfile<- system.file("MTL.txt", package = "RStoolbox")
meta<- readMeta("Z:/Band/MTL.txt")

i can read the MTL file perfecctly but when i would like to read the mtlfile so these error happen
lsat<- stackMeta(mtfile)

Error: Metadata file does not exist. Looking for:

lsat<- stackMeta("Z:/Band/MTL.txt")

Error in .local(.Object, ...) :

Error in .rasterObjectFromFile(x, band = band, objecttype = "RasterLayer", :

Cannot create a RasterLayer object from this file. (file does not exist)

lsat<- stackMeta(MTL.txt)

Error in inherits(file, "ImageMetaData") : object 'MTL.txt' not found

lsat<- stackMeta(meta)

Error in .local(.Object, ...) :

Error in .rasterObjectFromFile(x, band = band, objecttype = "RasterLayer", :
Cannot create a RasterLayer object from this file. (file does not exist)

@sdmed
Copy link

sdmed commented Sep 1, 2018

Hello everyone
I have trying to do radiometric and topographic correction for the Land Sat data using R.
However, when I call the meta data in R as you can see below, I have the same problem posted here that Meta data file does not exist . Looking for:

I am using the latest R version 3.5.1 in window.
I went through the discussion you had but didn't understand what is the solution at the end
Please kindly help me to fix the error. I have also attached the metadata i have.

Thank you.
LC08_L1TP_168050_20161225_20170315_01_T1_MTL.txt

library(raster)
Loading required package: sp

mtlFile <- system.file("G:/LandSat_processing_practice_in_R/LC08_L1TP_168050_20161225_20170315_01_T1/LC08_L1TP_168050_20161225_20170315_01_T1_MTL.txt", package="RStoolbox")
metaData <- readMeta(mtlFile)
Error: Metadata file does not exist. Looking for:

@aszeitz
Copy link
Author

aszeitz commented Sep 1, 2018

I downloaded your metadata file. The following works for me to import it in R version 3.4.1:

library(RStoolbox)
setwd("set/to/dir/where/metadata/exists")
meta <- readMeta("./LC08_L1TP_168050_20161225_20170315_01_T1_MTL.txt")

@sdmed
Copy link

sdmed commented Sep 4, 2018

Dear Aszeitz

Thank you very much indeed. Now i am able to import the meta data using the step you mentioned even in 3,5,1 version. I think the error is when i use the system.file function as i put in my earlier post. Now it is okay to import. Following this i was able to execute the radiometric correction using the command below but could not able to export the corrected stacked landSat image for later use in like in Arcgis ... Here is the step i went through and please kindly help me on how to export the final output as raster dataset to be able to use it outside R.

rm(list = ls())

library(raster)
library(RStoolbox)
library(devtools)
library(rgdal)

#Set the working directory
setwd("G:/LandSat_processing_practice_in_R/LC08_L1TP_168050_20161225_20170315_01_T1")

metaData <- readMeta("./LC08_L1TP_168050_20161225_20170315_01_T1_MTL.txt")

#To stack the different bands (layer stacking)
lsat <- stackMeta(metaData)
summary(lsat)
print(lsat)

#Radiometric correction
lsat_radcor<- radCor(lsat, metaData, method = "apref", bandSet = "full", hazeValues, hazeBands, atmosphere, darkProp = 0.01, clamp = TRUE)
summary(lsat_radcor)

writeGDAL(lsat_radcor, "G:/LandSat_processing_practice_in_R/lsat.tif", drivername="GTiff")

Thank you very much.

@aszeitz
Copy link
Author

aszeitz commented Sep 4, 2018

To quote bleutner from Feb 2017:

Please open a new issue. Every issue should address only one single problem.
Also please provide a complete, reproducible example, including the original error messages.

Your problem with exporting the raster stack does not belong in this thread, which refers specifically to issues in loading metadata. Furthermore, your issue with writeGDAL is separate from RStoolbox, as that function is from the rgdal library. I suggest you look into the using the writeRaster function in the raster library, or search for solutions / ask a question on stackoverflow.

@beulahevelyn
Copy link

Hi,

mtlFile <- readMeta("./LC08_L2SP_142051_20180107_20200902_02_T1_MTL.txt"), throws a POSIXct Error. Installed (RStoolbox) and (raster) packages. Error seems to point to ACQUISITION_DATE field of MTL.txt. Can someone help?

Error Message: Error in as.POSIXct.default(if (!legacy) meta$PRODUCT_METADATA["DATE_ACQUIRED", :
c("do not know how to convert 'if (!legacy) meta$PRODUCT_METADATA["DATE_ACQUIRED", ] else meta$PRODUCT_METADATA["ACQUISITION_DATE", ' to class “POSIXct”", "do not know how to convert ' ]' to class “POSIXct”")

@marcelocarvalhoalves
Copy link

I got the same error of beulahevelyn trying to read metadata of Landsat8. I think this error is due to NASA reconfiguration of this file sequence and some terms used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@bleutner @aszeitz @beulahevelyn @sethilana @Romaysa @sdmed @marcelocarvalhoalves