columns

rename a column

• https://docs.databricks.com/en/delta/column-mapping.html

_metadata virtual field

• built-in (albeit hidden) _metadata is a STRUCT column.
• access by SELECT _metadata from <table-name>
• It contains
  • _metadata.file_path
  • _metadata.file_name
  • _metadata.file_size
  • _metadata.file_block_start
  • _metadata.file_block_length
  • _metadata.file_modification_time
  • _metadata.row_index

Rescued Data Column _rescued_data: rescue malformed rows on ingestion

• provided by spark read_files(), spark spark.read or Auto Loader
• when raw data does not match the schema