Skip to content

FAQs

Jump to bottom
Bert Verbeek edited this page Apr 9, 2026 · 2 revisions

Frequently Asked Questions (FAQs)

Find answers to the most common questions about bc2adls. For additional questions, visit our GitHub Issues page.

📋 Table of Contents

General Questions

What is bc2adls?

bc2adls is an open-source tool that exports incremental data from Dynamics 365 Business Central to Microsoft Fabric or Azure Data Lake Storage. It supports three export options:

  1. Azure Data Lake with Synapse pipelines
  2. Microsoft Fabric with notebooks
  3. Microsoft Fabric Open Mirroring (native)

Which export option should I choose?

Choose Azure Data Lake if:

  • You need CDM format support
  • You have complex transformation requirements
  • You're already using Azure Synapse

Choose Microsoft Fabric with notebooks if:

  • You prefer modern lakehouse architecture
  • You want simpler setup than Azure Synapse
  • You don't need CDM format

Choose Fabric Open Mirroring if:

  • You want the absolute simplest setup
  • You're OK with table-level (not field-level) selection
  • You prefer a fully managed solution

Read detailed comparison

Is bc2adls free?

The bc2adls extension and code are free and open source. However, you'll incur costs for:

  • Azure Data Lake Storage
  • Azure Synapse Analytics (if using)
  • Microsoft Fabric capacity (if using)
  • Business Central licenses

Who maintains bc2adls?

Originally created by Microsoft (Soumya Dutta and Henri Schulte), bc2adls is now maintained by the community, led by Bert Verbeek at 4PS.

Setup and Configuration

📖 For detailed explanations of all configuration options, see the Configuration Reference page.

How do I run exports on a recurring schedule?

Use Business Central's Job Queue feature:

Option 1: Simple scheduling (all tables)

  1. Open Export to Azure Data Lake Storage page
  2. Click Schedule export action
  3. A Job Queue Entry is created
  4. Configure frequency in Job Queue

Option 2: Advanced scheduling (by category or table)

Use Report ADLSE Schedule Task Assignment for granular control:

  1. Open Job Queue Entries
  2. Create a new entry
  3. Set Object Type to Run: Report
  4. Set Object ID to Run: 82561 (ADLSE Schedule Task Assignment)
  5. Click Report Request Page Options
  6. Select:
    • Specific Categories to export, or
    • Specific Table IDs to export

Report Options

This allows different tables to export at different frequencies.

I have multiple BC environments in the same tenant. How should I organize data?

Recommended approach:

  • ✅ One data lake container per BC environment
  • ✅ Keep production and sandbox data separate
  • ✅ Use consistent naming conventions

After copying environments:

  1. Update the setup page in the new environment
  2. Point to a different container
  3. Verify configuration before running exports

How do I export data from multiple companies?

Exports run only for the current company. To export from multiple companies:

Option 1: Manual switching

  1. Switch to a company in BC
  2. Run the export
  3. Repeat for each company

Option 2: Scheduled exports (Recommended)

  1. Create Job Queue Entries for each company
  2. Configure the Run in Company field on each entry
  3. Schedule at appropriate times

Important:

  • Export the schema first from all companies
  • All company data goes to the same container (differentiated by company field)

Can I use the same storage for multiple BC environments?

⚠️ Not recommended. Best practice:

  • Different containers for different environments
  • Prevents data mixing
  • Simplifies troubleshooting
  • Clearer security boundaries

Export Operations

How do I know when an export is complete?

Check the Export to Azure Data Lake Storage page:

  1. Look at the Last exported status column
  2. When all tables show a status other than "In process", the export is complete
  3. Possible statuses:
    • ✅ Success: Completed successfully
    • 🔴 Failed: Error occurred
    • 🕐 In process: Currently running

Can I export tables at different frequencies?

Yes! Use the category feature:

  1. Assign tables to categories:
    • Example: "Hourly", "Daily", "Weekly"
  2. Create separate Job Queue Entries for each category
  3. Configure different schedules for each entry
  4. Use Report ADLSE Schedule Task Assignment to specify categories

Use cases:

  • Export transactional tables (orders, invoices) hourly
  • Export master data (customers, items) daily
  • Export configuration tables weekly

What happens if an export fails?

When an export fails:

  1. Status changes to "Failed"
  2. Error message is recorded
  3. Other tables continue to export
  4. Delta files are not uploaded for failed table
  5. Next export will retry

To resolve:

  1. Check the error message
  2. Fix the underlying issue
  3. Run export again
  4. Only failed table will re-export (incremental)

Can I pause an export in progress?

⚠️ No direct pause mechanism. You can:

  1. Disable tables you don't want to export
  2. Wait for current sessions to complete
  3. Exports are parallel and run in separate sessions

Data and Fields

Can I export calculated fields?

No. Only persistent fields on BC tables can be exported.

Workaround:

  1. Create a new table with calculated values
  2. Populate via AL code (e.g., on record insert/modify)
  3. Export the new table

See issue #88 for details

Can I export BLOB data?

No. BLOB fields are not supported directly.

Workaround:

  1. Convert BLOB to text using Base64 Convert codeunit
  2. Store in a new text field or separate table
  3. Export the text field

Which field types are NOT supported?

The following field types cannot be exported:

  • ❌ BLOB fields
  • ❌ Flow fields (calculated)
  • ❌ Filter fields
  • ❌ Fields marked Obsolete: Removed
  • ❌ Fields with Access property other than Public

What should I do when a field I was exporting becomes obsolete?

When a field is obsoleted:

  1. Before upgrade:

    • Review obsoletion documentation
    • Identify replacement fields
    • Update your data model

  2. During upgrade:

    • Disable the obsolete field from export
    • Enable the new replacement field
    • Test the export

  3. After upgrade:

    • Schema change will occur
    • Archive old data if needed
    • Update pipelines/queries to use new fields
    • Map old data to new schema if necessary

Note: Including pending obsoletion fields will cause upgrade errors when the field is fully removed.

Why does SystemCreatedAt show 01 Jan 1900?

Explanation:

  • Records created before the SystemCreatedAt audit field was introduced have NULL values
  • bc2adls sets an artificial value of 01 Jan 1900 for these records
  • This ensures consistency regardless of timezone

Impact:

  • Filter out these records if time-based analysis is critical
  • Use other date fields for accurate historical tracking

Performance and Scaling

My export job is timing out! What should I do?

Timeout issues can occur in two places:

1. Query Timeout (Row Version Sorting)

Symptom: Timeout when fetching records before export

Solution:

  1. Enable Skip row version sorting on the setup page
  2. Export completes without sorting
  3. ⚠️ May cause duplicate exports on subsequent runs
  4. Disable after initial data transfer completes

When to use:

  • Large tables (millions of records)
  • Initial export only
  • Temporary measure

2. Upload Timeout (Block Commit)

Symptom: Error message like Could not commit blocks... OperationTimedOut

Solution 1: Reduce payload size

  1. Open Export to Azure Data Lake Storage page
  2. Reduce Max payload size (MiBs) setting
  3. Try values: 4 → 2 → 1

Solution 2: Increase timeout (advanced)

  1. Open AL code: ADLSEGen2Util.Codeunit.al
  2. Find procedure CommitAllBlocksOnDataBlob
  3. Add timeout parameter to URL: ?timeout=XX
    • Replace XX with seconds (e.g., 300 for 5 minutes)
  4. Recompile and redeploy extension

Solution 3: Optimize table

  1. Export fewer fields
  2. Use Skip row version sorting
  3. Export during off-peak hours
  4. Enable read-only replica

How do I optimize performance for large datasets?

Business Central side:

✅ Enable read-only replica (BC 21+)

  • Go to Feature Management
  • Enable Report read-only data access
  • Exports read from replica database
  • Reduces impact on production

✅ Schedule wisely

  • Export during off-peak hours
  • Stagger company exports
  • Use categories for different frequencies

✅ Optimize field selection

  • Export only necessary fields
  • Fewer fields = faster exports
  • Smaller file sizes

✅ Configure payload size

  • Default: 4 MiB
  • Increase for faster uploads (if no timeouts)
  • Decrease if experiencing timeouts

Azure/Fabric side:

✅ Use Parquet format (ADLS)

  • Better compression
  • Faster queries
  • Handles special characters better

✅ Scale Synapse workspace (ADLS)

  • Increase DWU for pipelines
  • Add more Spark nodes
  • Optimize pipeline parallelism

✅ Use parallel notebooks (Fabric)

  • Use CopyBusinessCentral_parallel_option.ipynb
  • Process multiple tables simultaneously
  • Adjust parallel count based on capacity

✅ Adequate capacity (Fabric)

  • Ensure sufficient Fabric capacity
  • Monitor capacity metrics
  • Scale up during heavy loads

How many tables can I export?

Technical limits:

Best practices:

  • Export tables in batches using categories
  • Monitor session capacity
  • Tables not exported due to limits will retry on next run
  • Typical deployments: 50-200 tables

Session management:

  • bc2adls spawns one session per table per company
  • BC has limits on concurrent sessions
  • Use staggered schedules for many tables

Troubleshooting

How do I track files in the deltas folder?

Each delta file has metadata:

File properties:

  • Modified timestamp: When export finished
  • Size: Amount of data exported
  • Name pattern: <company>/<table>_<timestamp>.csv

Tracking methods:

  1. Compare with ADLSE Run table:

    • View in BC: Related > Runs
    • Match Modified time with Started/Ended fields
    • Correlate files to specific export runs

  2. Use telemetry:

    • Enable Emit telemetry on setup page
    • Track in Application Insights
    • See which run created which file

  3. Azure Storage Explorer:

Export succeeded but I don't see data in Power BI

Checklist:

✅ Verify consolidation ran

  • Check Synapse pipeline ran successfully (ADLS)
  • Check Fabric notebook executed (Fabric)
  • Look for data in data folder (ADLS) or tables (Fabric)

✅ Refresh Power BI

  • Click Refresh Preview in Power BI
  • Close and reopen dataset
  • Check connection settings

✅ Verify CDM manifest (ADLS only)

  • Look for data.cdm.manifest.json in container
  • Verify tables listed in manifest
  • Check file paths are correct

✅ Check permissions

  • Service principal has read access
  • Power BI credential is valid
  • Azure/Fabric permissions are correct

Pipeline fails with "File not found"

Common causes:

  1. No deltas exported:

    • Verify BC export completed
    • Check deltas folder has files
    • Ensure container name is correct

  2. Wrong container parameter:

    • Pipeline parameter must match BC configuration
    • Check spelling and case

  3. Permissions issue:

    • Synapse workspace identity needs Storage Blob Data Contributor
    • Service principal has correct roles

Why are there duplicate records?

Expected scenarios:

  1. Tables with DataPerCompany = false:

    • These tables export from all companies
    • Same data exported multiple times
    • Pipeline removes duplicates during consolidation

  2. Table reset and re-export:

    • When you reset a table's export
    • Historical data exports again
    • Creates duplicates in deltas
    • Pipeline handles deduplication

Solution:

  • ✅ Consolidation pipeline automatically removes duplicates
  • ✅ No action needed from your side
  • ℹ️ This is expected behavior

Advanced Topics

Can I customize the export logic?

Yes! bc2adls is open source:

  1. Clone the GitHub repository
  2. Modify AL code in businessCentral/app/src
  3. Common customizations:
    • Add custom field transformations
    • Implement custom validation
    • Add export events
    • Modify file naming conventions
  4. Recompile and deploy to your environment

Important:

  • ⚠️ Customizations make upgrades harder
  • 📝 Document your changes
  • 🧪 Test thoroughly
  • 🔄 Consider contributing back to community

How do I handle schema changes?

When table schema changes (add/remove fields):

Minor changes (add fields):

  1. Enable new field on export page
  2. Next export includes new field
  3. CDM manifest updates automatically
  4. Previous records have null/empty for new field

Major changes (remove fields, type changes):

  1. Archive current data folder
  2. Disable old field, enable replacement
  3. Export schema with new structure
  4. Update Power BI / downstream consumers
  5. Optionally migrate historical data

Can I use bc2adls with Business Central On-Premises?

Yes! bc2adls supports both:

  • ✅ Business Central Online
  • ✅ Business Central On-Premises

On-Premises considerations:

  • Ensure network connectivity to Azure/Fabric
  • Configure firewall rules for outbound connections
  • Service principal authentication still required
  • Consider VPN or ExpressRoute for security

⚠️ On-Premises: DisableWriteInsideTryFunctions

On BC On-Premises, the server configuration setting DisableWriteInsideTryFunctions is set to True by default. This blocks database writes (including IsolatedStorage) inside TryFunction procedures, which can cause token cache errors during export.

If you encounter issues with the token cache, set DisableWriteInsideTryFunctions to False in your BC server configuration.

This is a platform difference between BC On-Premises and SaaS — in BC SaaS this restriction is not enabled. See Microsoft documentation on TryFunction behavior for details.

How do I implement row-level security?

Row-level security is implemented in the consumption layer, not in bc2adls:

Power BI approach:

  1. Use Power BI Row-Level Security (RLS)
  2. Define roles and filters
  3. Apply to datasets

SQL approach (Shared metadata tables):

  1. Create views with filters
  2. Grant users access to views only
  3. Implement security in SQL layer

Synapse approach:

  1. Use Synapse serverless SQL pools
  2. Implement views with security filters
  3. Use SQL authentication

Fabric approach:

  1. Use Fabric's built-in security features
  2. Configure data access policies
  3. Implement in Power BI semantic models

Can I export attachments or documents?

Not directly. But you can:

  1. For file links:

    • Export the file path field
    • Consumers access files via path

  2. For embedded files:

    • Use BLOB workaround (Base64 conversion)
    • Store in separate table
    • Export as text

  3. For SharePoint/OneDrive:

    • Export link fields
    • Access files via URL

Getting Help

I didn't find my question here

Try these resources:

  1. GitHub Issues: Browse existing issues
  2. Create new issue: Report a bug or request feature
  3. Community: Join discussions on GitHub
  4. Documentation: Review other wiki pages

How do I report a bug?

  1. Go to GitHub Issues
  2. Provide:
    • BC version
    • Export option (ADLS/Fabric/Mirroring)
    • Steps to reproduce
    • Error messages
    • Screenshots if applicable
  3. Search existing issues first to avoid duplicates

How can I contribute?

We welcome contributions!

  1. Code: Submit pull requests
  2. Documentation: Improve wiki pages
  3. Testing: Test new features
  4. Feedback: Share your experience
  5. Support: Help others in issues

See Contributing Guidelines

Additional Resources

Last updated: 2025 Have more questions? Ask on GitHub

Clone this wiki locally