Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure Datalake Blob storage gen2 regression #41

Open
dev1010101 opened this issue Oct 14, 2023 · 1 comment
Open

Azure Datalake Blob storage gen2 regression #41

dev1010101 opened this issue Oct 14, 2023 · 1 comment
Labels
Azure Data Lake bug Something isn't working

Comments

@dev1010101
Copy link

Problem:
Try to create a blob connection azure blobstorage gen2 with FromConnectionString
did the StorageFactory.Modules.UseAzureBlobStorage() to register the azure blob storage extention
throws a NullReferenceException

Cause:
ExtendedSdk.GetHttpPipeline(BlobServiceClient sdkClient) is using internals of Azure BlobServiceClient and they have changed I assume.

This ugly hack did the trick:

            var accountName = <<StorageAccountName>>;
            var credential = new StorageSharedKeyCredential(accountName ,<<accessToken>>);
            
            var uri = new Uri($"https://{accountName}.blob.core.windows.net/");
            var client = new BlobServiceClient(uri, credential);


            FieldInfo BlobClientConfigurationField =
                typeof(BlobServiceClient).GetField("_clientConfiguration", BindingFlags.NonPublic | BindingFlags.Instance);

            var clientConfig = BlobClientConfigurationField.GetValue(client);
            var clientConfigType = clientConfig.GetType();
            PropertyInfo httpPipelineProperty =
                clientConfigType.GetProperty("Pipeline", BindingFlags.Public | BindingFlags.Instance);
            
            Azure.Core.Pipeline.HttpPipeline httpPipeline = httpPipelineProperty.GetValue(clientConfig) as Azure.Core.Pipeline.HttpPipeline;

            Type AzureDataLakeStorageType = typeof(IBlobStorage)
                .Assembly.GetType("FluentStorage.Azure.Blobs.AzureDataLakeStorage");
            // ConstructorInfo method = t.GetConstructor(new Type[] { typeof(BlobServiceClient), typeof(string), typeof(StorageSharedKeyCredential), typeof(string )} );
            //# Object o = method.Invoke(new Object[] {client, accountName, credential, null});
            Type extendedSdkType = typeof(IBlobStorage)
                .Assembly.GetType("Blobs.ExtendedSdk");

            var dlStorage = FormatterServices.GetUninitializedObject(AzureDataLakeStorageType);
            var extendedSdk = FormatterServices.GetUninitializedObject(extendedSdkType);

            var sdkClientField =
                extendedSdkType.GetField("_sdkClient", BindingFlags.Instance | BindingFlags.NonPublic);
            var pipelineField = extendedSdkType.GetField("_httpPipeline", BindingFlags.Instance | BindingFlags.NonPublic);
            var dsfBaseAddress = extendedSdkType.GetField("_dfsBaseAddress", BindingFlags.Instance | BindingFlags.NonPublic);

            sdkClientField.SetValue(extendedSdk, client);
            pipelineField.SetValue(extendedSdk, httpPipeline);
            dsfBaseAddress.SetValue(extendedSdk, $"https://{accountName}.dfs.core.windows.net/");

            var extendedField = AzureDataLakeStorageType.GetField("_extended", BindingFlags.Instance | BindingFlags.NonPublic);
            extendedField.SetValue(dlStorage, extendedSdk);
            

            var clientField2 = AzureDataLakeStorageType.BaseType.GetField("_client", BindingFlags.Instance | BindingFlags.NonPublic);
            clientField2.SetValue(dlStorage, client);

            var sasCredField = AzureDataLakeStorageType.BaseType.GetField("_sasSigningCredentials", BindingFlags.Instance | BindingFlags.NonPublic);
            sasCredField.SetValue(dlStorage, credential);

            var containerField = AzureDataLakeStorageType.BaseType.GetField("_containerNameToContainerClient", BindingFlags.Instance | BindingFlags.NonPublic);
            containerField.SetValue(dlStorage, new ConcurrentDictionary<string, BlobContainerClient>());
@robinrodricks
Copy link
Owner

Can you possibly file a PR with a fix and submit it? I would love to add a fix for this into the main library.

@robinrodricks robinrodricks added bug Something isn't working Azure Data Lake labels Oct 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Azure Data Lake bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants